Marble v1.4.4: +1.8V output from LTM4673 fails to stabilize, preventing other quads from powering on

[sdmurthy]

A consistent power-on failure on the new Marble v1.4.4 boards. Out of a recent batch of 25 (built by SLAC), all three boards tested so far exhibit the same problem after a successful programming of the MMC and the LTM4673 power management chip. While the first quad enables correctly with the +1V0 LED on and stable, the rest of the board fails to power up because the +1.8V rail struggles to come up, which is indicated by its corresponding LED blinking at a very low intensity. This prevents the remaining quads from receiving power.

[stijnwielandt]

Here's an update on the problem:

• The LTM4673 was going into hiccup mode on channel 1 (+1.8V). All loads and feedback resistor values were compared to a working Marble 1.4.1 board and exhibit normal values.
• LTM4673 registers in EEPROM on a failing Marble 1.4.4 board were compared to a working Marble 1.4.1 board, showing incorrect output voltage settings (1.2V instead of 1.8V) and TON_MAX_FAULT values (15ms instead of 150ms). I suspect that register values were not stored in EEPROM during mmc bringup, so we are most likely just looking at factory default settings.
• The LTM4673 was programmed using a USB-I2C/PMBUS converter. This brought up all system voltages, so there does not seem to be hardware faults in the power rails.

Remaining issue to be solved:

• Why is the standard MMC bring‑up procedure failing to store the configuration to the LTM4673 EEPROM?

[mdavidsaver]

@stijnwielandt Can you share a capture of the MMC UART console output from power on?

[sdmurthy]

␀␀␀␀�** Marble init done **
PCB Rev: Marble v1.4
Chip ID: DEVID 0x0411 REVID 0x203f
Disabling watchdog
XRP7724 not present; bypassed.
LTM4673 Status 0x0801:

• The PWRGD pin, if enabled, is negated. Power is not good.
  Skipping adn4600_init
  ** Policy: reset FPGA on MMC reset. Doing it now. **
  **
  Marble v2 UART Console

[mdavidsaver]

• Why is the standard MMC bring‑up procedure failing to store the configuration to the LTM4673 EEPROM?

I don't suppose the output log from the bringup script was kept?

In particular, I wonder what happened during this section?

# 2. Program LTM4673 power management chip
echo "Programming LTM4673 power management chip...."
if ! python3 "$SCRIPTS_PATH"/ltm4673.py -d "$TTY_MMC" write -f "$LTM_SCRIPT"; then
  echo "Could not program LTM4673."
  exit 1
else
  echo "##################################"
  python3 "$SCRIPTS_PATH"/ltm4673.py -d "$TTY_MMC" store
  echo "napping for 5 seconds.."
  sleep 5
  python3 "$SCRIPTS_PATH"/load.py -d "$TTY_MMC" "4b"
  echo "napping for 5 seconds.."
  sleep 5
  python3 "$SCRIPTS_PATH"/load.py -d "$TTY_MMC" "4B"
  echo "Successfully programmed LTM4673!"
fi

[mdavidsaver]

For reference during future troubleshooting...

• LTM4673 registers in EEPROM on a failing Marble 1.4.4 board were compared to a working Marble 1.4.1 board

Was this done with one of the scripts/*ltm4673* scripts in https://github.com/BerkeleyLab/Marble-MMC ?

• The LTM4673 was programmed using a USB-I2C/PMBUS converter

Did you first attempt to re-run the bringup script?

[mdavidsaver]

My initial suspicion would be that some of the i2c/pmbus commands in scripts/ltm4673.py did not take effect for some reason (timing?). Adding a read-back and compare step seems prudent. From a read through, it looks like the code to support this already exists. I have created BerkeleyLab/Marble-MMC#3 with a change to bringup_mmc.sh to add a readback step.

[stijnwielandt]

I don't suppose the output log from the bringup script was kept?

I'm currently working on the bringup script (and underlying code). Seems like there's some inconsistency between runs, but here's the initial output of the script. Note that the PSU is not powering off like it's supposed to (this is one of the inconsistencies though).

Writing program from file /Users/stijnwielandt/gitlab/marble_mmc/LTM4673_reglist.txt.
Estimated 19.8s to complete.
>   Wrote 198 lines
>   done
>   closing
##################################
Attempting store to EEPROM
>   Wrote 2 lines
>   done
>   closing
Success
napping for 5 seconds..
4b
>   Wrote 1 lines
>   done
>   closing
napping for 5 seconds..
4B
>   Wrote 1 lines
>   done
>   closing
Successfully programmed LTM4673!
bringup_mmc DONE

[stijnwielandt]

Was this done with one of the scripts/*ltm4673* scripts in https://github.com/BerkeleyLab/Marble-MMC ?

No, comparison of the EEPROM values was done using the LT tools and a USB/PMBus controller. This is how I noticed that output voltage settings and TON_MAX_FAULT values were off.

• The LTM4673 was programmed using a USB-I2C/PMBUS converter

Did you first attempt to re-run the bringup script?

Yes, we first tried running the bringup script (many times, with SW4 both in on/off position).

[stijnwielandt]

bringup_mmc.sh was unsuccessful for multiple reasons. Over the past year, multiple code improvements have been made, but these updates had never been tested on out-of-the-box boards (which have dead power supplies until they are brought up). The following problems have been resolved and we can now successfully flash the LTM EEPROM. Fixes will be merged into the master branch asap.

• UART cyclic buffers were written in two different code locations. This resulted in wrong buffer pointers whenever loopback was triggered. This problem has been resolved by removing one buffer write operation
• I2C communications relied on fixed delays and no error handling was implemented. During bringup, the lack of power for many I2C chips means that these chips will still be unresponsive, resulting in I2C NACKs. Since there was no error handling, these NACKs were locking up the bus (often resulting in I2C busy warnings). To resolve this, an error handler was implemented to release the bus on errors, and I2C bus availability is checked before initiating new transfers.
• While writing LTM4673 registers, power levels will gradually be restored, resulting in a "Power Good" signal at some point. This "Power Good" will trigger an initialization of the microcontroller and its peripherals, including the I2C bus that is still in use during the ongoing bringup process. This results in the EEPROM not being written. As a solution, a "_LTM_console_active" variable was introduced, which prevents initialization for a duration of LTM_CONSOLE_ACTIVE_TIMEOUT_MS (5000) after an LTM read/write operation.
• Further improvements to the ltm4673.py, load.py, and bringup_mmc.sh scripts improve reliability, error handling, and feedback. The bringup script now performs readback checks by default.

[mdavidsaver]

Fixes will be merged into the master branch asap.

@stijnwielandt I do not think that I see these changes as of today (BerkeleyLab/Marble-MMC@1ae334a).