Bazel/Regression: //src/pdn/test:pads_black_parrot_flipchip_connect_overpads-tcl_test fails randomly

[fredowski]

Describe the bug

The test //src/pdn/test:pads_black_parrot_flipchip_connect_overpads-tcl_test fails randomly on MacOS, Debian 13 and Ubuntu 2204 when tested with bazel. From 100 runs 16 fail on Debian 13, 17 fail on MacOS.

I saw this also in the ci regression in the bazel part. I could not observe this is the ctest results for the same test.

Expected Behavior

The test should either fail or pass. Preferred pass :-)

Environment

I tested this on MacOS 15.7.4 (arm64), Debian 13 x86_64 and in Ubuntu 2204 in a docker machine.

To Reproduce

Use the following shell script "blackparrot.sh" to run the test 100 times on your system.

#!/bin/bash

# Number of times to run the test
RUNS=100

# Counters
pass_count=0
fail_count=0

# Test target
TEST="//src/pdn/test:pads_black_parrot_flipchip_connect_overpads-tcl_test"

# Loop
for i in $(seq 1 $RUNS); do
    echo "Run #$i..."
    
    # Run test with no cache and no disk/remote cache
    bazelisk test --nocache_test_results "$TEST" --disk_cache= --remote_cache=
    
    # Check exit status ($? is 0 if success, non-zero if failure)
    if [ $? -eq 0 ]; then
        ((pass_count++))
        echo "PASS"
    else
        ((fail_count++))
        echo "FAIL"
    fi
done

# Summary
echo "-----------------------"
echo "Total runs: $RUNS"
echo "Passes: $pass_count"
echo "Fails: $fail_count"

Relevant log output

Debian 13 x86_64:


Executed 1 out of 1 test: 1 test passes.
PASS
Run #100...
INFO: Analyzed target //src/pdn/test:pads_black_parrot_flipchip_connect_overpads-tcl_test (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
Target //src/pdn/test:pads_black_parrot_flipchip_connect_overpads-tcl_test up-to-date:
  bazel-bin/src/pdn/test/pads_black_parrot_flipchip_connect_overpads-tcl_test_test.sh
INFO: Elapsed time: 0.795s, Critical Path: 0.56s
INFO: 2 processes: 2 linux-sandbox.
INFO: Build completed successfully, 2 total actions
//src/pdn/test:pads_black_parrot_flipchip_connect_overpads-tcl_test      PASSED in 0.5s

Executed 1 out of 1 test: 1 test passes.
PASS
-----------------------
Total runs: 100
Passes: 84
Fails: 16


MacOS:


Executed 1 out of 1 test: 1 fails locally.
FAIL
Run #100...
INFO: Analyzed target //src/pdn/test:pads_black_parrot_flipchip_connect_overpads-tcl_test (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
Target //src/pdn/test:pads_black_parrot_flipchip_connect_overpads-tcl_test up-to-date:
  bazel-bin/src/pdn/test/pads_black_parrot_flipchip_connect_overpads-tcl_test_test.sh
INFO: Elapsed time: 0.447s, Critical Path: 0.29s
INFO: 2 processes: 2 darwin-sandbox.
INFO: Build completed successfully, 2 total actions
//src/pdn/test:pads_black_parrot_flipchip_connect_overpads-tcl_test      PASSED in 0.3s

Executed 1 out of 1 test: 1 test passes.
PASS
-----------------------
Total runs: 100
Passes: 83
Fails: 17

Screenshots

No response

Additional Context

No response

[maliberty]

It is a bit flaky even on Linux and is in the todo list (at least on linux is a low frequency of failure). Are you noting this or intending to work on it?

[fredowski]

Im currently looking into it.

[fredowski]

Claudes current theory, but no fix yet.

===========================================================================
KEY ARCHITECTURAL INSIGHT
===========================================================================

Shape::ShapeTreeMap = std::map<odb::dbTechLayer*, ShapeTree>

The fundamental issue: this map's iteration order is pointer-address-based.
All loops "for (const auto& [layer, shapes] : some_shape_map)" that have
ORDER-DEPENDENT behavior are potential non-determinism sources.

[maliberty]

I'm not sure that is true as this should use ids not pointer comparison

OpenROAD/src/odb/include/odb/dbCompare.inc

Line 865 in ea64bf0

struct less<odb::dbTechLayer*>

[gadfort]

@maliberty last time I tried looking at this the "different" was from a rtree.query call in the PadConnect straps which sometimes returns a different item first and thus there is a difference.

[maliberty]

That is a bit surprising. I saw differences between cmake & bazel, presumably due to compiler differences, but I would expect the order to be deterministic with a single compiler.

[fredowski]

This test is really brittle. The two different results differ by one additional shape. I guess somebody with more insight in how pdn works needs to look into this. At least the problem is easy to reproduce with this test. I give up...