[RFQ] Content hashes at home

[oharboe]

ccache Performance Benchmark for OpenROAD Bazel Build

Why ccache?

Bazel's built-in action cache is keyed on the action graph (flags, BUILD files,
dependency versions), not on source-file contents. Common developer workflows
-- switching branches, editing a BUILD file, or running bazel clean -- invalidate
the action cache even when most source files are unchanged.

ccache caches based on preprocessed source content, so it survives these
invalidations and delivers near-instant cache hits for unchanged translation units.

Environment

Item	Value
CPU	Intel(R) Core(TM) i9-9900KF CPU @ 3.60GHz (16 cores)
RAM	30Gi
OS	Ubuntu 25.10
ccache	ccache version 4.11.2
Bazel	bazel 8.5.0
Commit	82027a28d3
Sources	1005 .cpp/.cc files, 815 .h/.hpp files, 33 modules
Build targets	83 cc_library targets (excluding gui)
Runs per scenario	3 (median reported)

Developer Personas (from git history)

We analyzed the last 500 commits to identify typical developer workflows:

Persona	Description	Simulated Change	Example Contributors
Single-tool bug fix	Fix a bug in one module	1 .cc file in rsz	Thinh Nguyen, Jaehyun Kim
Algorithm improvement	Improve algorithm in one tool	3 .cpp files in grt	Augusto Berndt, Eder Monteiro
Cross-cutting refactor	Rename API, clang-tidy cleanup	20 files across 5 modules	Henner Zeller

Methodology

1. Full build with --config=ccache to warm ccache (all 83 cc_library targets)
2. For each scenario, repeat 3 times:
   a. Append a unique comment to the scenario's source files (real content change, not just touch)
   b. bazel clean to invalidate Bazel's action cache
   c. Rebuild with ccache -- record wall-clock time and hit rate
   d. Restore files via git checkout
   e. Re-apply same edits, bazel clean again
   f. Rebuild without ccache (CCACHE_DISABLE=1) -- record wall-clock time
   g. Restore files
3. Bazel disk cache and remote cache disabled (--disk_cache= --remote_cache=) to isolate ccache effect
4. Machine was dedicated to the benchmark (no other builds running)
5. Median of 3 runs reported

Results

Scenario	Files Changed	With ccache (s)	Without ccache (s)	Speedup	ccache Hits
Single-tool bug fix	1	143.1	1521.3	10.6x	5790/5790
Algorithm improvement	3	147.6	1522.4	10.3x	5790/5790
Cross-cutting refactor	20	154.0	1522.8	9.8x	5790/5790

Key Findings

• 10x speedup across all scenarios -- ccache reduces rebuild time from ~25 minutes
  to ~2.5 minutes after bazel clean.
• 100% cache hit rate (5790/5790) -- even the cross-cutting refactor touching 20 files
  across 5 modules achieves full hits because Bazel recompiles all targets after
  bazel clean, but ccache recognizes unchanged preprocessed content.
• Consistent with-ccache times (~145s) -- the number of changed files barely affects
  rebuild time because ccache serves all unchanged translation units from cache.
  The small increase from 143s to 154s reflects the additional files that must be
  genuinely recompiled.
• Without-ccache times are stable (~1522s) -- confirming the machine was uncontested
  and results are reliable.

When does this help?

Any workflow that invalidates Bazel's action cache while leaving most source unchanged:

• bazel clean -- full action cache wipe, but ccache still has all objects
• Branch switching -- different action graph, same source content
• BUILD file edits -- invalidates the target's action cache even if no source changed
• Merging origin/master -- MODULE.bazel or BUILD changes cascade through the action graph
• Flag changes -- different -c opt vs -c dbg etc.

Setup

Add to your user.bazelrc:

echo 'build --config=ccache' >> user.bazelrc

Or invoke explicitly:

bazel build --config=ccache //...

Reproducing

The benchmark script is at tools/ccache_benchmark.sh:

bash tools/ccache_benchmark.sh --runs=3

[oharboe]

@hzeller Thoughts?

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[gemini-code-assist]

Code Review

This pull request introduces ccache support to improve local build times, which is a valuable enhancement for developer productivity. The implementation via a patch to toolchains_llvm and a new Bazel configuration is well-executed. The included refactoring of genrules to more specific bazel_skylib rules is also a good cleanup. I have one suggestion to make the ccache integration more robust for users with customized environments.

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[hzeller]

I always comment out the llvm toolchain, so I don't know what it does. But usually bazel does employ some content-based hashing, but @QuantamHD might know more.

One problem we have is that we have currently have super coarse grained libraries due to the historic way OpenROAD is compiled, so many files are compiled into one library instead have having many small libraries. So if any of the files in these change, the compilation that builds that probably has to rebuild all as they use these param files (don't know exactly).

Anyway, if ccache works for you, you should use it. Longer term, we should structure the project to be more fine-grained.

[oharboe]

The idea is to have ccache mitigate the coarse grained libraries.

[maliberty]

This is painful during active rebasing where header/flag churn causes frequent cache misses.

If headers change you should recompile. Flags change rarely. How often do you encounter this? It's never been a pain point for me.

[oharboe]

This is painful during active rebasing where header/flag churn causes frequent cache misses.

If headers change you should recompile. Flags change rarely. How often do you encounter this? It's never been a pain point for me.

I find myself recompiling all the time.

[vvbandeira]

Insufficient disk space on the Jenkins worker for the Bazel build cache and sandbox. This is an infrastructure issue, not a code issue.

Yesterday we reduced the workers disk from 500GB to 100GB to allow more workers with the same resources.
Even after all the dependencies are installed I am seeing less than 20GB of disk used and ~80GB available.
@oharboe, is it normal or expected for Bazel to use more than 80GB of local disk?

@jenkins-worker-central1-c3d-1kx6ey:~$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1   99G   17G   78G  18% /

[oharboe]

I have not studied disk usage.

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[QuantamHD]

I think this is not a good solution, and a better solution would be to break up the existing coarse grained targets into fine grained rules rather than mask this problem with another layer of caching.

[oharboe]

I think this is not a good solution, and a better solution would be to break up the existing coarse grained targets into fine grained rules rather than mask this problem with another layer of caching.

Makes sense to me if we can have it in some reasonable amount of time. This could be a stopgap. I have an experiment to measure the benefit of this, which is interesting information, regardless of solution.

[oharboe]

@QuantamHD @hzeller @maliberty Fantastic if it is true: 10x faster while we wait for bazel migration to complete at which point we can retire this. Unless bazel grows this feature and we can also retire this.

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[oharboe]

@maliberty @hzeller @QuantamHD Killer use-ase: caching across git work trees...

OpenROAD Build Optimization: ccache Integration (PR #9738)

Executive Summary

Pull Request #9738 introduces ccache support into the OpenROAD Bazel build system. While Bazel has a built-in action cache, it is often invalidated by common developer workflows (switching branches, editing BUILD files, or using git worktree). This PR adds a secondary layer of caching that looks at source content rather than action graphs.

The Problem: Why Bazel Cache Isn't Enough

Bazel's native cache is "fragile" in local development because it is keyed on the Action Graph. If any of the following change, Bazel may force a recompile even if the .cpp file is identical:

• Toolchain changes: Updating compiler versions or flags.
• Workspace paths: Moving code or using different git worktree locations.
• Metadata changes: Modifying a BUILD file or MODULE.bazel, even if it doesn't affect the specific C++ logic.

The Solution: How ccache Helps

ccache functions as a compiler wrapper. It hashes the preprocessed source code and the compiler options. If the resulting hash matches a previous build, it retrieves the object file from a global cache directory (usually ~/.cache/ccache) instead of recompiling.

[Image of ccache architecture showing how it intercepts compiler calls and interacts with the cache]

Key Use Case: Cross-Worktree & Branch Caching

The most significant benefit of this PR is for developers managing multiple tasks:

• Git Worktrees: Usually, each worktree has its own Bazel output base. Without ccache, you have to compile the same code from scratch for every new worktree. With this PR, worktrees share the global ccache storage