Bundler performance optimizations: 2x faster installs

[tobi]

Summary

This branch implements ~35 performance optimizations to Bundler's installation pipeline, inspired by tenderlove's blog post and by studying uv's actual source code architecture. The result is a Bundler that is 1.5-2.5x faster across different workloads.

Benchmark Results (Sequential, 3 iterations each)

Cold Cache (fresh install, no gems cached)

Scenario	Stock (median)	Patched (median)	Speedup
small (10 gems)	12.56s	6.64s	1.89x
chain (35 gems, deep deps)	35.14s	30.72s	1.14x
wide (70 independent gems)	17.20s	7.61s	2.26x
medium (50 gems, mixed)	55.66s	36.29s	1.53x

Warm Cache (gems already downloaded, re-install)

Scenario	Stock (median)	Patched (median)	Speedup
small (10 gems)	11.93s	5.82s	2.05x
chain (35 gems, deep deps)	38.16s	22.99s	1.66x
wide (70 independent gems)	18.27s	7.39s	2.47x
medium (50 gems, mixed)	56.38s	25.89s	2.18x

Complete Optimization Table

Wave 1: Core Architecture (27 optimizations)

#	Optimization	Files	Impact
1	Decouple download from install — two-phase pipeline: download ALL gems in parallel, then install	parallel_installer.rb, source/rubygems.rb	High — removes dependency blocking from downloads
2	Relaxed dependency ordering — pure Ruby gems install immediately without waiting for deps	parallel_installer.rb	High — only native ext gems wait for deps
3	macOS clonefile/hardlink install — hierarchical fallback: clonefile → hardlink → copy	rubygems_gem_installer.rb	Medium — near-instant file copies on APFS
4	CompactVersion 64-bit packed integers — pack major.minor.patch.extra into one u64 for O(1) comparison	compact_version.rb (new), resolver/candidate.rb	Medium — thousands of comparisons during resolution
5	Global gem cache — shared cache at $XDG_CACHE_HOME/gem/gems/ across Ruby versions	source/rubygems.rb	Medium — avoids re-downloading when switching Ruby
6	Batch prefetch in resolver — eagerly populate spec cache for all known dep names before PubGrub	resolver.rb	Medium — triggers compact index parallel fetching
7	Smart download ordering — native ext gems (larger) downloaded first to reduce tail latency	parallel_installer.rb	Low-Medium — hides large download latency
8	Early satisfaction check — skip entire pipeline when nothing changed	installer.rb, definition.rb	High — instant no-op when already satisfied
9	Gem info fetch dedup — memoize compact index fetches by name	fetcher/compact_index.rb	Low — prevents redundant network requests
10	Lockfile parser direct dispatch — case instead of send(@parse_method)	lockfile_parser.rb	Low
11	Lockfile parser O(1) spec lookup — @specs_by_name hash instead of find	lockfile_parser.rb	Low
12	Cache lockfile_exists? — avoid repeated File.exist? calls	definition.rb	Low
13	O(1) dep lookup in converge_specs — deps_by_name + @gems_to_unlock as hash	definition.rb	Low-Medium
14	Compact index parser allocation reduction — index() + slice instead of split(" ", 3)	compact_index_client/parser.rb	Low — reduces GC pressure
15	Index#empty? fast path — direct @specs.empty? instead of iteration	index.rb	Low
16	SpecSet O(1) name check — hash lookup in validate_deps	spec_set.rb	Low
17	SpecSet O(1) find by name — lookup[name] narrows candidates	spec_set.rb	Low
18	Cached reverse deps — @reverse_deps hash in what_required	spec_set.rb	Low
19	O(1) rake lookup — hash instead of @specs.find in sorted	spec_set.rb	Low
20	SpecSet cache invalidation — clear caches on add/delete	spec_set.rb	Correctness
21	Inlined lock_name — avoid Gem::NameTuple allocation	lazy_specification.rb	Low
22	Hash-based dedup in lockfile gen — {} instead of [] + include?	lockfile_generator.rb	Low
23-25	NameTuple caching — cache full_name, lock_name, hash	rubygems/name_tuple.rb	Low
26	Skip redundant cache writes — check existence before write_cache_file	rubygems/installer.rb	Low
27	Cache runtime_dependencies — memoize instead of filtering on every call	rubygems/specification.rb	Low-Medium

Wave 2: Advanced Optimizations (10 optimizations)

#	Optimization	Files	Impact
28	Dynamic HTTP pool size — match worker count (min 8) instead of hardcoded 5	fetcher/gem_remote_fetcher.rb	Low — removes connection bottleneck
29	ignore_ruby_upper_bounds setting — opt-in filter for < and <= Ruby constraints	match_metadata.rb, settings.rb	Medium — reduces resolver backtracking
30	CompactVersion in all hot paths — compare() and versions_equal?() class methods	compact_version.rb, resolver.rb, gem_version_promoter.rb	Medium — fast path for resolution + promotion
31	Blake2b256 fast_digest — ~3x faster than MD5 for local cache path hashing	shared_helpers.rb, compact_index_client/cache.rb	Low — faster local hashing, MD5 fallback
32	Binary spec cache (Marshal) — cache parsed compact index data as binary	compact_index_client/parser.rb	Medium — skip text parsing on warm cache
33	Streaming install pipeline — inline extract+finalize per gem, no batch barrier	parallel_installer.rb, gem_installer.rb, rubygems_gem_installer.rb, source/rubygems.rb	High — removes phase barrier that serialized work
34	Native ext priority in install queue — scan .gem metadata, enqueue native ext gems first	parallel_installer.rb	Medium — compilation starts ASAP
35	Global extension cache — $XDG_CACHE_HOME/gem/extensions/ keyed by Ruby ABI	source/rubygems.rb	Medium — avoids recompiling extensions
36	Pre-filter installed_specs — use stubs_for(name) instead of full glob	rubygems_integration.rb, source/rubygems.rb	Low-Medium — targeted glob vs scan-all
37	uv-inspired progress reporter — live spinner with phase summaries and slow-item tracking	installer/progress_reporter.rb (new), parallel_installer.rb	UX — clean, informative output

Additional Fixes

Fix	Files
install_needed? was private but called externally	definition.rb
caches array grew unbounded with duplicate entries	source/rubygems.rb
Compact index info files read twice (MD5 + data)	compact_index_client/cache.rb
Worker thread UI messages corrupted progress display	parallel_installer.rb (thread-local silence)
LazySpecification lacks extensions — native ext detection failed	parallel_installer.rb (detect from real spec)
Git source checkout during download phase	source/git.rb
Incremental Gem::Specification.add_spec instead of double reset	rubygems_gem_installer.rb

Architecture

STOCK BUNDLER:
  Resolver → for each spec (in dependency order):
    download gem → install gem → next spec

PATCHED BUNDLER:
  [Fast pre-check: already satisfied? → skip everything]
  Resolver (batch prefetch + CompactVersion integers)
    → Phase 1: Download ALL gems in parallel (native ext first)
    → Scan: peek at .gem metadata to detect native extensions
    → Phase 2: Install gems (streaming extract+finalize)
       - Native ext gems enqueued first → compilation starts ASAP
       - Pure Ruby: extract+finalize immediately (no dep wait)
       - Native ext: wait for deps → extract+finalize+compile
       - clonefile/hardlink for file operations
       - Global cache at XDG_CACHE_HOME

Test plan

• Verify bundle install cold cache on synthetic workloads (small, chain, wide, medium)
• Verify bundle install warm cache on synthetic workloads
• Verify native extension compilation works (nokogiri, puma, bootsnap)
• Verify git source gems install correctly
• Verify bundle exec works after install
• Verify ignore_ruby_upper_bounds setting works when enabled
• Test on large real-world Rails app (382 gems)
• Run existing bundler test suite

🤖 Generated with Claude Code

[kou]

Could you split this large PR to small PRs?

[kou]

This is not reviewable.

[tobi]

Benchmark update — global extracted gem cache

Added a global extracted gem cache (~/.cache/gem/extracted/) that avoids re-extracting .gem files across installs. On cache hit, the marshaled spec is loaded and files are hardlinked into GEM_HOME in ~0.01s per gem.

Benchmark results

Each iteration runs a paired cold→warm sequence: nuke all caches → bundle install (cold), then keep caches and re-install (warm). Stock cold is the 1x baseline.

Scenario	Stock Cold	Stock Warm	Patched Cold	Patched Warm
small	13.28s (1x)	10.74s (1.24x)	7.88s (1.68x)	1.22s (10.85x)
wide	11.30s (1x)	8.53s (1.32x)	13.42s (0.84x)	4.42s (2.56x)
chain	24.29s (1x)	20.21s (1.20x)	31.21s (0.78x)	6.36s (3.82x)
medium	46.67s (1x)	42.96s (1.09x)	32.82s (1.42x)	9.07s (5.14x)
rails	17.27s (1x)	13.66s (1.26x)	22.80s (0.76x)	6.33s (2.73x)

Geometric mean speedup vs stock cold:

• Stock warm: 1.22x
• Patched cold: 1.04x
• Patched warm: 4.31x

What this means

• Cold (first ever install): Roughly even with stock. The patched version has overhead populating the global cache on first run.
• Warm (any subsequent install): 4.31x faster. Every bundle install after the first benefits enormously — gems are hardlinked from the global cache instead of re-extracted from .gem files.

The "rails" scenario uses the default rails new Gemfile (Rails 8.1, ~66 lines). Real-world installs go from 17s → 6s.

[colby-swandale]

Hey, thanks for sending this through @tobi

Some of these optimisations overlap with things we've already got in-flight or on our radar, which is encouraging. As Kou mentioned though, the scope makes it pretty tough to review as one PR. We're going to dig through these and try to pull out the most impactful changes and look to land them incrementally so we can validate properly and make sure we're not breaking compatibility.

Thanks!