Investigate `gh-pages` impact on full clone size and time

[rwgk]

Description

A local controlled file:// clone experiment suggests that gh-pages is substantially inflating normal full-clone cost for NVIDIA/cuda-python.

Measured clone times:

• With gh-pages: 47.44 seconds
• Without gh-pages: 1.79 seconds

Measured size estimates from the same experiment:

• With gh-pages: .git size about 1.2G
• Without gh-pages: .git size about 38M
• Estimated reduction: about 1.19 GB by filesystem size, or about 1.17 GB by Git object disk-usage accounting

Full measurement details and evidence are in:

• Branch cleanup #2190 (comment)

For initial solution exploration, start from:

• Branch cleanup #2190 (comment)

---

Additional real-world measurements from two NVIDIA-internal wired-network machines (see comments below for details):

Machine	Clone mode	Elapsed time	Received	.git size	Git pack size	Packed objects
smc120-0009	normal full clone	122.85 s	1.25 GiB	1.3G	1.27 GiB	692,924
smc120-0009	--single-branch --branch main	5.77 s	36.11 MiB	38M	36.66 MiB	20,389
smc120-0009	--depth 1 --branch main	2.46 s	2.66 MiB	3.0M	2.68 MiB	903
Dell Precision 7875	normal full clone	92.60 s	1.25 GiB	1.3G	1.27 GiB	692,924
Dell Precision 7875	--single-branch --branch main	4.55 s	36.17 MiB	38M	36.72 MiB	20,389
Dell Precision 7875	--depth 1 --branch main	2.11 s	2.66 MiB	3.0M	2.68 MiB	903

In both runs, the normal clone fetched about 693k packed objects and 1.25 GiB, while the single-branch main clone fetched about 20k objects and 36 MiB.

[rwgk]

Quick script to get some real-world measurements (will post the outputs in following comments):

#!/usr/bin/env bash
set -euo pipefail
set -x

repo_url="git@github.com:NVIDIA/cuda-python.git"
base="${1:-/tmp/cuda-python-clone-exp-$(date +%s)}"

mkdir -p "$base"
cd "$base"

/usr/bin/time -f 'normal elapsed: %e s' \
  git clone "$repo_url" normal

/usr/bin/time -f 'single-branch elapsed: %e s' \
  git clone --single-branch --branch main "$repo_url" single-branch

/usr/bin/time -f 'shallow elapsed: %e s' \
  git clone --depth 1 --branch main "$repo_url" shallow

du -sh normal/.git single-branch/.git shallow/.git

git -C normal count-objects -vH
git -C single-branch count-objects -vH
git -C shallow count-objects -vH

[rwgk]

A general colossus machine with great but in no way special network connectivity (smc120-0009):

+ repo_url=git@github.com:NVIDIA/cuda-python.git
++ date +%s
+ base=/tmp/cuda-python-clone-exp-1781149887
+ mkdir -p /tmp/cuda-python-clone-exp-1781149887
+ cd /tmp/cuda-python-clone-exp-1781149887
+ /usr/bin/time -f 'normal elapsed: %e s' git clone git@github.com:NVIDIA/cuda-python.git normal
Cloning into 'normal'...
remote: Enumerating objects: 692924, done.
remote: Counting objects: 100% (3816/3816), done.
remote: Compressing objects: 100% (1088/1088), done.
remote: Total 692924 (delta 3517), reused 2785 (delta 2719), pack-reused 689108 (from 3)
Receiving objects: 100% (692924/692924), 1.25 GiB | 18.72 MiB/s, done.
Resolving deltas: 100% (664690/664690), done.
normal elapsed: 122.85 s
+ /usr/bin/time -f 'single-branch elapsed: %e s' git clone --single-branch --branch main git@github.com:NVIDIA/cuda-python.git single-branch
Cloning into 'single-branch'...
remote: Enumerating objects: 20389, done.
remote: Counting objects: 100% (463/463), done.
remote: Compressing objects: 100% (372/372), done.
remote: Total 20389 (delta 292), reused 97 (delta 91), pack-reused 19926 (from 3)
Receiving objects: 100% (20389/20389), 36.11 MiB | 16.20 MiB/s, done.
Resolving deltas: 100% (14713/14713), done.
single-branch elapsed: 5.77 s
+ /usr/bin/time -f 'shallow elapsed: %e s' git clone --depth 1 --branch main git@github.com:NVIDIA/cuda-python.git shallow
Cloning into 'shallow'...
remote: Enumerating objects: 903, done.
remote: Counting objects: 100% (903/903), done.
remote: Compressing objects: 100% (772/772), done.
remote: Total 903 (delta 181), reused 452 (delta 113), pack-reused 0 (from 0)
Receiving objects: 100% (903/903), 2.66 MiB | 5.71 MiB/s, done.
Resolving deltas: 100% (181/181), done.
shallow elapsed: 2.46 s
+ du -sh normal/.git single-branch/.git shallow/.git
1.3G	normal/.git
38M	single-branch/.git
3.0M	shallow/.git
+ git -C normal count-objects -vH
count: 0
size: 0 bytes
in-pack: 692924
packs: 1
size-pack: 1.27 GiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes
+ git -C single-branch count-objects -vH
count: 0
size: 0 bytes
in-pack: 20389
packs: 1
size-pack: 36.66 MiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes
+ git -C shallow count-objects -vH
count: 0
size: 0 bytes
in-pack: 903
packs: 1
size-pack: 2.68 MiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

[rwgk]

My Dell Precision 7875 Tower, internal wired network, with great but in no way special connectivity:

+ repo_url=git@github.com:NVIDIA/cuda-python.git
++ date +%s
+ base=/tmp/cuda-python-clone-exp-1781150477
+ mkdir -p /tmp/cuda-python-clone-exp-1781150477
+ cd /tmp/cuda-python-clone-exp-1781150477
+ /usr/bin/time -f 'normal elapsed: %e s' git clone git@github.com:NVIDIA/cuda-python.git normal
Cloning into 'normal'...
remote: Enumerating objects: 692924, done.
remote: Counting objects: 100% (3816/3816), done.
remote: Compressing objects: 100% (1088/1088), done.
remote: Total 692924 (delta 3517), reused 2785 (delta 2719), pack-reused 689108 (from 3)
Receiving objects: 100% (692924/692924), 1.25 GiB | 19.72 MiB/s, done.
Resolving deltas: 100% (664690/664690), done.
normal elapsed: 92.60 s
+ /usr/bin/time -f 'single-branch elapsed: %e s' git clone --single-branch --branch main git@github.com:NVIDIA/cuda-python.git single-branch
Cloning into 'single-branch'...
remote: Enumerating objects: 20389, done.
remote: Counting objects: 100% (443/443), done.
remote: Compressing objects: 100% (361/361), done.
remote: Total 20389 (delta 275), reused 88 (delta 82), pack-reused 19946 (from 3)
Receiving objects: 100% (20389/20389), 36.17 MiB | 17.01 MiB/s, done.
Resolving deltas: 100% (14701/14701), done.
single-branch elapsed: 4.55 s
+ /usr/bin/time -f 'shallow elapsed: %e s' git clone --depth 1 --branch main git@github.com:NVIDIA/cuda-python.git shallow
Cloning into 'shallow'...
remote: Enumerating objects: 903, done.
remote: Counting objects: 100% (903/903), done.
remote: Compressing objects: 100% (772/772), done.
remote: Total 903 (delta 180), reused 454 (delta 113), pack-reused 0 (from 0)
Receiving objects: 100% (903/903), 2.66 MiB | 6.82 MiB/s, done.
Resolving deltas: 100% (180/180), done.
shallow elapsed: 2.11 s
+ du -sh normal/.git single-branch/.git shallow/.git
1.3G	normal/.git
38M	single-branch/.git
3.0M	shallow/.git
+ git -C normal count-objects -vH
count: 0
size: 0 bytes
in-pack: 692924
packs: 1
size-pack: 1.27 GiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes
+ git -C single-branch count-objects -vH
count: 0
size: 0 bytes
in-pack: 20389
packs: 1
size-pack: 36.72 MiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes
+ git -C shallow count-objects -vH
count: 0
size: 0 bytes
in-pack: 903
packs: 1
size-pack: 2.68 MiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

[rwgk]

Question

If gh-pages were removed from the normal branch ref set, what would we expect a normal real-world GitHub SSH clone time to be?

Answer

Based on the two GitHub SSH measurements above, removing gh-pages from the default branch ref set would likely make a normal full clone comparable to today's --single-branch --branch main clone: roughly 5-8 seconds instead of roughly 90-125 seconds on the measured NVIDIA-internal machines.

Details

I would not use the local file:// result of 1.79 seconds as the expected real GitHub clone time. That local experiment is most useful for estimating the object-set delta.

For real GitHub SSH clone timing, the better anchor is the observed single-branch full-depth clone:

• smc120-0009: --single-branch --branch main took 5.77 seconds.
• Dell Precision 7875: --single-branch --branch main took 4.55 seconds.

The local simulation without gh-pages produced a .git size of about 38M, essentially the same scale as the real single-branch clones. Its object set was also only modestly larger than single-branch main: about 23k objects in the simulation versus about 20k objects in the real single-branch clone.

Current normal GitHub SSH clones fetched about 1.25 GiB and 692,924 packed objects in both runs. Removing gh-pages appears to remove nearly all of that excess.

This improvement should not depend on GitHub pruning orphan blobs. Once gh-pages is no longer an advertised branch ref, its old objects should stop being part of normal clone negotiation, even if GitHub still stores them internally for a while.

[rwgk]

Entirely designed, run, and reported by GTP-5.5

---

Summary

I ran a local-only experiment to estimate the effect of compacting gh-pages to a single snapshot commit while keeping it as a normal branch.

In this experiment, compacting gh-pages reduced the simulated all-refs clone payload from about 1.2G to about 96M. Removing gh-pages entirely reduced it to about 37M.

Local file:// clone timings:

Scenario	Branch heads	.git size after clone	Git pack size	Packed objects	Local file:// clone time
Current gh-pages history	34	1.2G	1.14 GiB	692,960	61.80 s
No gh-pages branch	33	37M	36.49 MiB	23,161	2.29 s
Compacted gh-pages snapshot	34	96M	94.75 MiB	49,534	7.37 s

So compacting gh-pages would not get us all the way to the no-gh-pages case, but it appears to remove most of the current clone weight while preserving a normal gh-pages branch.

Method

This was a local simulation. I did not modify upstream.

Experiment directory:

/tmp/cuda-python-gh-pages-compact-exp-1781152054

I first refreshed local upstream refs:

git -C /wrk/forked/cuda-python fetch upstream --prune --tags

At that point, the fetch updated:

gh-pages: 884661eb81..59a5fa2842
main:     50ed678613..ece27e44d4

I then created a bare local repository representing the current upstream branch and tag namespace:

git init --bare "$base/with-current-gh-pages.git"
git -C "$base/with-current-gh-pages.git" fetch --quiet /wrk/forked/cuda-python \
  '+refs/remotes/upstream/*:refs/heads/*' \
  '+refs/tags/*:refs/tags/*'
git -C "$base/with-current-gh-pages.git" update-ref -d refs/heads/HEAD 2>/dev/null || true
git -C "$base/with-current-gh-pages.git" symbolic-ref HEAD refs/heads/main
git -C "$base/with-current-gh-pages.git" gc --prune=now

For the no-gh-pages control, I copied that bare repo, deleted only refs/heads/gh-pages, expired reflogs, and garbage-collected:

cp -a "$base/with-current-gh-pages.git" "$base/without-gh-pages.git"
git -C "$base/without-gh-pages.git" update-ref -d refs/heads/gh-pages
git -C "$base/without-gh-pages.git" reflog expire --expire=now --all
git -C "$base/without-gh-pages.git" gc --prune=now

For the compacted gh-pages case, I copied the current bare repo, created a new orphan commit whose tree is exactly the current gh-pages tree, repointed refs/heads/gh-pages to that commit, expired reflogs, and garbage-collected:

cp -a "$base/with-current-gh-pages.git" "$base/with-compacted-gh-pages.git"
tree="$(git -C "$base/with-compacted-gh-pages.git" rev-parse refs/heads/gh-pages^{tree})"
commit="$(
  printf 'Compacted gh-pages snapshot for clone-size experiment\n' |
    GIT_AUTHOR_NAME='clone-size experiment' \
    GIT_AUTHOR_EMAIL='clone-size-experiment@example.invalid' \
    GIT_AUTHOR_DATE='2026-06-10T21:30:00-07:00' \
    GIT_COMMITTER_NAME='clone-size experiment' \
    GIT_COMMITTER_EMAIL='clone-size-experiment@example.invalid' \
    GIT_COMMITTER_DATE='2026-06-10T21:30:00-07:00' \
    git -C "$base/with-compacted-gh-pages.git" commit-tree "$tree"
)"
git -C "$base/with-compacted-gh-pages.git" update-ref refs/heads/gh-pages "$commit"
git -C "$base/with-compacted-gh-pages.git" reflog expire --expire=now --all
git -C "$base/with-compacted-gh-pages.git" gc --prune=now

The compacted snapshot used:

gh-pages tree:             62091bfd506abd02531007c2db8b6d96d451e77f
compacted gh-pages commit: 66d61c70e53341a9410205e4fac11e0cad663bae

Finally, I cloned each bare repo via file://:

/usr/bin/time -f 'clone-current elapsed: %e s' \
  git clone --quiet "file://$base/with-current-gh-pages.git" "$base/clone-current"

/usr/bin/time -f 'clone-without-gh-pages elapsed: %e s' \
  git clone --quiet "file://$base/without-gh-pages.git" "$base/clone-without-gh-pages"

/usr/bin/time -f 'clone-compacted-gh-pages elapsed: %e s' \
  git clone --quiet "file://$base/with-compacted-gh-pages.git" "$base/clone-compacted-gh-pages"

Using file:// is important here: cloning from a plain local path can hardlink or copy objects and distort the measurement, while file:// forces Git to use normal fetch/pack negotiation.

Bare Repository Measurements

Scenario	Branch heads	Tags	rev-list --disk-usage --objects --all	Object count	count-objects pack size	du -sh
Current gh-pages history	34	85	1,211,852,901 bytes	692,960	1.14 GiB	1.2G
No gh-pages branch	33	85	37,610,382 bytes	23,161	36.49 MiB	38M
Compacted gh-pages snapshot	34	85	97,959,486 bytes	49,534	94.75 MiB	96M

Compared with the current repository shape, compacting gh-pages reduced the bare-repo rev-list --disk-usage estimate by:

1,113,893,415 bytes

Compared with deleting gh-pages entirely, keeping a compacted gh-pages snapshot added:

60,349,104 bytes
26,373 objects

Clone Measurements

Scenario	Local file:// clone time	.git du -sb	.git du -sh	Git pack size	Packed objects
Current gh-pages history	61.80 s	1,234,157,728 bytes	1.2G	1.14 GiB	692,960
No gh-pages branch	2.29 s	38,481,551 bytes	37M	36.49 MiB	23,161
Compacted gh-pages snapshot	7.37 s	99,674,689 bytes	96M	94.75 MiB	49,534

Compared with the current repository shape, compacting gh-pages reduced the cloned .git filesystem size by:

1,134,483,039 bytes

Compared with deleting gh-pages entirely, keeping a compacted gh-pages snapshot added:

61,193,138 bytes
5.08 seconds in this local file:// clone experiment

Interpretation

This experiment suggests that compacting gh-pages to a single snapshot commit would achieve most of the clone-size improvement while preserving the branch.

The result is not as small as deleting gh-pages entirely, because the current generated docs snapshot would still be reachable from refs/heads/gh-pages. But the retained snapshot is much smaller than the accumulated gh-pages history.

This experiment does not directly measure GitHub SSH clone time. It measures the object-set and local fetch/pack behavior after compacting the branch. For real GitHub SSH clone estimates, this should be interpreted together with the earlier real-world measurements:

• Current normal GitHub SSH clone: about 90-125 seconds, about 1.25 GiB received.
• Current --single-branch --branch main GitHub SSH clone: about 4.5-5.8 seconds, about 36 MiB received.

The compacted case is about 95 MiB by Git pack size, so I would expect real-world GitHub SSH clone time to land above the current single-branch main clone, but far below the current normal full clone.

This also should not depend on GitHub pruning orphan blobs for normal clone behavior. Once old gh-pages history is no longer reachable from an advertised branch ref, it should stop participating in normal clone negotiation, even if GitHub retains unreachable objects internally for a while.

[rwgk]

GPT-5.5 Analysis

---

Options for durable clone-size improvement

There are two broad ways to get a durable clone-size improvement from the gh-pages side:

1. Move gh-pages content to another repository.
2. Keep gh-pages in NVIDIA/cuda-python, but periodically compact its history.

Move gh-pages To Another Repo

I think we should discard this option for now.

The main reason is that we almost certainly need to preserve the existing public URL exactly:

https://nvidia.github.io/cuda-python/...

Moving the Pages content to another repository would normally change the GitHub Pages project URL. That would introduce a much larger compatibility problem than the clone-size problem we are trying to solve. Existing links to release docs, latest docs, PR previews, and coverage pages may be embedded in issues, PRs, release notes, external docs, bookmarks, and downstream references.

If exact URL preservation is required, moving to another repository becomes much less attractive. It would likely require additional GitHub Pages/custom-domain/routing machinery, and that complexity does not seem justified when there is a simpler path that keeps the current URL.

Periodically Compact Upstream gh-pages

This seems like the more realistic option.

The experiment above suggests that compacting gh-pages to a single snapshot commit would preserve the current branch and URL structure while removing most of the clone-size overhead:

Scenario	.git size after local file:// clone	Git pack size	Packed objects	Local file:// clone time
Current gh-pages history	1.2G	1.14 GiB	692,960	61.80 s
No gh-pages branch	37M	36.49 MiB	23,161	2.29 s
Compacted gh-pages snapshot	96M	94.75 MiB	49,534	7.37 s

So compacting does not get all the way to the no-gh-pages case, because the current generated docs snapshot still has to remain reachable. But it appears to remove most of the accumulated history cost while preserving the branch.

This could be done periodically, for example after releases or whenever the normal clone cost grows beyond a threshold. The required operational pieces would be:

• A documented procedure or workflow that replaces gh-pages history with one orphan snapshot commit.
• A pre-cleanup backup/archive point, likely a tag or branch in a maintainer fork, in case rollback is needed.
• Post-cleanup smoke tests for the public docs, release docs, PR previews, and coverage pages.
• Agreement that gh-pages history is not relied on for audit/debugging and can be rewritten.

CCCL Reference Point

CCCL appears to have gone further than periodic manual compaction. Its current docs deployment workflow keeps gh-pages compact by construction.

In NVIDIA/cccl, the current docs deploy workflow uses:

uses: peaceiris/actions-gh-pages@v3
with:
  github_token: ${{ secrets.GITHUB_TOKEN }}
  publish_dir: ./deploy
  publish_branch: gh-pages
  keep_files: true
  force_orphan: true

The current NVIDIA/cccl gh-pages branch has exactly one commit, which is consistent with force_orphan: true producing a fresh orphan deployment commit.

Relevant CCCL PRs:

• Update docs build to deploy from gh-pages branch to docs/ directory with preserved branch history cccl#5605 moved docs deployment to branch-based gh-pages deployment.
• Refactor docs deployment to support future versioned docs cccl#5787 introduced the current versioned docs deployment flow with keep_files + force_orphan.
• Reduce deployed docs size by not shipping .doctrees and removing exhaustive API listing from table of contents cccl#7696 separately reduced the size of the deployed docs snapshot itself.

This suggests a durable model: instead of treating compaction as a rare manual cleanup, make the normal docs deployment path maintain a compact gh-pages branch.

CUDA Python Difference

CUDA Python has more gh-pages writers than CCCL appears to have in its current setup.

Today, CUDA Python writes to gh-pages from at least:

• Release/latest docs deployment under docs/.
• PR preview deployment under docs/pr-preview/pr-N/.
• PR preview cleanup.
• Coverage report deployment under docs/coverage/.

That makes a direct CCCL-style switch less trivial. If multiple workflows independently write to gh-pages, then any workflow that force-orphans the branch needs to preserve the other published subtrees correctly, and concurrent writes become a real operational concern.

The key question is whether CUDA Python can safely adopt a single compacting deployment model, or whether we first need to consolidate the gh-pages writers. The highest-confidence version would have one deployment/maintenance path responsible for materializing the complete desired Pages tree, then publishing it as a compact orphan commit. A lower-effort version might keep the current writers but add a periodic compacting maintenance workflow after confirming that the existing deploy actions tolerate the rewritten history.

My current preference would be:

1. Keep gh-pages in NVIDIA/cuda-python to preserve https://nvidia.github.io/cuda-python/....
2. Treat compacting gh-pages as the realistic improvement path.
3. Learn from CCCL's keep_files: true + force_orphan: true deployment model.
4. Before implementing, audit whether CUDA Python's multiple gh-pages writers can be made compatible with that model, or whether they should be consolidated first.

[rwgk]

Ralf in the words of GPT-5.5

---

@jrhemstad We are looking at ways to reduce the clone-size impact of gh-pages in NVIDIA/cuda-python while preserving https://nvidia.github.io/cuda-python/....

I noticed that NVIDIA/cccl now deploys docs to gh-pages with peaceiris/actions-gh-pages using both keep_files: true and force_orphan: true, and that the current CCCL gh-pages branch has exactly one commit.

Did that setup cause any operational issues in CCCL, especially around GitHub Pages serving, concurrent docs deploys, or preserving existing published subdirectories? Any lessons learned before we consider a similar compacting model for CUDA Python?