Add buildroot.py to mirror sources.buildroot.net via .mk parsing by yaoge123 · Pull Request #204 · tuna/tunasync-scripts

yaoge123 · 2026-05-24T09:21:16Z

Summary

Add buildroot.py to mirror sources.buildroot.net by extracting download URLs straight out of the buildroot source tree.

Why

sources.buildroot.net, the official Buildroot package source backup, has permanently disabled directory listing (Cloudflare 403 on every path that isn't a known file). Conventional approaches all fail:

Tool	Result
rsync	Upstream offers no rsync module
tsumugu / wget --mirror	Cannot enumerate, every directory request 403
recursive HTML scraping	No HTML index exists

The Buildroot maintainers have stated on the mailing list that they will not enable rsync. So a mirror has to derive the file list from somewhere else.

How it works

Shallow-clone the buildroot master branch
Parse boot/*.mk, linux/*.mk, and package/*/*.mk for VERSION, SITE, SOURCE
Expand version variables and the upstream macros ($(call github,...), $(call gitlab,...), $(call sourceforge,...))
For each derived URL, compare against the local mirror by Content-Length
Download new/changed files atomically (.tmp + rename)
Optionally clean up stale files up to TUNASYNC_BUILDROOT_MAXDELETE

Notes

Existing local data is preserved by default
GitHub API rate-limit is avoided by using archive tarball URLs, not the API
Version-variable substitution is heuristic; a small VERSION_OVERRIDES dict in the script handles known awkward cases

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a self-contained Python sync script to mirror Buildroot package sources without relying on upstream directory listing, by extracting download URLs from the Buildroot git tree and incrementally downloading missing artifacts.

Changes:

Introduces buildroot.py to clone/update Buildroot git, parse .mk metadata, and download package sources incrementally.
Adds basic logging + state tracking to skip runs when git HEAD hasn’t changed.
Implements atomic downloads via .tmp → rename with fallback to sources.buildroot.net.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

yaoge123 · 2026-05-24T09:48:02Z

+    return f"https://github.com/{repo_org}/{repo_name}/archive/{version}/{repo_name}-{version}.tar.gz"
+
+
+def emit_gitlab(repo_org: str, repo_name: str, version: str) -> str:
+    return f"https://gitlab.com/{repo_org}/{repo_name}/-/archive/{version}/{repo_name}-{version}.tar.gz"


Fixed. emit_github / emit_gitlab now return a URL whose path already ends with the tarball filename, and sync_package no longer re-appends the local filename on top of those URLs (only when the configured SITE is a directory).

yaoge123 · 2026-05-24T09:48:04Z

+        full = url.rstrip("/") + "/" + local_file if url else None
+        if full:


Same as the line-151 reply: emit_github / emit_gitlab URLs are now treated as full file URLs in sync_package via url_has_filename().

yaoge123 · 2026-05-24T09:48:05Z

+    # Skip if local file exists and is non-empty
+    if dest.exists() and dest.stat().st_size > 0:
+        stats["skipped"] += 1
+        return True
+


Done. bump('total') is now invoked at the very top of sync_package, before any skip return, so the final summary covers every package candidate.

yaoge123 · 2026-05-24T09:48:07Z

+        if stats["total"] % 500 == 0:
+            log(f"  Progress: {stats['total']}/{len(packages)} — "


Same as the line-317 reply: total is now bumped up front.

yaoge123 · 2026-05-24T09:48:08Z

+def wget(url: str, dest: Path) -> bool:
+    """Download a file via wget, follow redirects."""
+    dest.parent.mkdir(parents=True, exist_ok=True)
+    tmp = dest.with_suffix(dest.suffix + ".tmp")
+    tmp.unlink(missing_ok=True)


wget() now removes the .tmp file in a finally block, so it is cleaned up on every code path (success, non-zero curl, exception, or empty body).

yaoge123 · 2026-05-24T09:48:13Z

+MAXDELETE     = int(os.environ.get("TUNASYNC_BUILDROOT_MAXDELETE", "10000"))
+JOBS          = int(os.environ.get("TUNASYNC_BUILDROOT_JOBS", "4"))
+DRYRUN        = os.environ.get("TUNASYNC_BUILDROOT_DRYRUN", "") in ("1", "true", "yes")


Updated: after reflection I scaled this back. JOBS now defaults to 1 (serial) to match what most repo scripts assume — the wget/git invocations underneath aren't friendly to concurrent invocations on the same upstream and the mirror was never meant to be aggressive. Operators who do want parallelism can still set TUNASYNC_BUILDROOT_JOBS=N. The ThreadPoolExecutor + stats_lock plumbing stays so the option is available. Adjusted in fb0bc70.

yaoge123 · 2026-05-24T09:48:15Z

+def clean_stale_files() -> None:
+    """Remove local files that are no longer in the current buildroot tree."""
+    # Build set of known directories from packages
+    known_names = set()
+    # We don't actually know what's "stale" without a full scan —
+    # so we skip this phase unless the user explicitly wants it.
+    # The buildroot mirror is append-only by design.
+    log("Cleanup: skipped (buildroot mirror is append-only by default)")
+


Updated: after reflection I scaled this back. JOBS now defaults to 1 (serial) to match what most repo scripts assume — the wget/git invocations underneath aren't friendly to concurrent invocations on the same upstream and the mirror was never meant to be aggressive. Operators who do want parallelism can still set TUNASYNC_BUILDROOT_JOBS=N. The ThreadPoolExecutor + stats_lock plumbing stays so the option is available. Adjusted in fb0bc70.

yaoge123 · 2026-05-24T09:48:18Z

+    log("Step 3: Downloading new/changed files...")
+    for pkg in packages:
+        sync_package(pkg)


Updated: after reflection I scaled this back. JOBS now defaults to 1 (serial) to match what most repo scripts assume — the wget/git invocations underneath aren't friendly to concurrent invocations on the same upstream and the mirror was never meant to be aggressive. Operators who do want parallelism can still set TUNASYNC_BUILDROOT_JOBS=N. The ThreadPoolExecutor + stats_lock plumbing stays so the option is available. Adjusted in fb0bc70.

yaoge123 · 2026-05-24T09:48:20Z

+def run(cmd: list, **kw) -> subprocess.CompletedProcess:
+    return subprocess.run(cmd, capture_output=True, text=True, timeout=180, **kw)


Done. git fetch, git reset --hard, and git clone all check returncode now and log stderr; if the tree cannot be fetched the script aborts with rc=1 instead of running on an empty package list.

yaoge123 · 2026-05-24T09:48:21Z

+            run(["git", "-C", str(BR_GIT_DIR), "fetch", "--depth=1", "origin", BR_BRANCH])
+            run(["git", "-C", str(BR_GIT_DIR), "reset", "--hard", f"origin/{BR_BRANCH}"])
+        else:
+            r = run(["git", "clone", "--depth=1", "--branch", BR_BRANCH, BR_GIT_URL, str(BR_GIT_DIR)])
+            if r.returncode != 0:
+                raise Exception("clone failed")


Same as the line-85 reply.

sources.buildroot.net (the official Buildroot backup site) has permanently disabled directory listing (Cloudflare 403). Neither rsync, tsumugu, nor wget -m can discover what files exist on the server. This script clones the buildroot git tree, extracts the download URL for every package directly from the .mk files, and downloads only new or changed files. Existing local data is preserved (no deletion unless explicitly enabled). Steps: 1. Shallow-clone buildroot master 2. Parse boot/*.mk, linux/*.mk, package/*/*.mk for VERSION, SITE, SOURCE 3. Expand version variables and macro calls (github, gitlab, sourceforge) 4. Compare each candidate URL against the local mirror 5. Download new/changed files atomically (.tmp -> rename) 6. Clean up stale files up to TUNASYNC_BUILDROOT_MAXDELETE GitHub API rate-limit is avoided by using archive tarball URLs. SourceForge redirects are followed by wget, not by the script.

Address review feedback: - emit_github / emit_gitlab now return URLs that already include the filename, and sync_package no longer re-appends the local filename on top of them, fixing broken URLs like '.../tar.gz/repo-1.2.3.tar.gz'. - bump('total') is called before any skip path, so progress counters and the final summary include skipped packages. - wget() now removes the .tmp file in a finally block on every code path, so failed downloads no longer leave .tmp files behind. - Variable assignment regex now matches Makefile ':=' immediate assignment, not just '=', '?=', '+='. - JOBS is now real concurrency via concurrent.futures.ThreadPoolExecutor, and stats updates go through a lock. - TUNASYNC_BUILDROOT_CLEANUP enables stale-file cleanup; deletions are refused when they would exceed TUNASYNC_BUILDROOT_MAXDELETE. - git fetch / reset / clone return codes are checked; if the git tree cannot be fetched the script aborts (return code 1) instead of silently running on an empty package list.

Per Copilot review feedback: parallel sync of buildroot can amplify upstream load when many tunasync mirrors deploy this script. Default to serial (JOBS=1) and let operators opt into concurrency via TUNASYNC_BUILDROOT_JOBS.

Copilot AI review requested due to automatic review settings May 24, 2026 09:21

Copilot AI reviewed May 24, 2026

View reviewed changes

yaoge123 force-pushed the add-buildroot-py branch from 8f2e0cb to ff1cbf5 Compare May 24, 2026 09:25

yaoge123 added 2 commits May 24, 2026 17:47

buildroot.py: default JOBS to 1 (serial) instead of 4

fb0bc70

Per Copilot review feedback: parallel sync of buildroot can amplify upstream load when many tunasync mirrors deploy this script. Default to serial (JOBS=1) and let operators opt into concurrency via TUNASYNC_BUILDROOT_JOBS.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add buildroot.py to mirror sources.buildroot.net via .mk parsing#204

Add buildroot.py to mirror sources.buildroot.net via .mk parsing#204
yaoge123 wants to merge 3 commits into
tuna:masterfrom
yaoge123:add-buildroot-py

yaoge123 commented May 24, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

yaoge123 May 24, 2026

Uh oh!

yaoge123 May 24, 2026

Uh oh!

yaoge123 May 24, 2026

Uh oh!

yaoge123 May 24, 2026

Uh oh!

yaoge123 May 24, 2026

Uh oh!

yaoge123 May 24, 2026 •

edited

Loading

Uh oh!

yaoge123 May 24, 2026 •

edited

Loading

Uh oh!

yaoge123 May 24, 2026 •

edited

Loading

Uh oh!

yaoge123 May 24, 2026

Uh oh!

yaoge123 May 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		full = url.rstrip("/") + "/" + local_file if url else None
		if full:

		if stats["total"] % 500 == 0:
		log(f" Progress: {stats['total']}/{len(packages)} — "

		def run(cmd: list, **kw) -> subprocess.CompletedProcess:
		return subprocess.run(cmd, capture_output=True, text=True, timeout=180, **kw)

Conversation

yaoge123 commented May 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

How it works

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

yaoge123 May 24, 2026

Choose a reason for hiding this comment

Uh oh!

yaoge123 May 24, 2026

Choose a reason for hiding this comment

Uh oh!

yaoge123 May 24, 2026

Choose a reason for hiding this comment

Uh oh!

yaoge123 May 24, 2026

Choose a reason for hiding this comment

Uh oh!

yaoge123 May 24, 2026

Choose a reason for hiding this comment

Uh oh!

yaoge123 May 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yaoge123 May 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yaoge123 May 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yaoge123 May 24, 2026

Choose a reason for hiding this comment

Uh oh!

yaoge123 May 24, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yaoge123 commented May 24, 2026 •

edited

Loading

yaoge123 May 24, 2026 •

edited

Loading

yaoge123 May 24, 2026 •

edited

Loading

yaoge123 May 24, 2026 •

edited

Loading