-
Notifications
You must be signed in to change notification settings - Fork 4
Pr/upload script parallel #5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
asarkar157
wants to merge
5
commits into
stackgenhq:main
Choose a base branch
from
asarkar157:pr/upload-script-parallel
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
45f1ede
feat: parallelize module uploads with retries and progress tracking
asarkar157 0efb81d
docs: add README for tools directory
asarkar157 cfcd153
docs: update root README with new upload script flags and behavior
asarkar157 12b6319
add missing bulk-tag-modules.sh script
asarkar157 b0368e5
Merge remote-tracking branch 'upstream/main' into pr/upload-script-pa…
asarkar157 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,168 @@ | ||
| # Tools | ||
|
|
||
| Helper scripts for managing and publishing StackGen discovery modules. | ||
|
|
||
| --- | ||
|
|
||
| ## upload_stackgen_modules.sh | ||
|
|
||
| Bulk-uploads Terraform module directories to the StackGen custom module registry using the `stackgen` CLI. Supports parallel execution for fast uploads of large module libraries. | ||
|
|
||
| ### Prerequisites | ||
|
|
||
| - **`stackgen` CLI** installed and available on `$PATH` | ||
| - A valid **StackGen API token** (passed via `--token`) | ||
| - Modules organized under `aws/`, `azurerm/`, and/or `gcp/` provider directories | ||
|
|
||
| ### Usage | ||
|
|
||
| ```bash | ||
| ./tools/upload_stackgen_modules.sh --token <token> [OPTIONS] | ||
| ``` | ||
|
|
||
| ### Flags | ||
|
|
||
| | Flag | Required | Default | Description | | ||
| |------|----------|---------|-------------| | ||
| | `--token <token>` | **Yes** | — | StackGen API token for authentication | | ||
| | `--url <url>` | No | CLI default | StackGen instance URL (e.g., `https://seti.cloud.stackgen.com`) | | ||
| | `--project <id>` | No | — | Project ID for auth context (does **not** scope module visibility — modules are always org-wide) | | ||
| | `--provider <name>` | No | All providers | Filter to a single provider: `aws`, `azurerm`, or `gcp` | | ||
| | `--templates <list>` | No | All modules | Comma-separated list of module names to upload (e.g., `aws_s3_bucket,aws_iam_role`) | | ||
| | `--repo-url <url>` | No | — | Git repository URL for StackGen to reference (e.g., `https://github.com/org/repo`) | | ||
| | `--branch <branch>` | No | — | Git branch to reference. Mutually exclusive with `--tag`. Requires `--repo-url` | | ||
| | `--tag <tag>` | No | — | Git tag to reference. Mutually exclusive with `--branch`. Requires `--repo-url` | | ||
| | `--version <ver>` | No | `1.0` (CLI default) | Module version string | | ||
| | `--overwrite-version` | No | `false` | Overwrite an existing version instead of skipping | | ||
| | `--parallel <N>` | No | `10` | Number of concurrent uploads | | ||
|
|
||
| ### Examples | ||
|
|
||
| #### Upload all modules (all providers) | ||
|
|
||
| ```bash | ||
| ./tools/upload_stackgen_modules.sh \ | ||
| --token "$STACKGEN_TOKEN" \ | ||
| --url "https://seti.cloud.stackgen.com" | ||
| ``` | ||
|
|
||
| #### Upload only AWS modules | ||
|
|
||
| ```bash | ||
| ./tools/upload_stackgen_modules.sh \ | ||
| --token "$STACKGEN_TOKEN" \ | ||
| --url "https://seti.cloud.stackgen.com" \ | ||
| --provider aws | ||
| ``` | ||
|
|
||
| #### Upload specific modules by name | ||
|
|
||
| ```bash | ||
| ./tools/upload_stackgen_modules.sh \ | ||
| --token "$STACKGEN_TOKEN" \ | ||
| --url "https://seti.cloud.stackgen.com" \ | ||
| --templates "aws_s3_bucket,aws_iam_role,aws_lambda_function" | ||
| ``` | ||
|
|
||
| #### Upload from a specific branch with repo reference | ||
|
|
||
| ```bash | ||
| ./tools/upload_stackgen_modules.sh \ | ||
| --token "$STACKGEN_TOKEN" \ | ||
| --url "https://seti.cloud.stackgen.com" \ | ||
| --repo-url "https://github.com/stackgenhq/discovery-modules" \ | ||
| --branch "main" | ||
| ``` | ||
|
|
||
| #### Overwrite existing module versions | ||
|
|
||
| ```bash | ||
| ./tools/upload_stackgen_modules.sh \ | ||
| --token "$STACKGEN_TOKEN" \ | ||
| --url "https://seti.cloud.stackgen.com" \ | ||
| --overwrite-version | ||
| ``` | ||
|
|
||
| #### Bump to a new version | ||
|
|
||
| ```bash | ||
| ./tools/upload_stackgen_modules.sh \ | ||
| --token "$STACKGEN_TOKEN" \ | ||
| --url "https://seti.cloud.stackgen.com" \ | ||
| --version "2.0" \ | ||
| --provider aws | ||
| ``` | ||
|
|
||
| #### Increase parallelism for faster uploads | ||
|
|
||
| ```bash | ||
| ./tools/upload_stackgen_modules.sh \ | ||
| --token "$STACKGEN_TOKEN" \ | ||
| --url "https://seti.cloud.stackgen.com" \ | ||
| --parallel 20 | ||
| ``` | ||
|
|
||
| ### How it works | ||
|
|
||
| 1. **Module discovery** — Scans `aws/`, `azurerm/`, and `gcp/` directories (or a single provider if `--provider` is set) for subdirectories. Each subdirectory is treated as one module. If `--templates` is provided, only the named modules are uploaded. | ||
|
|
||
| 2. **Parallel dispatch** — Modules are uploaded concurrently using `xargs -P`. The default concurrency is 10, tunable with `--parallel`. | ||
|
|
||
| 3. **Retry with backoff** — Each module upload is retried up to 3 times with exponential backoff (1s → 2s → 4s) for transient API failures. "Version already exists" errors are detected immediately and skipped without retrying. | ||
|
|
||
| 4. **Provider mapping** — The provider name is derived from the parent directory (`aws` → `aws`, `gcp` → `gcp`). The `azurerm` directory is mapped to `azure` since the StackGen CLI uses `azure` as the provider name. | ||
|
|
||
| 5. **Progress tracking** — A thread-safe atomic counter (using `mkdir`-based spinlocking) provides real-time `[N/Total]` progress output, even across parallel subshells. Works on both macOS (BSD) and Linux (GNU). | ||
|
|
||
| 6. **Summary report** — After all uploads complete, a structured summary shows succeeded, skipped (version exists), and failed counts. Failed module names and error messages are listed. The script exits non-zero only if there were failures. | ||
|
|
||
| ### Output example | ||
|
|
||
| ``` | ||
| Uploading 817 module(s) (parallelism: 10)... | ||
| [1/817] ✓ aws/aws_s3_bucket | ||
| [2/817] ✓ aws/aws_iam_role | ||
| [3/817] ⊘ aws/aws_lambda_function (skipped: version exists) | ||
| [4/817] ✗ aws/aws_bad_module (FAILED after 3 attempts) | ||
| ... | ||
|
|
||
| ===== Upload Summary ===== | ||
| Succeeded: 815 | ||
| Skipped: 1 (version already exists) | ||
| Failed: 1 | ||
| Total: 817 | ||
|
|
||
| ===== Failed Modules ===== | ||
| aws/aws_bad_module: Error: invalid module configuration | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## bulk-tag-modules.sh | ||
|
|
||
| Creates `v1.0.0` Git tags for all module subdirectories. These tags are required by the `module-backfill.yml` GitHub Actions workflow to discover and upload modules. | ||
|
|
||
| ### Usage | ||
|
|
||
| ```bash | ||
| ./tools/bulk-tag-modules.sh # Dry run — shows what would be tagged | ||
| ./tools/bulk-tag-modules.sh --apply # Creates tags locally | ||
| ./tools/bulk-tag-modules.sh --apply --push # Creates tags and pushes to remote | ||
| ``` | ||
|
|
||
| ### Tag format | ||
|
|
||
| ``` | ||
| <module-subdirectory-name>-v1.0.0 | ||
| ``` | ||
|
|
||
| Examples: `aws_s3_bucket-v1.0.0`, `azurerm_resource_group-v1.0.0`, `google_compute_instance-v1.0.0` | ||
|
|
||
| --- | ||
|
|
||
| ## Other files | ||
|
|
||
| | File | Description | | ||
| |------|-------------| | ||
| | `stackgen_yaml_schema.json` | JSON Schema for `.stackgen/stackgen.yaml` files. Use for editor validation and CI linting. | | ||
| | `dummy.yaml` | Sample fixture input for development and testing. | | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,132 @@ | ||
| #!/bin/bash | ||
| # bulk-tag-modules.sh | ||
| # | ||
| # Creates v1.0.0 tags for all module subdirectories in aws/, azurerm/, and gcp/. | ||
| # These tags are required for the module-backfill.yml workflow to discover and upload modules. | ||
| # | ||
| # Usage: | ||
| # ./tools/bulk-tag-modules.sh # Dry run (default) — shows what would be tagged | ||
| # ./tools/bulk-tag-modules.sh --apply # Creates tags locally | ||
| # ./tools/bulk-tag-modules.sh --apply --push # Creates tags and pushes to remote | ||
| # | ||
| # Tag format: <module-subdirectory-name>-v1.0.0 | ||
| # Examples: | ||
| # aws_s3_bucket-v1.0.0 | ||
| # azurerm_resource_group-v1.0.0 | ||
| # google_compute_instance-v1.0.0 | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| # Parse flags | ||
| APPLY=false | ||
| PUSH=false | ||
| for arg in "$@"; do | ||
| case "$arg" in | ||
| --apply) APPLY=true ;; | ||
| --push) PUSH=true ;; | ||
| --help|-h) | ||
| echo "Usage: $0 [--apply] [--push]" | ||
| echo "" | ||
| echo " --apply Create tags locally (without this flag, dry run only)" | ||
| echo " --push Push tags to remote (implies --apply)" | ||
| exit 0 | ||
| ;; | ||
| esac | ||
| done | ||
|
|
||
| if [ "$PUSH" = true ]; then | ||
| APPLY=true | ||
| fi | ||
|
|
||
| # Ensure we're in the repo root | ||
| REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || echo "")" | ||
| if [ -z "$REPO_ROOT" ]; then | ||
| echo "❌ Not in a git repository" | ||
| exit 1 | ||
| fi | ||
| cd "$REPO_ROOT" | ||
|
|
||
| VERSION="v1.0.0" | ||
| CREATED=0 | ||
| SKIPPED=0 | ||
| TOTAL=0 | ||
| TAGS_TO_PUSH=() | ||
|
|
||
| echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" | ||
| echo "📦 BULK MODULE TAGGER" | ||
| echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" | ||
| echo "" | ||
| if [ "$APPLY" = false ]; then | ||
| echo "🔍 DRY RUN MODE — no tags will be created" | ||
| echo " Run with --apply to create tags" | ||
| echo "" | ||
| fi | ||
|
|
||
| for PROVIDER_DIR in aws azurerm gcp; do | ||
| if [ ! -d "$PROVIDER_DIR" ]; then | ||
| echo "⚠️ Directory not found: $PROVIDER_DIR — skipping" | ||
| continue | ||
| fi | ||
|
|
||
| echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" | ||
| echo "📂 Processing: $PROVIDER_DIR/" | ||
| echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" | ||
|
|
||
| DIR_COUNT=0 | ||
| for MODULE_DIR in "$PROVIDER_DIR"/*/; do | ||
| # Strip trailing slash to get directory name | ||
| MODULE_DIR="${MODULE_DIR%/}" | ||
| MODULE_NAME="$(basename "$MODULE_DIR")" | ||
| TAG_NAME="${MODULE_NAME}-${VERSION}" | ||
|
|
||
| TOTAL=$((TOTAL + 1)) | ||
| DIR_COUNT=$((DIR_COUNT + 1)) | ||
|
|
||
| # Check if tag already exists | ||
| if git rev-parse "$TAG_NAME" >/dev/null 2>&1; then | ||
| echo " ⏭ $TAG_NAME (already exists)" | ||
| SKIPPED=$((SKIPPED + 1)) | ||
| continue | ||
| fi | ||
|
|
||
| if [ "$APPLY" = true ]; then | ||
| git tag "$TAG_NAME" | ||
| echo " ✅ $TAG_NAME" | ||
| TAGS_TO_PUSH+=("$TAG_NAME") | ||
| else | ||
| echo " 🔍 $TAG_NAME (would create)" | ||
| fi | ||
| CREATED=$((CREATED + 1)) | ||
| done | ||
|
|
||
| echo " → $DIR_COUNT modules in $PROVIDER_DIR/" | ||
| echo "" | ||
| done | ||
|
|
||
| # Summary | ||
| echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" | ||
| echo "📊 SUMMARY" | ||
| echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" | ||
| echo " Total modules: $TOTAL" | ||
| echo " Tags to create: $CREATED" | ||
| echo " Already tagged: $SKIPPED" | ||
| if [ "$APPLY" = false ]; then | ||
| echo " Mode: DRY RUN" | ||
| echo "" | ||
| echo "👉 Run with --apply to create tags locally" | ||
| echo "👉 Run with --apply --push to create and push tags" | ||
| else | ||
| echo " Mode: APPLIED" | ||
| fi | ||
| echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" | ||
|
|
||
| # Push if requested | ||
| if [ "$PUSH" = true ] && [ ${#TAGS_TO_PUSH[@]} -gt 0 ]; then | ||
| echo "" | ||
| echo "📤 Pushing ${#TAGS_TO_PUSH[@]} tags to origin..." | ||
| git push origin "${TAGS_TO_PUSH[@]}" | ||
| echo "✅ All tags pushed" | ||
| elif [ "$PUSH" = true ] && [ ${#TAGS_TO_PUSH[@]} -eq 0 ]; then | ||
| echo "" | ||
| echo "ℹ️ No new tags to push" | ||
| fi |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.