Skip to content

docs: add WorkQueueOps and BatchOps design pattern pages#25178

Merged
mnkiefer merged 6 commits intomainfrom
copilot/create-workqueue-and-batch-ops-docs
Apr 8, 2026
Merged

docs: add WorkQueueOps and BatchOps design pattern pages#25178
mnkiefer merged 6 commits intomainfrom
copilot/create-workqueue-and-batch-ops-docs

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 7, 2026

Adds two missing pattern pages covering work-queue and batch-processing — common agentic use cases with no prior documentation coverage.

New pages

patterns/workqueue-ops.md

Four queue strategies, each with a complete frontmatter + prompt example:

  • Issue checklist — checkboxes as a lightweight human-visible queue
  • Sub-issues — one sub-issue per item; open = pending, closed = done
  • Cache-memory JSON — resumable queue surviving multi-day/multi-run horizons
  • Discussion-based — unresolved replies as pending work items

Includes a concurrency/idempotency reference table and guidance on preventing race conditions.

patterns/batch-ops.md

Four batch strategies with complete examples:

  • Chunked processing — deterministic pagination via GITHUB_RUN_NUMBER
  • Matrix fan-out — modulo-sharded parallel workers (fail-fast: false)
  • Rate-limit-aware — sub-batch processing with explicit pauses and 429 retry handling
  • Result aggregation — consolidating multi-run results into a summary issue via cache-memory

Includes a sequential-vs-batch decision table, retry/failure-isolation patterns, and a real-world label migration example (100+ issues).

Config

  • Added both pages to the Design Patterns sidebar (alphabetical order)
  • Added unhyphenated redirect slugs (/patterns/batchops/, /patterns/workqueueops/) matching existing conventions

Copilot AI changed the title [WIP] Create WorkQueueOps and BatchOps pattern documentation pages docs: add WorkQueueOps and BatchOps design pattern pages Apr 7, 2026
Copilot AI requested a review from pelikhan April 7, 2026 22:31
@github-actions github-actions bot added the spam label Apr 7, 2026
@github github deleted a comment from g66b166-hub Apr 7, 2026
@github github deleted a comment from g66b166-hub Apr 7, 2026
@pelikhan
Copy link
Copy Markdown
Collaborator

pelikhan commented Apr 7, 2026

@copilot use the unbloat agent to review the changes and clean up

Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 7, 2026

@copilot use the unbloat agent to review the changes and clean up

Applied the unbloat principles from the unbloat-docs workflow to both pages (commit 89ae343):

  • Removed the "When to Use" bullet list from workqueue-ops.md (folded into the intro sentence)
  • Replaced **Best for:** / **Caveats:** bold-header pairs with inline prose in each strategy section
  • Removed redundant ## Instructions sub-headers inside workflow prompts, and flattened a./b./c. sub-steps into flat numbered lists
  • Removed the standalone idempotency code block (redundant with examples already showing cancel-in-progress: false)
  • In batch-ops.md: replaced the full retry workflow example block with a short prose paragraph
  • Removed ## Shard Assignment and ## Aggregate Data sub-sections (folded into prompt prose)
  • Condensed the label migration prompt

Result: 248 → 187 lines (workqueue-ops.md, −25%) and 351 → 266 lines (batch-ops.md, −24%), preserving all essential information.

@pelikhan
Copy link
Copy Markdown
Collaborator

pelikhan commented Apr 7, 2026

@mnkiefer can you take a peek?

@mnkiefer mnkiefer marked this pull request as ready for review April 8, 2026 06:06
Copilot AI review requested due to automatic review settings April 8, 2026 06:06
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds missing documentation for two common agentic workflow design patterns—queue-based processing and batch processing—and wires them into the docs navigation and redirects.

Changes:

  • Added new pattern pages: WorkQueueOps and BatchOps, each with multiple strategy examples.
  • Added unhyphenated redirect slugs for both patterns.
  • Added both pages to the Design Patterns sidebar.
Show a summary per file
File Description
docs/src/content/docs/patterns/workqueue-ops.md New WorkQueueOps pattern page with queue strategy examples and concurrency/idempotency guidance
docs/src/content/docs/patterns/batch-ops.md New BatchOps pattern page with chunking, fan-out, rate-limit-aware batching, and aggregation examples
docs/astro.config.mjs Adds redirects for unhyphenated slugs and adds new pages to the Design Patterns sidebar

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comments suppressed due to low confidence (6)

docs/src/content/docs/patterns/workqueue-ops.md:45

  • This example instructs the agent to close the queue issue when all items are checked, but the configured safe outputs do not allow closing or status updates. Add close-issue (or enable update-issue.status) so the workflow can actually complete the queue by closing the issue.
1. Read issue #${{ inputs.queue_issue }} and find all unchecked items (`- [ ]`).
2. For each unchecked item (at most 10 per run): perform the required work, then edit the issue body to change `- [ ]` to `- [x]`.
3. Add a comment summarizing what was completed and what remains.
4. If all items are checked, close the issue with a summary comment.
**docs/src/content/docs/patterns/workqueue-ops.md:154**
* This scheduled Discussion queue processor needs `add-comment` configured for discussions (and typically `target: "*"`), otherwise the agent can't post replies to the discovered discussion thread. Update `safe-outputs.add-comment` to set `discussion: true` (and appropriate targeting) to match the described behavior.

safe-outputs:
add-comment:
max: 5
create-discussion:
title-prefix: "[queue-log] "
category: "General"

**docs/src/content/docs/patterns/batch-ops.md:85**
* This example uses `jobs.<job>.strategy.matrix`, but gh-aw custom `jobs:` explicitly do not support `strategy`/matrix (see docs/src/content/docs/reference/frontmatter.md:686). As written, the workflow won't compile—consider rewriting this section to use a supported fan-out approach (e.g., sharded runs via inputs/schedules) or describe a plain GitHub Actions `.yml` matrix that invokes compiled gh-aw workflows.

jobs:
batch:
strategy:
matrix:
shard: [0, 1, 2, 3]
fail-fast: false # Continue other shards even if one fails

**docs/src/content/docs/patterns/batch-ops.md:93**
* The prompt describes posting references/comments when duplicates are found, but `safe-outputs` here only allows `add-labels`. Either add the needed safe output(s) (e.g., `add-comment`) or adjust the instructions so they only perform operations that are permitted by the configured safe outputs.

safe-outputs:
add-labels:
allowed: [reviewed, duplicate, wontfix]
max: 50

**docs/src/content/docs/patterns/batch-ops.md:186**
* In bash, if the directory exists but contains no `*.json` files, `jq -s ... "$RESULTS_DIR"/*.json` will fail because the glob won't expand. Add a guard (e.g., check for matching files) before invoking `jq` so the aggregation step is robust on first run / empty results.
  # Aggregate results from all result files written by previous batch runs
  RESULTS_DIR="/tmp/gh-aw/cache-memory/batch-results"
  if [ -d "$RESULTS_DIR" ]; then
    jq -s '
      {
        total_processed: (map(.processed) | add // 0),
        total_failed: (map(.failed) | add // 0),
        total_skipped: (map(.skipped) | add // 0),
        runs: length,
        errors: (map(.errors // []) | add // [])
      }
    ' "$RESULTS_DIR"/*.json > /tmp/gh-aw/cache-memory/aggregate.json
    cat /tmp/gh-aw/cache-memory/aggregate.json
**docs/src/content/docs/patterns/batch-ops.md:170**
* `safe-outputs` does not appear to support an `edit-issue` handler; the documented handler is `update-issue`. Also, the prompt includes creating sub-issues for failed items, but `safe-outputs` does not enable `create-issue`/`link-sub-issue`. Update the safe-outputs configuration to include the correct issue-update handler and the additional handlers required by the described behavior.

safe-outputs:
add-comment:
max: 1
edit-issue: true

</details>


- **Files reviewed:** 3/3 changed files
- **Comments generated:** 2


@mnkiefer
Copy link
Copy Markdown
Contributor

mnkiefer commented Apr 8, 2026

@mnkiefer can you take a peek?

@pelikhan LGTM! :shipit:

Copilot AI requested a review from mnkiefer April 8, 2026 07:04
@mnkiefer mnkiefer merged commit a5f2a40 into main Apr 8, 2026
@mnkiefer mnkiefer deleted the copilot/create-workqueue-and-batch-ops-docs branch April 8, 2026 07:51
github-actions bot added a commit that referenced this pull request Apr 8, 2026
Add two new operational pattern entries to the glossary based on
pattern documentation added in #25178:

- BatchOps: chunked pagination, matrix fan-out, rate-limit-aware batching
- WorkQueueOps: durable queue processing using issue checklists,
  sub-issues, cache-memory, or GitHub Discussions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[plan] Create WorkQueueOps and BatchOps pattern documentation pages

4 participants