diff --git a/docs/case-studies/bounded-real-environment-pilot-protocol.md b/docs/case-studies/bounded-real-environment-pilot-protocol.md new file mode 100644 index 0000000..72db5a7 --- /dev/null +++ b/docs/case-studies/bounded-real-environment-pilot-protocol.md @@ -0,0 +1,227 @@ +# Bounded Real-Environment Pilot Protocol + +## Purpose + +This protocol gives external AWS IAM and cloud security reviewers a safe way to +test IAMScope on a bounded real environment. The goal is to judge whether the +approach generalizes beyond the controlled prod-like IAM sandbox without +turning the pilot into a production-readiness claim, a broad correctness claim, +or a score. + +## What This Pilot Is Trying To Learn + +The pilot should answer practical reviewer questions: + +- Can IAMScope collect the scoped environment successfully? +- Are the emitted findings explainable to the reviewer? +- Do the verdict categories help separate valid paths, blocked paths, + inconclusive paths, environmental extras, unsupported rows, and + not-currently-live-comparable rows? +- Does the reviewer find the uncertainty boundaries useful? +- Which IAM edge cases or account-specific conditions does IAMScope miss, + overstate, or preserve as inconclusive? + +The pilot is not trying to prove complete IAM attack-path coverage. + +## Minimum Safe Pilot Scope + +Prefer the smallest environment that can still teach something: + +- a non-production AWS account; +- a development or staging account; +- a sandbox organizational unit; +- a small set of accounts selected by the reviewer; +- a tightly scoped production read-only review only if the reviewer controls the + environment and explicitly authorizes that scope. + +The default pilot should be read-only and collection-focused. Do not start with +resource creation, mutation, Lambda invocation, service launch, or live +validation actions unless a separate written plan explicitly authorizes them. + +## Required Access Model + +IAMScope does not need production administrator access for a first pilot. Use a +reviewer-controlled AWS profile, role, or session with the minimum read-only +permissions needed for IAMScope collection. + +The access model should be documented before collection: + +- AWS account or account set in scope, using redacted identifiers in public + notes; +- AWS profile or role name used by the reviewer, not committed to the repo; +- allowed AWS services and read APIs; +- explicit out-of-scope services, accounts, and regions; +- whether Organizations, SCP, permission-boundary, or resource-policy evidence + is expected to be available; +- who may see raw artifacts before sanitization. + +Do not share long-lived secrets. Do not commit credentials, session tokens, +profile files, raw account IDs, raw ARNs, or raw production artifacts. + +## What IAMScope Collects + +For a collection-focused pilot, IAMScope may emit local artifacts such as: + +- `scenario.json`; +- `binding_metadata.json`; +- `findings.json`; +- logs or command output generated by the reviewer-run collection flow. + +Treat these artifacts as sensitive by default. Even when IAM policies are +read-only, the resulting graph can expose account structure, trust +relationships, role names, policy names, principal paths, and inferred risk +patterns. + +## What IAMScope Does Not Do + +In the default pilot, IAMScope should not: + +- create, update, or delete AWS resources; +- invoke Lambda functions; +- create triggers, function URLs, event source mappings, aliases, or versions; +- launch ECS tasks or other services; +- call STS probes for runtime proof; +- call `iam:PassRole`; +- run Terraform apply or destroy; +- require raw artifacts to be committed; +- require production administrator access. + +Any live validation or mutation must be a separate, explicitly authorized plan. + +## Suggested Command Flow + +The exact command depends on the reviewer environment and IAMScope collection +entry point available at the time of the pilot. A safe flow should look like +this in shape: + +```bash +# 1. Fresh local setup from the public README Quick Start. +python -m pip install -e ".[dev]" +./scripts/check.sh +./scripts/test_fast.sh + +# 2. Confirm the selected AWS profile/account is intentionally in scope. +# Use reviewer-approved commands and do not publish raw account identifiers. + +# 3. Run IAMScope collection into a scratch directory outside the repo. +mkdir -p /tmp/iamscope-real-env-pilot +# Example placeholder only; use the current IAMScope collection command and +# reviewer-approved profile/scope for the actual pilot. +# iamscope collect --profile --out /tmp/iamscope-real-env-pilot + +# 4. Review artifacts locally before sharing or sanitizing. +ls -la /tmp/iamscope-real-env-pilot +``` + +Do not paste raw `scenario.json`, `binding_metadata.json`, `findings.json`, or +logs into public issues or pull requests. Share sanitized excerpts only after +review. + +## Artifact Handling And Sanitization + +Raw account IDs, ARNs, `scenario.json`, `binding_metadata.json`, `findings.json`, +and logs should be treated as sensitive unless sanitized. + +Recommended handling: + +- write outputs under `/tmp` or a reviewer-controlled scratch path outside the + repo; +- do not commit raw collection artifacts; +- redact account IDs, principal ARNs, role ARNs, user names, policy names, and + environment-specific paths before sharing publicly; +- preserve enough structure in sanitized excerpts for reviewers to understand + the finding or limitation; +- record the sanitization choices alongside any shared excerpt; +- delete scratch artifacts when the reviewer no longer needs them. + +## Reviewer Validation Workflow + +Review should be qualitative and evidence-layered, not score-based: + +1. Select a small sample of IAMScope findings. +2. Ask the reviewer to classify each sampled row as one of: + - valid path; + - blocked path; + - inconclusive or needs additional context; + - environmental extra; + - unsupported; + - not-currently-live-comparable. +3. For each row, record why the reviewer agrees, disagrees, or cannot decide. +4. Preserve false-positive and false-negative discussions as examples, not as a + fake score. +5. Identify one or two IAM edge cases that should become future fixtures or + tests. + +## What Counts As Useful Evidence + +Useful evidence includes: + +- IAMScope collects the scoped environment successfully; +- findings are explainable to the reviewer; +- a sample of findings can be manually classified as valid path, blocked path, + inconclusive or needs context, or environmental extra; +- false positives and false negatives are discussed qualitatively with concrete + examples; +- at least one finding or non-finding teaches something about the account's IAM + graph, or identifies a clear limitation in IAMScope; +- raw evidence is handled safely. + +## What Does Not Count As Proof + +The pilot does not become proof just because IAMScope emits findings. It also +does not become proof because a reviewer agrees with a small sample. + +The following do not count as proof: + +- a large number of findings; +- a small number of findings; +- a single manually confirmed row; +- an unreviewed `validated` verdict; +- a spreadsheet percentage or score; +- unsanitized raw artifacts; +- live mutation outside the approved pilot scope. + +## Stop Conditions + +Stop the pilot if any of these occur: + +- the selected AWS profile or account is not the intended one; +- the command would require broader access than agreed; +- output paths point into the repository or another shared location by mistake; +- raw credentials, session tokens, account IDs, ARNs, or logs are about to be + committed or posted publicly; +- collection begins to call mutation, launch, invocation, Terraform, STS probe, + `iam:PassRole`, or other out-of-scope actions; +- the reviewer cannot determine whether the environment is authorized for the + test; +- the findings cannot be sanitized safely enough to discuss. + +## Non-Claims + +This protocol does not claim: + +- production readiness; +- broad IAMScope correctness; +- complete IAM attack-path coverage; +- exploitability proof; +- downstream authorization proof; +- Lambda invocation behavior; +- generic Deny correctness; +- composite benchmark score; +- pass/fail benchmark label. + +## Suggested Pilot Success Criteria + +A bounded pilot is successful if it produces reviewable evidence, even when that +evidence identifies limitations. Suggested criteria: + +- IAMScope collects the scoped environment successfully. +- Findings are explainable to the reviewer. +- A sample of findings can be manually classified as valid path, blocked path, + inconclusive or needs context, or environmental extra. +- False positives and false negatives are discussed qualitatively with examples. +- At least one finding or non-finding teaches something about the account's IAM + graph, or identifies a clear limitation in IAMScope. +- Raw evidence is handled safely. +- No composite score is produced. +- No pass/fail benchmark label is assigned.