Add bounded real-environment pilot protocol

docs/case-studies/bounded-real-environment-pilot-protocol.md

@@ -0,0 +1,227 @@
+# Bounded Real-Environment Pilot Protocol
+
+## Purpose
+
+This protocol gives external AWS IAM and cloud security reviewers a safe way to
+test IAMScope on a bounded real environment. The goal is to judge whether the
+approach generalizes beyond the controlled prod-like IAM sandbox without
+turning the pilot into a production-readiness claim, a broad correctness claim,
+or a score.
+
+## What This Pilot Is Trying To Learn
+
+The pilot should answer practical reviewer questions:
+
+- Can IAMScope collect the scoped environment successfully?
+- Are the emitted findings explainable to the reviewer?
+- Do the verdict categories help separate valid paths, blocked paths,
+  inconclusive paths, environmental extras, unsupported rows, and
+  not-currently-live-comparable rows?
+- Does the reviewer find the uncertainty boundaries useful?
+- Which IAM edge cases or account-specific conditions does IAMScope miss,
+  overstate, or preserve as inconclusive?
+
+The pilot is not trying to prove complete IAM attack-path coverage.
+
+## Minimum Safe Pilot Scope
+
+Prefer the smallest environment that can still teach something:
+
+- a non-production AWS account;
+- a development or staging account;
+- a sandbox organizational unit;
+- a small set of accounts selected by the reviewer;
+- a tightly scoped production read-only review only if the reviewer controls the
+  environment and explicitly authorizes that scope.
+
+The default pilot should be read-only and collection-focused. Do not start with
+resource creation, mutation, Lambda invocation, service launch, or live
+validation actions unless a separate written plan explicitly authorizes them.
+
+## Required Access Model
+
+IAMScope does not need production administrator access for a first pilot. Use a
+reviewer-controlled AWS profile, role, or session with the minimum read-only
+permissions needed for IAMScope collection.
+
+The access model should be documented before collection:
+
+- AWS account or account set in scope, using redacted identifiers in public
+  notes;
+- AWS profile or role name used by the reviewer, not committed to the repo;
+- allowed AWS services and read APIs;
+- explicit out-of-scope services, accounts, and regions;
+- whether Organizations, SCP, permission-boundary, or resource-policy evidence
+  is expected to be available;
+- who may see raw artifacts before sanitization.
+
+Do not share long-lived secrets. Do not commit credentials, session tokens,
+profile files, raw account IDs, raw ARNs, or raw production artifacts.
+
+## What IAMScope Collects
+
+For a collection-focused pilot, IAMScope may emit local artifacts such as:
+
+- `scenario.json`;
+- `binding_metadata.json`;
+- `findings.json`;
+- logs or command output generated by the reviewer-run collection flow.
+
+Treat these artifacts as sensitive by default. Even when IAM policies are
+read-only, the resulting graph can expose account structure, trust
+relationships, role names, policy names, principal paths, and inferred risk
+patterns.
+
+## What IAMScope Does Not Do
+
+In the default pilot, IAMScope should not:
+
+- create, update, or delete AWS resources;
+- invoke Lambda functions;
+- create triggers, function URLs, event source mappings, aliases, or versions;
+- launch ECS tasks or other services;
+- call STS probes for runtime proof;
+- call `iam:PassRole`;
+- run Terraform apply or destroy;
+- require raw artifacts to be committed;
+- require production administrator access.
+
+Any live validation or mutation must be a separate, explicitly authorized plan.
+
+## Suggested Command Flow
+
+The exact command depends on the reviewer environment and IAMScope collection
+entry point available at the time of the pilot. A safe flow should look like
+this in shape:
+
+```bash
+# 1. Fresh local setup from the public README Quick Start.
+python -m pip install -e ".[dev]"
+./scripts/check.sh
+./scripts/test_fast.sh
+
+# 2. Confirm the selected AWS profile/account is intentionally in scope.
+# Use reviewer-approved commands and do not publish raw account identifiers.
+
+# 3. Run IAMScope collection into a scratch directory outside the repo.
+mkdir -p /tmp/iamscope-real-env-pilot
+# Example placeholder only; use the current IAMScope collection command and
+# reviewer-approved profile/scope for the actual pilot.
+# iamscope collect --profile <reviewer-approved-profile> --out /tmp/iamscope-real-env-pilot
+
+# 4. Review artifacts locally before sharing or sanitizing.
+ls -la /tmp/iamscope-real-env-pilot
+```
+
+Do not paste raw `scenario.json`, `binding_metadata.json`, `findings.json`, or
+logs into public issues or pull requests. Share sanitized excerpts only after
+review.
+
+## Artifact Handling And Sanitization
+
+Raw account IDs, ARNs, `scenario.json`, `binding_metadata.json`, `findings.json`,
+and logs should be treated as sensitive unless sanitized.
+
+Recommended handling:
+
+- write outputs under `/tmp` or a reviewer-controlled scratch path outside the
+  repo;
+- do not commit raw collection artifacts;
+- redact account IDs, principal ARNs, role ARNs, user names, policy names, and
+  environment-specific paths before sharing publicly;
+- preserve enough structure in sanitized excerpts for reviewers to understand
+  the finding or limitation;
+- record the sanitization choices alongside any shared excerpt;
+- delete scratch artifacts when the reviewer no longer needs them.
+
+## Reviewer Validation Workflow
+
+Review should be qualitative and evidence-layered, not score-based:
+
+1. Select a small sample of IAMScope findings.
+2. Ask the reviewer to classify each sampled row as one of:
+   - valid path;
+   - blocked path;
+   - inconclusive or needs additional context;
+   - environmental extra;
+   - unsupported;
+   - not-currently-live-comparable.
+3. For each row, record why the reviewer agrees, disagrees, or cannot decide.
+4. Preserve false-positive and false-negative discussions as examples, not as a
+   fake score.
+5. Identify one or two IAM edge cases that should become future fixtures or
+   tests.
+
+## What Counts As Useful Evidence
+
+Useful evidence includes:
+
+- IAMScope collects the scoped environment successfully;
+- findings are explainable to the reviewer;
+- a sample of findings can be manually classified as valid path, blocked path,
+  inconclusive or needs context, or environmental extra;
+- false positives and false negatives are discussed qualitatively with concrete
+  examples;
+- at least one finding or non-finding teaches something about the account's IAM
+  graph, or identifies a clear limitation in IAMScope;
+- raw evidence is handled safely.
+
+## What Does Not Count As Proof
+
+The pilot does not become proof just because IAMScope emits findings. It also
+does not become proof because a reviewer agrees with a small sample.
+
+The following do not count as proof:
+
+- a large number of findings;
+- a small number of findings;
+- a single manually confirmed row;
+- an unreviewed `validated` verdict;
+- a spreadsheet percentage or score;
+- unsanitized raw artifacts;
+- live mutation outside the approved pilot scope.
+
+## Stop Conditions
+
+Stop the pilot if any of these occur:
+
+- the selected AWS profile or account is not the intended one;
+- the command would require broader access than agreed;
+- output paths point into the repository or another shared location by mistake;
+- raw credentials, session tokens, account IDs, ARNs, or logs are about to be
+  committed or posted publicly;
+- collection begins to call mutation, launch, invocation, Terraform, STS probe,
+  `iam:PassRole`, or other out-of-scope actions;
+- the reviewer cannot determine whether the environment is authorized for the
+  test;
+- the findings cannot be sanitized safely enough to discuss.
+
+## Non-Claims
+
+This protocol does not claim:
+
+- production readiness;
+- broad IAMScope correctness;
+- complete IAM attack-path coverage;
+- exploitability proof;
+- downstream authorization proof;
+- Lambda invocation behavior;
+- generic Deny correctness;
+- composite benchmark score;
+- pass/fail benchmark label.
+
+## Suggested Pilot Success Criteria
+
+A bounded pilot is successful if it produces reviewable evidence, even when that
+evidence identifies limitations. Suggested criteria:
+
+- IAMScope collects the scoped environment successfully.
+- Findings are explainable to the reviewer.
+- A sample of findings can be manually classified as valid path, blocked path,
+  inconclusive or needs context, or environmental extra.
+- False positives and false negatives are discussed qualitatively with examples.
+- At least one finding or non-finding teaches something about the account's IAM
+  graph, or identifies a clear limitation in IAMScope.
+- Raw evidence is handled safely.
+- No composite score is produced.
+- No pass/fail benchmark label is assigned.