You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add test/validation mode section to skit-analyze plan
Adds comprehensive test mode specifications including:
- Test case structure and organization
- Test configuration formats (manifest.json, test-config.json)
- Four validation strategies (structural, content, build, functional)
- Implementation components overview
- Usage examples and CLI flags
- Benefits of automated testing for prompt validation
This enables validation of Claude's code generation against known-good
outputs, preventing regressions and ensuring quality.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: Docs/skit-analyze-plan.md
+118Lines changed: 118 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -815,6 +815,124 @@ Include full API request/response, intermediate parsing steps, file collection d
815
815
- Generated code compiles
816
816
- Feature actually works as expected
817
817
818
+
## 8. Test/Validation Mode
819
+
820
+
To ensure the prompt generates correct code changes reliably, we need a test mode that validates Claude's responses against known-good outputs.
821
+
822
+
### Overview
823
+
824
+
Test mode (`--test` flag) runs the tool against a suite of test cases with predefined inputs and expected outputs, then validates that Claude's generated code matches expectations.
825
+
826
+
### Test Case Structure
827
+
828
+
```
829
+
test-cases/
830
+
├── subscript-basic/
831
+
│ ├── input/
832
+
│ │ ├── dsl.swift # Input DSL code
833
+
│ │ └── expected.swift # Expected Swift output
834
+
│ ├── expected-changes/
835
+
│ │ ├── manifest.json # List of expected file changes
0 commit comments