feat: add score command by adamaltman · Pull Request #2648 · Redocly/redocly-cli

adamaltman · 2026-03-12T01:21:13Z

What/Why/How?

What: Adds a new score command to Redocly CLI that analyzes OpenAPI 3.x descriptions and produces two composite scores: Integration Simplicity (0-100, how easy is this API to integrate) and Agent Readiness (0-100, how usable is this API by AI agents/LLM tooling).

Why: API producers currently lack a quick, deterministic way to assess how developer-friendly or AI-agent-friendly their API descriptions are. The existing stats command counts structural elements but doesn't evaluate quality signals like documentation coverage, example presence, schema complexity, or error response structure. This command fills that gap with actionable, explainable scores.

How: The implementation follows the same pattern as the stats command (bundle + analyze), with a clean separation between metric collection and score calculation:

Metric collection (collectors/): Walks the bundled document, resolving internal $refs, to gather per-operation raw metrics (parameter counts, schema depth, polymorphism, description/constraint/example coverage, structured error responses, workflow dependency depth via shared schema refs).
Scoring (scoring.ts): Pure functions that normalize raw metrics into subscores and compute weighted composite scores. Thresholds and weights are configurable constants. anyOf is penalized more heavily than oneOf/allOf; discriminator presence improves the agent readiness polymorphism clarity subscore.
Hotspots (hotspots.ts): Identifies the operations with the most issues, sorted by number of reasons, with human-readable explanations.
Output: --format=stylish (default, with color bar charts) and --format=json (machine-readable for CI/dashboards).

Reference

Related to API governance and developer experience tooling. No existing issue -- this is a new feature.

Testing

43 unit tests across 3 test files covering schema depth calculation, $ref resolution, polymorphism counting (oneOf/anyOf/allOf), constraint detection (including const), example coverage scoring, anyOf penalty multiplier, discriminator impact on agent readiness, deterministic output, and score range validation.
Manually tested against three real OpenAPI descriptions (Redocly Cafe: 12 operations, Reunite Main: 299 operations, Rebilly: 606 operations) to verify scores are reasonable and hotspot reasoning is actionable.
TypeScript compiles cleanly (tsc --noEmit), all existing tests continue to pass.

Screenshots (optional)

Stylish output for Redocly Cafe:

  Scores

  Integration Simplicity:  85.3/100
  Agent Readiness:         94.4/100

  Integration Simplicity Subscores

  Parameter Simplicity     [█████████████████░░░] 83%
  Schema Simplicity        [████████████░░░░░░░░] 62%
  Documentation Quality    [███████████████████░] 97%
  Constraint Clarity       [███████████████████░] 96%
  Example Coverage         [██████████████████░░] 92%
  Error Clarity            [████████████████████] 100%
  Workflow Clarity         [█████████████████░░░] 83%

  Top 4 Hotspot Operations

  GET /orders (listOrders)
    Integration Simplicity: 69.1  Agent Readiness: 93.9
    - High parameter count (6)
    - Deep schema nesting (depth 5)

  PATCH /orders/{orderId} (updateOrder)
    Integration Simplicity: 77.6  Agent Readiness: 85.3
    - Missing request body examples

Check yourself

Code changed? - Tested with Redoc/Realm/Reunite (internal)
All new/updated code is covered by tests
New package installed? - Tested in different environments (browser/node)
Documentation update considered

Security

The security impact of the change has been considered
Code follows company security practices and guidelines

changeset-bot · 2026-03-12T01:21:17Z

🦋 Changeset detected

Latest commit: d6175e2

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@redocly/cli	Minor
@redocly/openapi-core	Minor
@redocly/respect-core	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2026-03-12T01:23:24Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	79.64% (🎯 79%)	7060 / 8864
🔵	Statements	78.98% (🎯 78%)	7315 / 9261
🔵	Functions	83.19% (🎯 82%)	1406 / 1690
🔵	Branches	70.99% (🎯 71%)	4742 / 6679

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
packages/cli/src/commands/score/collect-metrics.ts	55.97%	45.6%	72.72%	60.33%	26, 80-82, 85-87, 90-92, 95, 108-111, 133, 175-192, 209, 234-235, 242-294
packages/cli/src/commands/score/constants.ts	100%	100%	100%	100%
packages/cli/src/commands/score/hotspots.ts	100%	97.67%	100%	100%
packages/cli/src/commands/score/index.ts	96.42%	72.72%	100%	96.42%	125
packages/cli/src/commands/score/scoring.ts	99.01%	92.15%	100%	100%	282
packages/cli/src/commands/score/collectors/dependency-graph.ts	100%	90%	100%	100%
packages/cli/src/commands/score/collectors/document-metrics.ts	91.82%	74.57%	100%	95.48%	121, 127-129, 149-155, 318, 332, 354, 385, 409-419, 450
packages/cli/src/commands/score/formatters/json.ts	100%	100%	100%	100%
packages/cli/src/commands/score/formatters/stylish.ts	58.65%	29.82%	70%	60.24%	10, 78, 79, 107, 146-268

Generated in workflow #9207 for commit d6175e2 by the Vitest Coverage Report Action

github-actions · 2026-03-12T01:23:44Z

CLI Version	Mean Time ± Std Dev (s)	Relative Performance (Lower is Faster)
cli-latest	3.483s ± 0.019s	▓ 1.00x
cli-next	3.474s ± 0.020s	▓ 1.00x (Fastest)

.changeset/add-score-command.md

docs/@v2/commands/index.md

docs/@v2/commands/score.md

packages/cli/src/commands/score/collectors/document-metrics.ts

- Add "AI" before "agent readiness" in changeset and docs for clarity - Replace <pre> block with fenced code block in score.md - Add security scheme coverage to metrics documentation - Remove resolveIfRef helper, replace with resolveNode that falls back to the original node when resolution fails - Refactor to use walkDocument visitor approach (matching stats command pattern) instead of manually iterating the document tree - Use resolveDocument + normalizeVisitors + walkDocument from openapi-core for proper $ref resolution and spec-format extensibility - Update index.test.ts to mock the new walk infrastructure Made-with: Cursor

Made-with: Cursor

docs/@v2/commands/score.md

.changeset/add-score-command.md

Co-authored-by: Jacek Łękawa <164185257+JLekawa@users.noreply.github.com>

tatomyr

Left a couple of comments. I haven'd fully reviewed the scoring and collectors though as it takes time.

docs/@v2/commands/score.md

packages/cli/src/commands/score/__tests__/document-metrics.test.ts

packages/cli/src/commands/score/__tests__/formatters.test.ts

docs/@v2/commands/score.md

packages/cli/src/commands/score/index.ts

tatomyr

I found some dead code. Please check if it's needed and remove if not. Also, I'm not sure what tests to review as many appear to only test that dead code, so let's handle that first.

packages/cli/src/commands/score/__tests__/collect-metrics.ts

packages/cli/src/commands/score/collectors/document-metrics.ts

packages/cli/src/commands/score/__tests__/document-metrics.test.ts

packages/cli/src/commands/score/__tests__/formatters.test.ts

- Inline collect-metrics.ts test helper into document-metrics.test.ts - Use parseYaml as yaml directly instead of wrapper function - Remove default parameter from getStylishOutput in formatter tests - Use getMajorSpecVersion + exitWithError for spec version check - Add explicit case 'stylish' before default in format switch - Remove unsupported 'markdown' from score command format choices - Add comment explaining depth=-1 initialization - Clarify anyOf penalty and dependency terminology in docs - Update non-oas3 rejection test for exitWithError (throws) Made-with: Cursor

- Add type cast for parseYaml (returns unknown) in document-metrics tests - Inline collectDocumentMetrics helper into example-coverage tests - Remove collect-metrics.js import from scoring tests, use direct metrics - Add missing debugLogs property to accumulator mock in index tests Made-with: Cursor

packages/cli/src/commands/score/__tests__/document-metrics.test.ts

Extract the metric-collection pipeline from handleScore into a standalone collect-metrics.ts module with two exports: - collectMetrics(): low-level function used by handleScore - collectDocumentMetrics(): high-level convenience used by tests This eliminates ~300 lines of duplicated walker setup across three test files (document-metrics, example-coverage, index) and ensures tests exercise the same code path as the production command. Add $ref-keyed memoization to walkSchema so repeated references to the same component schema return cached stats instead of re-walking. Stripe API: 37.6s → 11.3s (3.3× faster). Made-with: Cursor

Add median alongside averages for parameters, schema depth, polymorphism, and properties in the stylish output. Rename the misleading "Avg max schema depth" to "Schema depth". Made-with: Cursor

Rename workflowClarity to dependencyClarity, workflowDepths to dependencyDepths, and workflow-graph to dependency-graph to align code naming with the "Dependency Clarity" display label. Also adds discoverability subscore, recursive composition keyword stripping for accurate property counting, and updates e2e snapshots. Made-with: Cursor

packages/cli/src/commands/score/collect-metrics.ts

tatomyr · 2026-03-24T17:00:27Z

packages/cli/src/commands/score/__tests__/formatters.test.ts

+}
+
+describe('printScoreStylish', () => {
+  it('outputs scores, subscores, metrics, and hotspots', () => {


Instead of mocking and presetting, it's better to simply create e2e tests for that cases.

Did this, but:

ERROR: Coverage for branches (70.99%) does not meet global threshold (71%)
Error: Process completed with exit code 1.

That's not a problem. You can lower the threshold in vitest.config.ts. It's expected that we only cover with unit tests only what's reasonable.

packages/cli/src/commands/score/__tests__/hotspots.test.ts

- Use isPlainObject from openapi-core for schema cycle detection instead of raw typeof checks that match arrays - Remove formatter unit tests in favor of e2e snapshot coverage - Clarify makeScores() purpose in hotspot tests Made-with: Cursor

tatomyr · 2026-03-31T07:54:00Z

docs/@v2/commands/score.md

+# `score`
+
+## Introduction
+


Suggested change

{% admonition type="warning" name="Important" %}

The `score` command is considered an experimental feature. This means it's still a work in progress and may go through major changes.

The `score` command supports OpenAPI 3.x descriptions only.

{% /admonition %}

I suggest adding the 'experimental' admonition similar to the join command.

tatomyr · 2026-03-31T07:54:34Z

docs/@v2/commands/score.md

+{% admonition type="warning" name="OpenAPI 3.x only" %}
+The `score` command supports OpenAPI 3.0, 3.1, and 3.2 descriptions.
+OpenAPI 2.0 (Swagger) and AsyncAPI are not currently supported.
+{% /admonition %}


Suggested change

{% admonition type="warning" name="OpenAPI 3.x only" %}

The `score` command supports OpenAPI 3.0, 3.1, and 3.2 descriptions.

OpenAPI 2.0 (Swagger) and AsyncAPI are not currently supported.

{% /admonition %}

tatomyr · 2026-03-31T07:59:13Z

docs/@v2/commands/score.md

+
+```bash
+redocly score <api>
+redocly score <api> [--format=<option>] [--config=<path>]


Suggested change

redocly score <api> [--format=<option>] [--config=<path>]

redocly score <api> [--format=<option>]

I don't think we explicitly use any redocly.yaml configuration in this command, so for me it doesn't make too much sense to list it in the examples.

tatomyr · 2026-03-31T08:09:19Z

packages/cli/src/commands/score/__tests__/index.test.ts

+
+describe('handleScore', () => {
+  beforeEach(() => {
+    mockOutput.mockClear();


This is redundant as we configured restoreMocks: true in vitest config.

Suggested change

mockOutput.mockClear();

tatomyr · 2026-03-31T08:09:57Z

packages/cli/src/commands/score/__tests__/index.test.ts

+      metrics: makeDocumentMetrics(new Map([['listItems', makeTestMetrics()]])),
+      debugLogs: [],
+    });
+    process.exitCode = undefined;


Seems redundant. At least, I don't see where we use exitCode in the tests.

tatomyr · 2026-03-31T08:16:44Z

packages/cli/src/commands/score/collectors/document-metrics.ts

+        }
+      },
+    },
+    Paths: {


What is this for? Could PathItems reside outside the Paths?

tatomyr · 2026-03-31T08:20:02Z

packages/cli/src/commands/score/formatters/json.ts

+        Array.from(result.rawMetrics.operations.entries()).map(([key, m]) => [
+          key,
+          {
+            path: m.path,


What is this mapping useful for? Why not simply use m or { ...m }? Are you trying to clean some properties?

tatomyr · 2026-03-31T09:05:40Z

packages/cli/src/commands/score/collect-metrics.ts

+            .filter((v) => typeof v === 'string' && v.startsWith('#/'))
+            .map((v) => ({ $ref: v }))
+        : null;
+    const hasDiscriminatorBranches = discriminatorRefs != null && discriminatorRefs.length > 0;


Suggested change

const hasDiscriminatorBranches = discriminatorRefs != null && discriminatorRefs.length > 0;

const hasDiscriminatorBranches = Array.isArray(discriminatorRefs) && discriminatorRefs.length > 0;

tatomyr · 2026-03-31T09:09:14Z

packages/cli/src/commands/score/collect-metrics.ts

+        for (let i = 1; i < discriminatorRefs!.length; i++) {
+          const branchStats = walkSchema(discriminatorRefs![i], debug);


Suggested change

for (let i = 1; i < discriminatorRefs!.length; i++) {

const branchStats = walkSchema(discriminatorRefs![i], debug);

for (let i = 1; i < discriminatorRefs.length; i++) {

const branchStats = walkSchema(discriminatorRefs[i], debug);

I don't think you need those asserts.

tatomyr · 2026-03-31T09:10:34Z

packages/cli/src/commands/score/collectors/document-metrics.ts

+        }
+
+        const localPropertyNames: string[] = [];
+        if (schema.properties && typeof schema.properties === 'object') {


Suggested change

if (schema.properties && typeof schema.properties === 'object') {

if (isPlainObject(schema.properties)) {

tatomyr · 2026-03-31T09:12:19Z

packages/cli/src/commands/score/collectors/document-metrics.ts

+
+function hasExample(mediaType: { example?: unknown; examples?: Record<string, unknown> }): boolean {
+  if (mediaType.example !== undefined) return true;
+  if (mediaType.examples && typeof mediaType.examples === 'object') {


Suggested change

if (mediaType.examples && typeof mediaType.examples === 'object') {

if (isNotEmptyObject(mediaType.examples)) {

tatomyr · 2026-03-31T09:15:59Z

packages/cli/src/commands/score/collect-metrics.ts

+
+  const walkSchema = (schemaNode: any, debug = false): SchemaStats => {
+    let resolved = schemaNode;
+    const ref: string | undefined =


It's better to reuse the isRef utility function here.

feat: add score command

9e40ca1

adamaltman requested review from a team as code owners March 12, 2026 01:21

RomanHotsiy reviewed Mar 12, 2026

View reviewed changes

adamaltman added 2 commits March 11, 2026 20:39

chore: add more tests

2c19efb

adamaltman force-pushed the aa/api-score branch from 79078d6 to 0e328aa Compare March 12, 2026 02:00

adamaltman and others added 3 commits March 11, 2026 21:02

fix: add missing version property to test CommandArgs

ded736c

Made-with: Cursor

chore: add e2e

05bd98c

chore: refactor the visitor to use native pattern

ddfdf10

JLekawa reviewed Mar 12, 2026

View reviewed changes

docs/@v2/commands/score.md Outdated Show resolved Hide resolved

.changeset/add-score-command.md Outdated Show resolved Hide resolved

chore: fix semantic line breaks

1acac45

Co-authored-by: Jacek Łękawa <164185257+JLekawa@users.noreply.github.com>

tatomyr reviewed Mar 12, 2026

View reviewed changes

adamaltman added 2 commits March 13, 2026 09:56

chore: refactor based on feedback

6cd3af4

fix: adjust walking method and add debugging arguments

92d40f7

tatomyr requested changes Mar 16, 2026

View reviewed changes

adamaltman added 2 commits March 24, 2026 06:58

adamaltman commented Mar 24, 2026

View reviewed changes

packages/cli/src/commands/score/__tests__/document-metrics.test.ts Outdated Show resolved Hide resolved

adamaltman added 3 commits March 24, 2026 07:42

chore: add median to raw metrics summary and fix label

2d33420

Add median alongside averages for parameters, schema depth, polymorphism, and properties in the stylish output. Rename the misleading "Avg max schema depth" to "Schema depth". Made-with: Cursor

tatomyr reviewed Mar 24, 2026

View reviewed changes

chore: address PR review feedback

d6175e2

- Use isPlainObject from openapi-core for schema cycle detection instead of raw typeof checks that match arrays - Remove formatter unit tests in favor of e2e snapshot coverage - Clarify makeScores() purpose in hotspot tests Made-with: Cursor

adamaltman requested a review from tatomyr March 30, 2026 16:33

tatomyr reviewed Mar 31, 2026

View reviewed changes

+{% admonition type="warning" name="Important" %}
+The `score` command is considered an experimental feature. This means it's still a work in progress and may go through major changes.
+The `score` command supports OpenAPI 3.x descriptions only.
+{% /admonition %}

	redocly score <api> [--format=<option>] [--config=<path>]
	redocly score <api> [--format=<option>]

	const hasDiscriminatorBranches = discriminatorRefs != null && discriminatorRefs.length > 0;
	const hasDiscriminatorBranches = Array.isArray(discriminatorRefs) && discriminatorRefs.length > 0;

		for (let i = 1; i < discriminatorRefs!.length; i++) {
		const branchStats = walkSchema(discriminatorRefs![i], debug);

	if (schema.properties && typeof schema.properties === 'object') {
	if (isPlainObject(schema.properties)) {

	if (mediaType.examples && typeof mediaType.examples === 'object') {
	if (isNotEmptyObject(mediaType.examples)) {

Conversation

adamaltman commented Mar 12, 2026

What/Why/How?

Reference

Testing

Screenshots (optional)

Check yourself

Security

Uh oh!

changeset-bot bot commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

github-actions bot commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report

Uh oh!

github-actions bot commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tatomyr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tatomyr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tatomyr Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

changeset-bot bot commented Mar 12, 2026 •

edited

Loading

github-actions bot commented Mar 12, 2026 •

edited

Loading

github-actions bot commented Mar 12, 2026 •

edited

Loading

tatomyr Mar 31, 2026 •

edited

Loading