docs(evaluation): revamp evals documentation for new eval system#648
Open
KarthikAvinashFI wants to merge 7 commits into
Open
docs(evaluation): revamp evals documentation for new eval system#648KarthikAvinashFI wants to merge 7 commits into
KarthikAvinashFI wants to merge 7 commits into
Conversation
Aligns the evaluation docs with the post-revamp platform: three eval types (Agents / LLM-As-A-Judge / Code), three output types (Pass/fail / Scoring / Choices), composite templates, versioning, ground truth, error localization, and updated apply flows for datasets, trace projects (now via Tasks), and simulation. Concepts (rewritten / new): - eval-types: 3-type taxonomy matching the create-page tabs - eval-templates: built-in vs custom, single vs composite, versioning - eval-results: result formats per output type - judge-models: Turing models + bring-your-own - understanding-evaluation: surfaces and how it all fits - output-types (new): Pass/fail, Scoring (label-based), Choices - data-injection (new): the six Context options - composite-evals (new): aggregation functions and child axis - versioning (new): Set as Default, Restore Version, pinning Features (rewritten / new): - custom: full create flow for all 3 types with field reference - evaluate: dataset apply flow + SDK - test-playground (new): four source modes, AI generate - error-localization (new): toggle, run lifecycle, SDK - ground-truth (new): upload, mapping, embedding statuses Surface-specific updates: - observe/features/evals: rewritten around the Tasks page flow (Basic Info / Evaluations / Filters / Scheduling) - quickstart/running-evals-in-simulation: aligned with the 4-step Create a Simulation wizard Eval Groups was removed from docs as the feature is no longer exposed in the main UI navigation. TH-4638
Adds reference pages for built-in evals that were missing documentation (deterministic, statistical, and agent-mode templates). Also fixes Detect Hallucination input requirement.
Adds rows for the freshly-generated reference pages so users can find them from the Built-in Evals catalog.
Cleaned up auto-generated descriptions and parameter tables across 90 built-in eval reference pages. Removed truncated description suffixes, replaced placeholder parameter descriptions (e.g. "The output.") with concrete type and value-shape information.
…k (TH-4934) - Concept: how connectors plug into Agent-mode evals, what the judge sees, cost and latency. - Feature: UI walkthrough for attaching connectors to an eval, troubleshooting table. - Cookbook: end-to-end example using a CRM MCP server to verify support replies. - Nav: register all three under Evaluation / Cookbooks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings the evaluation docs in line with the post-revamp platform. Replaces the old four-method taxonomy (LLM as Judge / Deterministic / Statistical Metric / LLM as Ranker) with what the UI actually shows today: Agents, LLM-As-A-Judge, and Code. Adds new concept and feature pages for things that were undocumented (composite evals, versioning, ground truth, error localization, test playground, data injection, output types in their new label-based form, MCP connectors). Rewrites the trace and simulation eval guides around the actual Tasks and Create a Simulation flows.
Linear: TH-4638, TH-4934
What changed
Concepts (under
evaluation/concepts/)Rewritten:
eval-types,eval-templates,eval-results,judge-models,understanding-evaluation.New:
output-types,data-injection,composite-evals,versioning,mcp-connectors.Features (under
evaluation/features/)Rewritten:
custom,evaluate.New:
test-playground,error-localization,ground-truth,mcp-connectors.Minor:
custom-models(added trace projects to the surfaces list).Cookbooks (under
cookbook/evaluation/)New:
eval-with-mcp-connectors— end-to-end CRM lookup example.Surface-specific eval guides (outside
evaluation/)observe/features/evalsrewritten around the Tasks flow (Basic Info / Evaluations / Filters / Scheduling) and theHistorical data/New incoming datarun modes.quickstart/running-evals-in-simulationaligned with the 4-step Create a Simulation wizard (Add simulation details, Choose Scenario(s), Select Evaluations, Summary) and updated mapping fields.Navigation
src/lib/navigation.tsupdated to include the new concept, feature, and cookbook pages in the sidebar.Removed
eval-groups.mdxand all references. The Groups feature is no longer reachable from the main UI navigation.Style guide compliance
## About; no UI walkthrough screenshots in concept pages.Verification
pnpm audit-links— 0 broken nav links, 0 broken content links.EvalCreatePage.jsx,EvalPickerConfigFull.jsx,TestPlayground.jsx, etc.) and backend (model_hub/types.py,evaluations/engine/instance.py,ee/evals/llm/agent_evaluator/).Test plan
pnpm audit-linkspasses.pnpm buildpasses.pnpm dev.