Skip to content

Commit 9d5c697

Browse files
committed
Default evals to 'base', now that only codebuff set is run by default
1 parent a8ed4a2 commit 9d5c697

File tree

6 files changed

+6
-1
lines changed

6 files changed

+6
-1
lines changed
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"sha":"00e88602aa42434b29918217257804fbd63413cc","spec":"The `initialSessionState` function needs to be enhanced with automatic knowledge file detection functionality.\n\nWhen the `knowledgeFiles` parameter is not provided (undefined), the system should automatically identify and extract knowledge files from the `projectFiles` parameter. \n\nThe automatic detection should:\n1. Only activate when `knowledgeFiles` is undefined (not when it's explicitly set to an empty object or other value)\n2. Scan through all files in the `projectFiles` parameter \n3. Identify files that have filenames ending with 'knowledge.md' or 'claude.md' (case-insensitive matching)\n4. Automatically populate the `knowledgeFiles` with these identified files, using the same file path as the key and file contents as the value\n\nThis behavior should allow users to include knowledge files in their project files without having to explicitly separate them into a distinct `knowledgeFiles` parameter, making the SDK more convenient to use when knowledge files follow standard naming conventions.\n\nThe changelog should be updated to reflect this new automatic parsing capability as a \"Changed\" item under version 0.1.9.","fileStates":[{"path":"sdk/CHANGELOG.md","preContent":"# Changelog\n\nAll notable changes to the @codebuff/sdk package will be documented in this file.\n\n## [0.1.9] - 2025-08-13\n\n### Added\n\n- `closeConnection` method in `CodebuffClient`\n\n## [0.1.8] - 2025-08-13\n\n### Added\n\n- `withAdditionalMessage` and `withMessageHistory` functions\n - Add images, files, or other messages to a previous run\n - Modify the history of any run\n- `initialSessionState` and `generateInitialRunState` functions\n - Create a SessionState or RunState object from scratch\n\n### Removed\n\n- `getInitialSessionState` function\n\n## [0.1.7] - 2025-08-12\n\n### Updated types! AgentConfig has been renamed to AgentDefinition.\n\n## [0.1.5] - 2025-08-09\n\n### Added\n\n- Complete `CodebuffClient`\n- Better docs\n- New `run()` api\n\n## [0.0.1] - 2025-08-05\n\n### Added\n\n- Initial release of the Codebuff SDK\n- `CodebuffClient` class for interacting with Codebuff agents\n- `runNewChat` method for starting new chat sessions\n- TypeScript support with full type definitions\n- Support for all Codebuff agent types\n- Event streaming for real-time responses\n","postContent":"# Changelog\n\nAll notable changes to the @codebuff/sdk package will be documented in this file.\n\n## [0.1.9] - 2025-08-13\n\n### Added\n\n- `closeConnection` method in `CodebuffClient`\n\n### Changed\n\n- Automatic parsing of `knowledgeFiles` if not provided\n\n## [0.1.8] - 2025-08-13\n\n### Added\n\n- `withAdditionalMessage` and `withMessageHistory` functions\n - Add images, files, or other messages to a previous run\n - Modify the history of any run\n- `initialSessionState` and `generateInitialRunState` functions\n - Create a SessionState or RunState object from scratch\n\n### Removed\n\n- `getInitialSessionState` function\n\n## [0.1.7] - 2025-08-12\n\n### Updated types! AgentConfig has been renamed to AgentDefinition.\n\n## [0.1.5] - 2025-08-09\n\n### Added\n\n- Complete `CodebuffClient`\n- Better docs\n- New `run()` api\n\n## [0.0.1] - 2025-08-05\n\n### Added\n\n- Initial release of the Codebuff SDK\n- `CodebuffClient` class for interacting with Codebuff agents\n- `runNewChat` method for starting new chat sessions\n- TypeScript support with full type definitions\n- Support for all Codebuff agent types\n- Event streaming for real-time responses\n"},{"path":"sdk/src/run-state.ts","preContent":"import * as os from 'os'\n\nimport { getInitialSessionState } from '../../common/src/types/session-state'\n\nimport type { ServerAction } from '../../common/src/actions'\nimport type { AgentDefinition } from '../../common/src/templates/initial-agents-dir/types/agent-definition'\nimport type { CodebuffMessage } from '../../common/src/types/message'\nimport type { SessionState } from '../../common/src/types/session-state'\n\nexport type RunState = {\n sessionState: SessionState\n toolResults: ServerAction<'prompt-response'>['toolResults']\n}\n\nexport function initialSessionState(\n cwd: string,\n options: {\n // TODO: Parse projectFiles into fileTree, fileTokenScores, tokenCallers\n projectFiles?: Record<string, string>\n knowledgeFiles?: Record<string, string>\n agentDefinitions?: AgentDefinition[]\n maxAgentSteps?: number\n },\n) {\n const { knowledgeFiles = {}, agentDefinitions = [] } = options\n\n // Process agentDefinitions array and convert handleSteps functions to strings\n const processedAgentTemplates: Record<string, any> = {}\n agentDefinitions.forEach((definition) => {\n const processedConfig = { ...definition } as Record<string, any>\n if (\n processedConfig.handleSteps &&\n typeof processedConfig.handleSteps === 'function'\n ) {\n processedConfig.handleSteps = processedConfig.handleSteps.toString()\n }\n if (processedConfig.id) {\n processedAgentTemplates[processedConfig.id] = processedConfig\n }\n })\n\n const initialState = getInitialSessionState({\n projectRoot: cwd,\n cwd,\n fileTree: [],\n fileTokenScores: {},\n tokenCallers: {},\n knowledgeFiles,\n userKnowledgeFiles: {},\n agentTemplates: processedAgentTemplates,\n gitChanges: {\n status: '',\n diff: '',\n diffCached: '',\n lastCommitMessages: '',\n },\n changesSinceLastChat: {},\n shellConfigFiles: {},\n systemInfo: {\n platform: process.platform,\n shell: process.platform === 'win32' ? 'cmd.exe' : 'bash',\n nodeVersion: process.version,\n arch: process.arch,\n homedir: os.homedir(),\n cpus: os.cpus().length ?? 1,\n },\n })\n\n if (options.maxAgentSteps) {\n initialState.mainAgentState.stepsRemaining = options.maxAgentSteps\n }\n\n return initialState\n}\n\nexport function generateInitialRunState({\n cwd,\n projectFiles,\n knowledgeFiles,\n agentDefinitions,\n maxAgentSteps,\n}: {\n cwd: string\n projectFiles?: Record<string, string>\n knowledgeFiles?: Record<string, string>\n agentDefinitions?: AgentDefinition[]\n maxAgentSteps?: number\n}): RunState {\n return {\n sessionState: initialSessionState(cwd, {\n projectFiles,\n knowledgeFiles,\n agentDefinitions,\n maxAgentSteps,\n }),\n toolResults: [],\n }\n}\n\nexport function withAdditionalMessage({\n runState,\n message,\n}: {\n runState: RunState\n message: CodebuffMessage\n}): RunState {\n // Deep copy\n const newRunState = JSON.parse(JSON.stringify(runState)) as typeof runState\n\n newRunState.sessionState.mainAgentState.messageHistory.push(message)\n\n return newRunState\n}\n\nexport function withMessageHistory({\n runState,\n messages,\n}: {\n runState: RunState\n messages: CodebuffMessage[]\n}): RunState {\n // Deep copy\n const newRunState = JSON.parse(JSON.stringify(runState)) as typeof runState\n\n newRunState.sessionState.mainAgentState.messageHistory = messages\n\n return newRunState\n}\n","postContent":"import * as os from 'os'\n\nimport { getInitialSessionState } from '../../common/src/types/session-state'\n\nimport type { ServerAction } from '../../common/src/actions'\nimport type { AgentDefinition } from '../../common/src/templates/initial-agents-dir/types/agent-definition'\nimport type { CodebuffMessage } from '../../common/src/types/message'\nimport type { SessionState } from '../../common/src/types/session-state'\n\nexport type RunState = {\n sessionState: SessionState\n toolResults: ServerAction<'prompt-response'>['toolResults']\n}\n\nexport function initialSessionState(\n cwd: string,\n options: {\n // TODO: Parse projectFiles into fileTree, fileTokenScores, tokenCallers\n projectFiles?: Record<string, string>\n knowledgeFiles?: Record<string, string>\n agentDefinitions?: AgentDefinition[]\n maxAgentSteps?: number\n },\n) {\n const { projectFiles = {}, agentDefinitions = [] } = options\n let { knowledgeFiles } = options\n\n if (knowledgeFiles === undefined) {\n knowledgeFiles = {}\n for (const [filePath, fileContents] of Object.entries(projectFiles)) {\n if (filePath in projectFiles) {\n continue\n }\n const lowercasePathName = filePath.toLowerCase()\n if (\n !lowercasePathName.endsWith('knowledge.md') &&\n !lowercasePathName.endsWith('claude.md')\n ) {\n continue\n }\n\n knowledgeFiles[filePath] = fileContents\n }\n }\n\n // Process agentDefinitions array and convert handleSteps functions to strings\n const processedAgentTemplates: Record<string, any> = {}\n agentDefinitions.forEach((definition) => {\n const processedConfig = { ...definition } as Record<string, any>\n if (\n processedConfig.handleSteps &&\n typeof processedConfig.handleSteps === 'function'\n ) {\n processedConfig.handleSteps = processedConfig.handleSteps.toString()\n }\n if (processedConfig.id) {\n processedAgentTemplates[processedConfig.id] = processedConfig\n }\n })\n\n const initialState = getInitialSessionState({\n projectRoot: cwd,\n cwd,\n fileTree: [],\n fileTokenScores: {},\n tokenCallers: {},\n knowledgeFiles,\n userKnowledgeFiles: {},\n agentTemplates: processedAgentTemplates,\n gitChanges: {\n status: '',\n diff: '',\n diffCached: '',\n lastCommitMessages: '',\n },\n changesSinceLastChat: {},\n shellConfigFiles: {},\n systemInfo: {\n platform: process.platform,\n shell: process.platform === 'win32' ? 'cmd.exe' : 'bash',\n nodeVersion: process.version,\n arch: process.arch,\n homedir: os.homedir(),\n cpus: os.cpus().length ?? 1,\n },\n })\n\n if (options.maxAgentSteps) {\n initialState.mainAgentState.stepsRemaining = options.maxAgentSteps\n }\n\n return initialState\n}\n\nexport function generateInitialRunState({\n cwd,\n projectFiles,\n knowledgeFiles,\n agentDefinitions,\n maxAgentSteps,\n}: {\n cwd: string\n projectFiles?: Record<string, string>\n knowledgeFiles?: Record<string, string>\n agentDefinitions?: AgentDefinition[]\n maxAgentSteps?: number\n}): RunState {\n return {\n sessionState: initialSessionState(cwd, {\n projectFiles,\n knowledgeFiles,\n agentDefinitions,\n maxAgentSteps,\n }),\n toolResults: [],\n }\n}\n\nexport function withAdditionalMessage({\n runState,\n message,\n}: {\n runState: RunState\n message: CodebuffMessage\n}): RunState {\n // Deep copy\n const newRunState = JSON.parse(JSON.stringify(runState)) as typeof runState\n\n newRunState.sessionState.mainAgentState.messageHistory.push(message)\n\n return newRunState\n}\n\nexport function withMessageHistory({\n runState,\n messages,\n}: {\n runState: RunState\n messages: CodebuffMessage[]\n}): RunState {\n // Deep copy\n const newRunState = JSON.parse(JSON.stringify(runState)) as typeof runState\n\n newRunState.sessionState.mainAgentState.messageHistory = messages\n\n return newRunState\n}\n"}]}

evals/git-evals/logs/codebuff-yw_Q5Gr1Tls/eval-commit-212590d.json

Lines changed: 1 addition & 0 deletions
Large diffs are not rendered by default.

evals/git-evals/logs/codebuff-yw_Q5Gr1Tls/eval-commit-26140c8.json

Lines changed: 1 addition & 0 deletions
Large diffs are not rendered by default.

evals/git-evals/logs/codebuff-yw_Q5Gr1Tls/eval-commit-fa43720.json

Lines changed: 1 addition & 0 deletions
Large diffs are not rendered by default.

evals/git-evals/logs/codebuff-yw_Q5Gr1Tls/eval-commit-fe667af.json

Lines changed: 1 addition & 0 deletions
Large diffs are not rendered by default.

evals/git-evals/run-eval-set.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ class RunEvalSetCommand extends Command {
8080
}),
8181
agent: Flags.string({
8282
description: 'Codebuff agent id to use',
83-
default: 'base-lite',
83+
default: 'base',
8484
}),
8585
help: Flags.help({ char: 'h' }),
8686
}

0 commit comments

Comments
 (0)