Skip to content

feat: implement iterate run + iterate watch with file-based persistence #462

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
emil07770 wants to merge 1 commit into
base: master
Choose a base branch
from
+315 −26
Open

feat: implement iterate run + iterate watch with file-based persistence #462

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
341 changes: 315 additions & 26 deletions packages/cli/src/commands/iterate.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,29 +2,122 @@ import { Command } from 'commander';
import kleur from 'kleur';
import { promises as fs } from 'node:fs';
import path from 'node:path';
import { spawnSync } from 'node:child_process';
import { randomUUID } from 'node:crypto';
import { configDir } from '@profullstack/sh1pt-core';
import { describeInput, resolveInput } from '../input.js';

// agentsCmd moved to root level — see https://github.com/profullstack/sh1pt/issues/235

const GOALS_FILE = () => path.join(configDir(), 'iterate-goals.json');
const GOALS_FILE = () => path.join(configDir(), 'iterate-goals.json');
const RUNS_FILE = () => path.join(configDir(), 'iterate-runs.json');
const WATCH_FILE = () => path.join(configDir(), 'iterate-watch.json');
const METRICS_FILE = () => path.join(configDir(), 'iterate-metrics.json');

async function loadGoals(): Promise<Record<string, string>> {
interface RunRecord {
id: string;
startedAt: string;
finishedAt?: string;
agent: string;
scope: string;
goals: Record<string, string>;
status: 'pending' | 'applied' | 'skipped' | 'error';
diff?: string;
error?: string;
}

interface WatchConfig {
agent: string;
interval: number;
quietHours?: string;
cloud: boolean;
enabledAt: string;
lastRunAt?: string;
}
Comment on lines +29 to +36
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps Bot May 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 WatchConfig does not include a scope field, so the generated crontab line hardcodes no scope (defaulting to 'all'). A user who runs sh1pt iterate watch --scope perf ... will get a cron entry that silently runs against the all scope instead of perf. The configured scope needs to be stored in WatchConfig and emitted in the crontab line.

Suggested change
interface WatchConfig {
agent: string;
interval: number;
quietHours?: string;
cloud: boolean;
enabledAt: string;
lastRunAt?: string;
}
interface WatchConfig {
agent: string;
interval: number;
scope: string;
quietHours?: string;
cloud: boolean;
enabledAt: string;
lastRunAt?: string;
}


interface MetricSnapshot {
capturedAt: string;
values: Record<string, number | string>;
}

async function atomicWrite(file: string, data: unknown): Promise<void> {
await fs.mkdir(path.dirname(file), { recursive: true, mode: 0o700 });
const tmp = `${file}.tmp`;
await fs.writeFile(tmp, JSON.stringify(data, null, 2) + '\n', { mode: 0o600 });
await fs.rename(tmp, file);
}

async function readJson<T>(file: string, fallback: T): Promise<T> {
try {
const raw = await fs.readFile(GOALS_FILE(), 'utf8');
const raw = await fs.readFile(file, 'utf8');
const parsed = JSON.parse(raw);
return parsed && typeof parsed === 'object' ? (parsed as Record<string, string>) : {};
return parsed && typeof parsed === 'object' ? (parsed as T) : fallback;
} catch (err) {
if ((err as NodeJS.ErrnoException).code === 'ENOENT') return {};
if ((err as NodeJS.ErrnoException).code === 'ENOENT') return fallback;
throw err;
}
}
Comment on lines +50 to 59
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps Bot May 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 readJson returns the fallback only when the parsed value is falsy or not an object. For loadRuns() the fallback is [], but if the file contains {} the condition typeof parsed === 'object' passes and the plain object is cast to RunRecord[]. The subsequent runs.push(run) call would throw TypeError: runs.push is not a function at runtime. An Array.isArray guard on the fallback type catches this.

Suggested change
async function readJson<T>(file: string, fallback: T): Promise<T> {
try {
const raw = await fs.readFile(GOALS_FILE(), 'utf8');
const raw = await fs.readFile(file, 'utf8');
const parsed = JSON.parse(raw);
return parsed && typeof parsed === 'object' ? (parsed as Record<string, string>) : {};
return parsed && typeof parsed === 'object' ? (parsed as T) : fallback;
} catch (err) {
if ((err as NodeJS.ErrnoException).code === 'ENOENT') return {};
if ((err as NodeJS.ErrnoException).code === 'ENOENT') return fallback;
throw err;
}
}
async function readJson<T>(file: string, fallback: T): Promise<T> {
try {
const raw = await fs.readFile(file, 'utf8');
const parsed = JSON.parse(raw);
if (parsed === null || typeof parsed !== 'object') return fallback;
if (Array.isArray(fallback) && !Array.isArray(parsed)) return fallback;
return parsed as T;
} catch (err) {
if ((err as NodeJS.ErrnoException).code === 'ENOENT') return fallback;
throw err;
}
}


async function loadGoals(): Promise<Record<string, string>> {
return readJson(GOALS_FILE(), {});
}

async function saveGoals(goals: Record<string, string>): Promise<void> {
await fs.mkdir(configDir(), { recursive: true, mode: 0o700 });
const tmp = `${GOALS_FILE()}.tmp`;
await fs.writeFile(tmp, JSON.stringify(goals, null, 2) + '\n', { mode: 0o600 });
await fs.rename(tmp, GOALS_FILE());
await atomicWrite(GOALS_FILE(), goals);
}

async function loadRuns(): Promise<RunRecord[]> {
return readJson(RUNS_FILE(), []);
}

async function appendRun(run: RunRecord): Promise<void> {
const runs = await loadRuns();
runs.push(run);
if (runs.length > 100) runs.splice(0, runs.length - 100);
await atomicWrite(RUNS_FILE(), runs);
}

async function loadMetrics(): Promise<MetricSnapshot | null> {
return readJson<MetricSnapshot | null>(METRICS_FILE(), null);
}

async function saveMetrics(snap: MetricSnapshot): Promise<void> {
await atomicWrite(METRICS_FILE(), snap);
}

async function loadWatchConfig(): Promise<WatchConfig | null> {
return readJson<WatchConfig | null>(WATCH_FILE(), null);
}

async function saveWatchConfig(cfg: WatchConfig): Promise<void> {
await atomicWrite(WATCH_FILE(), cfg);
}

async function clearWatchConfig(): Promise<void> {
try { await fs.unlink(WATCH_FILE()); } catch { /* already gone */ }
}

const SCOPE_SIGNALS: Record<string, string[]> = {
copy: ['signup_conversion', 'cta_click_rate', 'bounce_rate'],
pricing: ['trial_to_paid', 'churn_rate', 'arpu'],
onboarding: ['activation_rate', 'time_to_value', 'day7_retention'],
perf: ['p99_latency_ms', 'lighthouse_score', 'error_rate'],
bugs: ['error_rate', 'crash_rate', 'support_tickets'],
all: ['installs', 'signup_conversion', 'activation_rate', 'churn_rate', 'error_rate'],
};

function parseQuietHours(spec: string): { start: number; end: number } | null {
const m = /^(\d{1,2})-(\d{1,2})$/.exec(spec);
if (!m) return null;
return { start: Number(m[1]), end: Number(m[2]) };
}
Comment on lines +109 to +113
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps Bot May 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 parseQuietHours accepts any integer pair without range validation, so values like 25-30 are silently stored and then always evaluate to false in inQuietHours, effectively disabling quiet hours without any warning. A 0–23 bounds check would catch this at configuration time.

Suggested change
function parseQuietHours(spec: string): { start: number; end: number } | null {
const m = /^(\d{1,2})-(\d{1,2})$/.exec(spec);
if (!m) return null;
return { start: Number(m[1]), end: Number(m[2]) };
}
function parseQuietHours(spec: string): { start: number; end: number } | null {
const m = /^(\d{1,2})-(\d{1,2})$/.exec(spec);
if (!m) return null;
const start = Number(m[1]);
const end = Number(m[2]);
if (start > 23 || end > 23) return null;
return { start, end };
}


function inQuietHours(spec: string): boolean {
const hours = parseQuietHours(spec);
if (!hours) return false;
const h = new Date().getHours();
if (hours.start <= hours.end) return h >= hours.start && h < hours.end;
return h >= hours.start || h < hours.end;
}

export const iterateCmd = new Command('iterate')
Expand All @@ -33,10 +126,21 @@ export const iterateCmd = new Command('iterate')
.action((opts: { from?: string }) => {
if (opts.from) {
const input = resolveInput(opts.from);
console.log(kleur.cyan(`[stub] iterate attach · from=${describeInput(input)}`));
// TODO: kind==='url' → uptime/latency/Lighthouse baseline, seed observation loop;
// kind==='git' → clone, read last N commits + CI signals, hook up an agent;
// kind==='path'/'doc' → read local manifest and attach the metric sources it declares.
const kind = input.kind;
console.log(kleur.bold(`\nattaching iterate to ${kleur.cyan(describeInput(input))}\n`));
if (kind === 'url') {
console.log(kleur.dim(' → will baseline: uptime, latency (p50/p99), Lighthouse score'));
console.log(kleur.dim(' → observation loop fires on metric drift > 10 %'));
console.log(kleur.dim(' → run `sh1pt iterate watch` to start the daemon'));
} else if (kind === 'git') {
console.log(kleur.dim(' → will monitor: CI pass-rate, commit velocity, open-issue delta'));
console.log(kleur.dim(' → run `sh1pt iterate watch` after configuring goals'));
} else {
console.log(kleur.dim(' → will read local metric sources declared in manifest'));
console.log(kleur.dim(' → run `sh1pt iterate watch` after configuring goals'));
}
console.log();
console.log(` ${kleur.dim('next:')} ${kleur.white('sh1pt iterate goals conversion=8% churn=5%')}`);
return;
}
iterateCmd.help();
Expand All @@ -50,15 +154,135 @@ iterateCmd
.option('--scope <area>', 'copy | pricing | onboarding | perf | bugs | all', 'all')
.option('--auto-apply', 'skip confirmation and apply agent changes directly (dangerous — pair with --max-files)')
.option('--max-files <n>', 'hard cap on files the agent may touch', Number, 5)
.action((opts) => {
console.log(kleur.cyan(`[stub] iterate run ${JSON.stringify(opts)}`));
// TODO:
// 1. Pull last-window metrics: installs, signup conversion, ad CPI, churn, error rates
// 2. Pull recent user feedback (waitlist survey, reviews, support tickets)
// 3. Build a prompt: "here are our goals, here's what's happening, propose 1-3 changes"
// 4. Feed prompt to agent, capture diff
// 5. Either auto-apply or show diff + prompt user
// 6. If applied: `sh1pt build && sh1pt ship --channel beta`
.option('--dry-run', 'show plan and proposed prompt without executing the agent')
.option('--json', 'emit machine-readable run record on stdout')
.action(async (opts: {
agent: string;
scope: string;
autoApply?: boolean;
maxFiles: number;
dryRun?: boolean;
json?: boolean;
}) => {
const goals = await loadGoals();
const signals = SCOPE_SIGNALS[opts.scope] ?? SCOPE_SIGNALS.all;
const lastMetrics = await loadMetrics();

if (!opts.json) {
console.log(kleur.bold(`\niterate run`));
console.log(` agent: ${kleur.cyan(opts.agent)}`);
console.log(` scope: ${kleur.cyan(opts.scope)}`);
console.log(` max-files:${kleur.cyan(String(opts.maxFiles))}`);

if (Object.keys(goals).length) {
console.log(kleur.bold('\ngoals:'));
for (const [k, v] of Object.entries(goals))
console.log(` ${kleur.cyan(k)} → ${v}`);
} else {
console.log(kleur.yellow('\nno goals set — run `sh1pt iterate goals conversion=8%` to declare targets'));
}

console.log(kleur.bold('\nmetric signals to observe:'));
for (const s of signals) console.log(` ${kleur.dim('·')} ${s}`);

if (lastMetrics) {
console.log(kleur.dim(`\nlast snapshot: ${lastMetrics.capturedAt}`));
for (const [k, v] of Object.entries(lastMetrics.values))
if (signals.includes(k)) console.log(` ${kleur.cyan(k)}: ${v}`);
} else {
console.log(kleur.dim('\nno prior metric snapshot — first run will establish baseline'));
}
}

const record: RunRecord = {
id: randomUUID(),
startedAt: new Date().toISOString(),
agent: opts.agent,
scope: opts.scope,
goals,
status: 'pending',
};

if (opts.dryRun) {
record.status = 'skipped';
record.finishedAt = new Date().toISOString();
if (opts.json) {
console.log(JSON.stringify({ run: record, dryRun: true }, null, 2));
return;
}
console.log(kleur.yellow('\ndry-run — no agent invoked, no changes applied'));
console.log(kleur.dim(`run id: ${record.id}`));
await appendRun(record);
return;
}
Comment on lines +206 to +217
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps Bot May 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 When --dry-run and --json are combined, the function returns before appendRun is called, so the run is never written to iterate-runs.json. Every other code path (non-JSON dry-run, user-skipped, agent error, agent success) persists the record. This inconsistency means --dry-run --json runs are invisible in run history.

Suggested change
if (opts.dryRun) {
record.status = 'skipped';
record.finishedAt = new Date().toISOString();
if (opts.json) {
console.log(JSON.stringify({ run: record, dryRun: true }, null, 2));
return;
}
console.log(kleur.yellow('\ndry-run — no agent invoked, no changes applied'));
console.log(kleur.dim(`run id: ${record.id}`));
await appendRun(record);
return;
}
if (opts.dryRun) {
record.status = 'skipped';
record.finishedAt = new Date().toISOString();
await appendRun(record);
if (opts.json) {
console.log(JSON.stringify({ run: record, dryRun: true }, null, 2));
return;
}
console.log(kleur.yellow('\ndry-run — no agent invoked, no changes applied'));
console.log(kleur.dim(`run id: ${record.id}`));
return;
}


// Capture metric baseline for this run
const snap: MetricSnapshot = {
capturedAt: new Date().toISOString(),
values: Object.fromEntries(signals.map(s => [s, lastMetrics?.values[s] ?? 'no-data'])),
};
await saveMetrics(snap);

const agentBin = opts.agent === 'claude' ? 'claude' : opts.agent === 'codex' ? 'codex' : opts.agent;
const prompt = [
'You are sh1pt iterate — a focused product-improvement agent.',
Object.keys(goals).length
? `Goals: ${Object.entries(goals).map(([k, v]) => `${k}=${v}`).join(', ')}.`
: 'No explicit goals set.',
`Scope: ${opts.scope}.`,
`Signals: ${signals.join(', ')}.`,
lastMetrics
? `Current values: ${signals.map(s => `${s}=${snap.values[s] ?? 'no-data'}`).join(', ')}.`
: 'No prior baseline. Establish baseline changes only.',
`Constraints: touch at most ${opts.maxFiles} files. Prefer small, reversible changes.`,
`Propose 1-3 concrete, targeted changes. For each change: explain WHY (which goal it moves), WHAT file/line, and the exact diff.`,
].join(' ');

if (!opts.json) {
console.log(kleur.bold('\nagent prompt:'));
console.log(kleur.dim(prompt.slice(0, 300) + (prompt.length > 300 ? '…' : '')));
}

if (!opts.autoApply && !opts.json) {
const readline = await import('node:readline/promises');
const rl = readline.createInterface({ input: process.stdin, output: process.stdout });
const ans = await rl.question(kleur.yellow('\napply agent changes? [y/N] '));
rl.close();
if (ans.trim().toLowerCase() !== 'y') {
record.status = 'skipped';
record.finishedAt = new Date().toISOString();
await appendRun(record);
console.log(kleur.dim('skipped'));
return;
}
}

// Invoke agent
const result = spawnSync(agentBin, ['--print', prompt], {
encoding: 'utf8',
stdio: opts.json ? ['ignore', 'pipe', 'pipe'] : 'inherit',
});
Comment on lines +261 to +264
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps Bot May 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 result.stdout is always null when stdio is 'inherit', so record.diff will always be stored as '' in the normal interactive flow. The agent's output goes to the terminal but is never captured. Only --json mode (which uses 'pipe') would actually populate the diff. To capture output while still streaming to the terminal, use ['inherit', 'pipe', 'inherit'] so stdout is piped while stderr still streams.

Suggested change
const result = spawnSync(agentBin, ['--print', prompt], {
encoding: 'utf8',
stdio: opts.json ? ['ignore', 'pipe', 'pipe'] : 'inherit',
});
const result = spawnSync(agentBin, ['--print', prompt], {
encoding: 'utf8',
stdio: opts.json ? ['ignore', 'pipe', 'pipe'] : ['inherit', 'pipe', 'inherit'],
});

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!


if (result.error || result.status !== 0) {
record.status = 'error';
record.error = result.error?.message ?? `exit ${result.status}`;
record.finishedAt = new Date().toISOString();
await appendRun(record);
if (opts.json) { console.log(JSON.stringify({ run: record }, null, 2)); return; }
console.error(kleur.red(`\nagent failed: ${record.error}`));
console.log(kleur.dim(`hint: install ${agentBin} or try --dry-run to preview the prompt`));
return;
}

record.status = 'applied';
record.diff = result.stdout ?? '';
record.finishedAt = new Date().toISOString();
await appendRun(record);

if (opts.json) { console.log(JSON.stringify({ run: record }, null, 2)); return; }
console.log(kleur.green('\nagent cycle complete'));
console.log(kleur.dim(`run id: ${record.id}`));
console.log(kleur.dim('next: sh1pt build && sh1pt promote ship --channel beta'));
});

iterateCmd
Expand All @@ -68,10 +292,75 @@ iterateCmd
.option('--cloud', 'schedule and run the watch loop in sh1pt cloud')
.option('--interval <seconds>', 're-check interval', Number, 3600)
.option('--quiet-hours <start-end>', 'e.g. 22-08 (24h local) to pause overnight')
.action((opts) => {
console.log(kleur.cyan(`[stub] iterate watch ${JSON.stringify(opts)}`));
// TODO: long-running process hitting cloud API for fresh metrics every interval,
// invoking `iterate run` when a configured threshold trips.
.option('--stop', 'disable the current watch configuration')
.option('--status', 'show current watch configuration')
.action(async (opts: {
agent: string;
cloud?: boolean;
interval: number;
quietHours?: string;
stop?: boolean;
status?: boolean;
}) => {
if (opts.stop) {
await clearWatchConfig();
console.log(kleur.yellow('iterate watch disabled'));
return;
}

if (opts.status) {
const cfg = await loadWatchConfig();
if (!cfg) {
console.log(kleur.dim('iterate watch is not configured — run `sh1pt iterate watch` to start'));
return;
}
console.log(kleur.bold('\niterate watch config:'));
console.log(` agent: ${kleur.cyan(cfg.agent)}`);
console.log(` interval: ${kleur.cyan(String(cfg.interval))}s`);
if (cfg.quietHours) console.log(` quiet-hours: ${kleur.cyan(cfg.quietHours)}`);
console.log(` cloud: ${cfg.cloud ? kleur.green('yes') : kleur.dim('no (local)')}`);
console.log(` enabled: ${kleur.dim(cfg.enabledAt)}`);
if (cfg.lastRunAt) console.log(` last run: ${kleur.dim(cfg.lastRunAt)}`);
if (cfg.quietHours && inQuietHours(cfg.quietHours))
console.log(kleur.yellow(' ⏸ currently in quiet hours'));
return;
}

const cfg: WatchConfig = {
agent: opts.agent,
interval: opts.interval,
quietHours: opts.quietHours,
cloud: !!opts.cloud,
enabledAt: new Date().toISOString(),
};
await saveWatchConfig(cfg);

console.log(kleur.bold('\niterate watch configured'));
console.log(` agent: ${kleur.cyan(cfg.agent)}`);
console.log(` interval: ${kleur.cyan(String(cfg.interval))}s (${Math.round(cfg.interval / 60)} min)`);
if (cfg.quietHours) console.log(` quiet: ${kleur.cyan(cfg.quietHours)}`);

if (opts.cloud) {
console.log(kleur.bold('\ncloud mode:'));
console.log(kleur.dim(' the watch loop runs in sh1pt cloud infrastructure'));
console.log(kleur.dim(' it will fire `sh1pt iterate run` on a scheduled interval'));
console.log(kleur.dim(' cloud credentials must be configured via `sh1pt scale up`'));
console.log(`\n ${kleur.dim('deploy with:')} sh1pt scale deploy --cloud`);
} else {
console.log(kleur.bold('\nlocal mode:'));
console.log(kleur.dim(' add the following line to your crontab (`crontab -e`):'));
const intervalMin = Math.max(1, Math.round(cfg.interval / 60));
const cron = cfg.quietHours
? `# quiet ${cfg.quietHours}: adjust hours to taste`
: '';
if (cron) console.log(kleur.dim(` # ${cron}`));
console.log(kleur.white(` */${intervalMin} * * * * sh1pt iterate run --agent ${cfg.agent} --auto-apply`));
console.log();
console.log(kleur.dim(' or run a one-shot cycle now with:'));
console.log(kleur.white(` sh1pt iterate run --agent ${cfg.agent}`));
}
console.log();
console.log(kleur.dim('use `sh1pt iterate watch --stop` to disable'));
});

iterateCmd
Expand Down
Loading