feat(studio,cli): music beat detection with timeline guides + headless beats CLI

[vanceingalls]

Summary

Adds music beat detection to HyperFrames. The Studio draws beat guides on the timeline; beats are user-editable and persist to a per-audio project file; and a new hyperframes beats CLI generates that file headlessly so guides are ready before the Studio is ever opened.

Beat detection is music-only: an <audio data-timeline-role="music"> (or an id like music/bgm/soundtrack) is analyzed; voiceover and other audio are excluded by the shared isMusicTrack rule.

What's included

Detection — @hyperframes/core/beats (shared by Studio + CLI)

• Energy-onset detector cross-validated with bpm-detective. When the two tempos agree, beats are regularized to an octave-aligned grid (so a half-time reading doesn't double the density); otherwise raw onsets are used.
• Silence gating drops beats in intro/outro/quiet regions.
• Each beat carries a loudness 0–1 (local RMS ÷ track peak), used for dot/line brightness.
• bpm-detective is loaded lazily (it touches window at module top-level) so the module is safe to import in Node/SSR/tests.

Studio

• Full-height green beat lines behind the tracks + green dots on the music track's row (brightness/size scale with loudness).
• Add beat at the playhead (drum-icon toolbar button next to split), drag to move, double-click to delete. Strength is always measured from the audio — never set by hand.
• Editing (and playhead scrubbing) scrubs the music audio at 25% volume.
• Undo/redo for beat ops, interleaved with the file-history stack by timestamp.
• Edits persist to beats/<audio>.json in audio-file coordinates, so they survive trimming/moving the music clip and are matched back to the same audio if it's removed and re-added.
• Clip edges snap to the beat grid (clamped to the clip's minDuration and source/composition limits).

CLI — hyperframes beats [dir]

• Finds the music track, runs the same detection in headless Chrome (identical Web Audio decode + bpm-detective), and writes the beat file the Studio loads as-is.
• The browser bundle is prebuilt into dist at CLI build time (build:beat-analyzer), so it works for published users without shipping source; a dev fallback bundles from core source.
• Surfaces in-page errors, guards oversized inputs, and points at browser ensure when Chrome is missing.

Beat file format

{ "version": 1, "audio": "music.wav", "beats": [{ "time": 1.23, "strength": 0.8 }] }

Notes

• Results are near-identical between CLI and Studio (both Chrome); a different headless audio sample rate can shift beat times by a frame or two.
• examples/ is gitignored, so the demo composition/audio used during development isn't part of this PR.

Testing

• core (1466), studio (798), cli typecheck/lint all green; pre-commit hooks (lint, format, fallow complexity gate, filesize, typecheck) pass.
• Verified end-to-end: hyperframes beats on a demo composition analyzes the music track only (voiceover excluded) and writes a file the Studio reloads with edits intact.

🤖 Generated with Claude Code

Timeline UX refinements (follow-up)

• Center-anchored magnify — zooming the timeline via the toolbar/slider keeps the time at the viewport center fixed instead of anchoring at the left edge; pinch-to-zoom still anchors at the cursor.

• Move-snap to beats — dragging a clip snaps whichever edge (start or end) is nearest a beat, matching the existing resize-edge snapping.

• Beat lines on track backgrounds — faint full-height beat lines now paint behind the clips on every track lane (brightness scales with loudness); the green dots stay on the active track's top bar.

• Waveform follows zoom — waveform bars fill the full clip width and resample the windowed peaks, so the waveform stretches with zoom instead of stopping partway across a widened clip.

• Dots centered in the top bar — the dot band aligns to the clip top so the dots sit vertically centered in the dark top bar instead of being bisected by the clip's selection border.

• Trim survives moving other clips — re-applies the cached probe sourceDuration on every element re-derivation, so moving a non-music clip no longer drops the music clip's source length (which made the trimmed waveform reset to the full source pinned at the track start).

• Music excluded from beat-snap — moving/trimming the music track no longer snaps to its own beats (it defines them).

• Move-snap target highlight — while dragging a non-music clip, the beat its edge will snap to is drawn as a bright neon line, so the snap target is visible before drop (the floating clip follows the cursor; the lit line is the cue).

How the beat analysis works

Pipeline order (packages/core/src/beats/beatDetection.ts, analyzeMusicFromBuffer). Everything operates in audio-file seconds, not timeline seconds.

1. Decode — fetch the music file, decode to PCM via Web Audio (decodeAudioData), work on channel 0 (mono).

2. Onset detection (our detector) — slide a 1024-sample window with a 512-sample hop; per window compute energy (mean of squared samples). A window is an onset when it's a local peak and exceeds 1.5× the local mean of the surrounding ±20 windows (adaptive threshold). Debounce: reject onsets within 100ms of the previous. Output = where the transients actually are — accurate in position, slightly irregular.

3. Onset BPM — take the median inter-onset interval (robust to misses/outliers) → 60 / median. Grounded in real hits, but octave-ambiguous (half/double time).

4. bpm-detective (independent verifier) — handed the same AudioBuffer. It low-pass filters, finds rhythmic peaks, histograms peak-to-peak intervals, and returns a single global BPM (no positions), folded into ~90–180. Loaded lazily (touches window at import) and wrapped in try/catch so non-browser/low-peak cases degrade gracefully. It is a tempo verifier/tie-breaker — it never places beats.

5. Cross-validation — compare onsetBPM vs detectiveBPM octave-safely (fold both into 60–120, take % diff):

   • < 5% → agreement → confidence: high: trust detective's tempo, re-pick its octave to match our onset pulse.
   • 5–10% → low: average the two.
   • > 10% → uncertain: fall back to our onset BPM, don't force a grid.
   • Only one available → use it at low.

6. Octave alignment (octaveAlignBpm) — detective folds tempo into a band, so test ÷2/×1/×2 and pick the octave whose beat interval is closest to our onset pulse. Prevents a half-time reading from producing a double-density grid.

7. Regularize to a grid (regularizeBeats) — with a trusted tempo, beat interval = 60/bpm. Phase-fit: try the first ~10 onsets as anchors, keep the phase that aligns the most raw onsets within ±25% of a beat, then emit an evenly-spaced grid anchored to the real hits. (Skipped when tempo is uncertain — raw onsets are kept.)

8. Silence gate + strength (gateBeatsBySilence) — measure local RMS (±50ms) at each grid beat; drop any below 12% of the loudest beat's RMS (kills phantom beats in intros/outros/quiet sections). Each surviving beat gets a strength 0–1 (RMS ÷ peak) that drives dot size and line brightness.

9. Result (MusicBeatAnalysis) — beatTimes + beatStrengths, bpm + bpmConfidence, and the retained channelData/sampleRate/peak so strength can be measured on demand when the user adds a beat by hand (strength is always read from audio, never typed).

In short: our energy-onset detector provides position truth; bpm-detective provides an independent tempo truth; agreement between them earns high confidence, after which we octave-align, snap to a phase-fit grid, gate out silence, and score each beat's loudness.

• Beat-drag hides the playhead — dragging a beat dot hides the playhead guideline so its line doesn't track the scrub and clutter the beat being moved.
• Dots default to the music track — beat dots render on the selected track, falling back to the music track when nothing is selected.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
hyperframes	🟢 Ready	View Preview	Jun 14, 2026, 2:22 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

[vanceingalls]

• feat(studio): drag keyframes with live beat snapping #1439
• feat(studio,cli): music beat detection with timeline guides + headless beats CLI #1424 👈 (View in Graphite)
• fix(core,player,studio): bound trimmed audio playback to the clip window #1430
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[james-russo-rames-d-jusso]

Verified at HEAD b2fc792 — Approve-with-concerns. Strong PR; determinism contract holds, wire contract for #1439 is clean, a couple of follow-up items below.

What ships
• @hyperframes/core/beats — shared detection (energy onsets + bpm-detective cross-validation, octave alignment, grid regularization, silence gate, strength normalization) + persistence (beats/<audio>.json, version: 1).
• Studio: timeline guides (full-height green lines + dots), add/move/delete beats, undo/redo, debounced persist, audio scrub during edits.
• CLI: hyperframes beats [dir] runs the SAME detection in headless Chrome (prebuilt beat-analyzer.global.js bundled at CLI build time, dev fallback bundles from core source).

Determinism contract — PASS
Detection lives in core/src/beats and is consumed only by studio/src/hooks/useMusicBeatAnalysis.ts (author-time, in the editor) and cli/src/commands/beats.ts (offline). Grepped: no imports from engine, player, producer, sdk render-path packages. The render-time contract is the persisted JSON file (pure read). Date.now() is studio undo-stack only (playerStore.ts:2369); setTimeout is studio playback-scrub + persist-debounce; the two fetch() calls in beatDetection.ts are inside detectBeatsFromUrl/analyzeMusicFromUrl which only the studio hook + CLI invoke. bpm-detective is lazy-imported (touches window at top-level — important since it's now in core's deps).

Wire contract for downstream #1439
HF#1439's drag-with-beat-snap will consume:
• usePlayerStore(s => s.beatAnalysis) — MusicBeatAnalysis | null (exported from @hyperframes/core/beats).
• usePlayerStore(s => s.beatEdits) — BeatEditState | null.
• remapBeatAnalysisToComposition(beatAnalysis, musicElement, beatEdits) in studio/src/utils/beatEditActions.ts — returns composition-coordinate beat times filtered to clip range.
Times in beatAnalysis.beatTimes are audio-file seconds; consumers must remap via playbackStart/start offset (the helper does this correctly). #1439 should consume the helper, not the raw selector — flag if it doesn't.

Concerns
• beatFile.ts:parseBeats — doesn't gate on data.version === 1. A future v2 file with breaking schema would be parsed silently as v1 (since the BeatFileData cast in JSON.parse is unchecked). One-liner: if (data?.version !== 1) return null; before reading data.beats. Cheap insurance for the migration you'll inevitably do when you add confidence scores or beat strength variants.
• useMusicBeatAnalysis.ts:analysisCache retains the decoded channelData: Float32Array for up to 4 music files. Typical 3-min stereo at 48kHz mono = ~35MB raw; cap = ~140MB worst-case across project switches. Likely fine in practice but worth a // memory note on the constant — not a blocker.
• Determinism across machines: PR body acknowledges "a different headless audio sample rate can shift beat times by a frame or two" between CLI-generated and Studio-generated beat files. So CI-generated beat files can't be compared byte-for-byte against locally-generated ones in tests. Author-aware; mention here for downstream consumers.
• CodeQL ReDoS alert open on beatFile.ts:17 against the /^(blob:|data:)/i prefix check — anchored alternation on a 5-char prefix is linear and not exploitable. Probably worth dismissing in the GH UI with the "false positive — anchored prefix" rationale so the alert doesn't follow the file forever.

Nits
• MAX_AUDIO_BYTES = 80 * 1024 * 1024 hardcoded — fine default, but consider a --max-bytes CLI flag for power users with long WAV stems. (nit)
• findMusicAudioSrc (beatFile.ts) and isMusicTrack (studio/src/utils/timelineInspector.ts) both encode the music-detection rule. PR body says they agree; comment in one file pointing at the other would close the loop. (nit)
• BeatFileData interface is module-private — fine, but exporting the type would let downstream tools (Vance's future MCP server?) consume the file format without duplicating the shape. (nit)

Stack notes
• #1439 (drag with beat snapping) depends on the wire contract above. Stable; reviewer there should verify it consumes remapBeatAnalysisToComposition, not raw beatTimes.
• #1430 (trimmed audio playback) sits downstack of this one and touches musicElement.playbackStart / playbackEnd — heavily used by remapBeatAnalysisToComposition here. Worth a sanity cross-check that trim changes correctly invalidate / re-derive the beat composition mapping.
• #1448 (clip-relative GSAP positions, merged) — no conflict; touches a different timeline axis.

CI
All required green except 2 pending regression shards (shard-2, mergeability_check) and the Perf: ${{ matrix.shard }} placeholder failure (matrix-expansion artifact, not a real check). Preflight, typecheck, perf shards (drift/fps/load/parity/scrub), and 6/8 regression shards pass.

Review by Rames D Jusso

[miguel-heygen]

Strengths

1. CodeQL self-remediation (packages/core/src/beats/beatFile.ts:18-29): The PR introduced a ReDoS-vulnerable regex in an earlier commit and then fixed it in the same branch before merging -- the HEAD already uses indexOf/slice with a single-char-class [?#] strip. The inline comment even names the CWE. Clean.

2. Beat edit undo/redo interleaving (packages/studio/src/hooks/useAppHotkeys.ts:55-79, playerStore.ts:250-279): Timestamp-based dispatch between the beat undo stack and the file-history stack is an elegant solution to the "two independent undo domains" problem. The logic is correct: commit stores {restore: prevState}; undo restores and pushes {restore: currentState} to redo; redo mirrors it. No phantom state.

3. Resource cleanup discipline (useMusicBeatAnalysis.ts:109-152, useTimelinePlayer.ts:279+557): Both setTimeout instances are cleaned on effect teardown; stopScrubPreviewAudio is called both on play() and in the unmount return of useTimelinePlayer; the analysis useEffect sets a cancelled flag and the persist effect calls flush() before tearing down. The scrub timer chain (applyScrub -> clearTimeout -> setTimeout(stopScrub, 140)) also properly clears in stopScrubPreviewAudio.

---

Audited (read end-to-end):

• packages/core/src/beats/beatDetection.ts
• packages/core/src/beats/beatFile.ts
• packages/core/src/beats/index.ts
• packages/cli/src/beats/headlessAnalyzer.ts
• packages/cli/src/commands/beats.ts
• packages/studio/src/hooks/useMusicBeatAnalysis.ts
• packages/studio/src/utils/beatEditing.ts
• packages/studio/src/utils/beatEditActions.ts
• packages/studio/src/utils/timelineInspector.ts
• packages/studio/src/player/store/playerStore.ts (beat slice)
• packages/studio/src/player/lib/timelineIframeHelpers.ts (scrub section)
• packages/studio/src/player/lib/playbackScrub.ts
• packages/studio/src/player/lib/mediaProbe.ts

Trusting (integration points only):

• packages/studio/src/player/components/BeatStrip.tsx -- React component, pointer event wiring matches setBeatDragging store calls
• packages/studio/src/player/components/TimelineCanvas.tsx -- prop threading only
• packages/studio/src/player/components/TimelineRuler.tsx -- SVG beat lines, density guard confirmed
• packages/studio/src/player/components/useTimelineClipDrag.ts -- snap logic reviewed; resolveTimelineResize integration trusted
• packages/studio/src/player/components/useTimelinePlayhead.ts -- center-anchor zoom; skipCenterAnchorRef flag pattern reviewed
• packages/studio/src/player/hooks/useTimelinePlayer.ts -- probe refactor trusted; cleanup calls confirmed
• packages/studio/src/player/lib/timelineDOM.ts -- timelineRole attribute threading trusted
• packages/cli/scripts/build-beat-analyzer.mjs -- esbuild IIFE config trusted
• packages/studio/src/components/TimelineToolbar.tsx -- toolbar button wiring trusted
• packages/studio/src/hooks/useAppHotkeys.ts -- timestamp interleaving audited; broader hotkey logic trusted
• docs/packages/cli.mdx, packages/studio/vite.config.ts, lockfile -- mechanical changes trusted

---

Blockers

None.

---

Important

1. No unit tests for the new @hyperframes/core/beats module (packages/core/src/beats/). Every other core subsystem has a *.test.ts sibling (e.g. lint/, parsers/, compiler/). The beat detection pipeline has non-trivial algorithmic logic (detectBeats, regularizeBeats, gateBeatsBySilence, the onset/detective cross-validation). Pure functions + AudioBuffer mocks would exercise these cheaply.

   Similarly, packages/cli/src/commands/beats.ts and packages/cli/src/beats/headlessAnalyzer.ts have no .test.ts. Every other CLI command in the suite has one (add.test.ts, batchRender.test.ts, etc.). At minimum unit tests for resolveMusicTarget, report, and the MAX_AUDIO_BYTES guard in analyzeBeatsHeadless would be straightforward without requiring a real Chrome.

2. bpm-detective declared as prod dep in both packages/core/package.json and packages/studio/package.json. Studio pulls @hyperframes/core as a workspace dep, so it already resolves bpm-detective transitively. The explicit studio dep exists to feed optimizeDeps.include in vite.config.ts, but Vite can pre-bundle transitive deps without a direct declaration. The studio dep is harmless but redundant and could cause version drift if the two entries diverge. Consider removing it from studio and keeping only the core dep.

---

Nits

• beatDetection.ts:140: fallow-ignore-next-line complexity on regularizeBeats is acceptable, but extracting the inner phase-fit loop to findBestPhaseOffset would make the complexity annotation less of a catch-all and the function easier to test in isolation.

• headlessAnalyzer.ts:24-32 (findPrebuiltBundle): The third candidate path join(here, "../dist/beat-analyzer.global.js") appears unreachable given the tsup output layout (only the first two paths are live). A brief comment on the expected layout would prevent future confusion.

• timelineIframeHelpers.ts:196-201: Module-level mutable state for the scrub audio is fine for a singleton browser context, but HMR in dev (module reloads; iframe does not) could leave the audio element muted if stopScrubPreviewAudio is not called before the reload. Not a prod bug; just a dev-ergonomics note worth a comment.

• BeatStrip.tsx:36: key={${t}-${i}} -- two beats at the exact same time (e.g. a bad manual edit circumventing the MATCH_EPS guard) would produce duplicate keys. Using i alone as the key would be more defensive.

---

Verdict: APPROVE

Reasoning: All CI checks pass (all regression shards green; the "fail" in the check list is a cancelled Perf runner, not a code error); the CodeQL alert was on an earlier commit and is already remediated in HEAD; resource cleanup is thorough across all new effects and module-level state; the undo/redo interleaving logic is correct. The missing unit tests for core/beats and the CLI beats command are a genuine gap relative to the project's test culture but not a merge blocker for a feature PR of this scope.

-- Magi

The base branch was changed.