feat(regulatory): classify and publish regulatory actions

[lspassos1]

Summary

This adds the second regulatory RSS step by classifying normalized actions into high / medium / low, computing aggregate counts, and publishing the final payload to regulatory:actions:v1 via runSeed.

Root cause

The fetch/parse seeder from #2564 produces stable normalized actions, but the pipeline still needed tiering logic and Redis publication before cross-source signals could consume regulatory events.

Changes

• add the HIGH_KEYWORDS and MEDIUM_KEYWORDS classification rules from feat(regulatory): seed-regulatory-actions.mjs — classify + Redis write #2493
• classify each action with tier and matchedKeywords, using word-boundary matching to avoid false positives like ban inside bank
• build the final seed payload with actions, fetchedAt, recordCount, highCount, and mediumCount
• wire main() through runSeed('regulatory', 'actions', 'regulatory:actions:v1', ...) with TTL 7200 and an empty-array-safe validateFn
• expand tests/regulatory-seed-unit.test.mjs to cover classification, payload counts, fetch-to-payload flow, and the runSeed wiring

Validation

• node --test tests/regulatory-seed-unit.test.mjs
• node -e "import('./scripts/seed-regulatory-actions.mjs').then(async (m) => { const data = await m.fetchRegulatoryActionPayload(); process.stdout.write(JSON.stringify({recordCount: data.recordCount, highCount: data.highCount, mediumCount: data.mediumCount, first: data.actions[0]}, null, 2) + '\\n'); })"
• node -e "import('./scripts/seed-regulatory-actions.mjs').then(() => process.stdout.write('import-ok\\n'))"

Risk

Low risk. This only evolves the new regulatory seeder introduced in #2564 and adds focused test coverage around the new classification/publish behavior.

Note

This branch is currently stacked on #2564. Until #2564 merges, this PR includes the parent fetch/parse commit in the diff.

Depends on #2564
Closes #2493
Refs #2494
Refs #2495

[vercel]

@lspassos1 is attempting to deploy a commit to the Elie Team on Vercel.

A member of the Team first needs to authorize it.

[greptile-apps]

Greptile Summary

This PR completes the second step of the regulatory RSS pipeline by adding keyword-based tiering (high/medium/low) to normalized regulatory actions and publishing the resulting payload to regulatory:actions:v1 via runSeed with a 2-hour TTL. Classification uses word-boundary regex matching to avoid false positives (e.g. ban inside bank), and the final payload includes highCount/mediumCount aggregate fields for downstream consumers. The implementation is well-structured and the test coverage is thorough.

Key findings:

• Missing bootstrap hydration (api/bootstrap.js) — AGENTS.md requires all new data sources to be wired for bootstrap hydration; regulatory:actions:v1 is absent from api/bootstrap.js, meaning first page loads will not receive regulatory data until a cache-miss fetch cycle completes.
• fetch convention violation — All four functions that accept a fetchImpl parameter default to globalThis.fetch directly (fetchImpl = globalThis.fetch). AGENTS.md explicitly requires the (...args) => globalThis.fetch(...args) arrow-wrapper pattern instead. This is consistent with how other seed scripts in the repo invoke fetch inline via globalThis.fetch(url, opts).
• FINRA feed over plain HTTP — All other feed URLs use HTTPS; the FINRA entry uses http://feeds.finra.org/FINRANotices.
• Regex pre-compilation — findMatchedKeywords compiles a new RegExp for each keyword on every classifyAction call; since the keyword lists are constant, pre-compiling at module load would be a minor efficiency improvement.

Confidence Score: 4/5

• Safe to merge after wiring bootstrap hydration in api/bootstrap.js; all other findings are style/convention issues.
• One P1 finding: regulatory:actions:v1 is not wired in api/bootstrap.js, which AGENTS.md marks as a MUST for new data sources. Without it, the key is invisible to first-load hydration. The remaining findings (fetch convention, HTTP URL, regex pre-compilation) are all P2 style/best-practice items that do not affect runtime correctness in the Node.js Railway environment where these scripts execute.
• scripts/seed-regulatory-actions.mjs — bootstrap wiring and fetch convention; api/bootstrap.js — needs regulatory:actions:v1 entry added.

Important Files Changed

Filename	Overview
scripts/seed-regulatory-actions.mjs	New regulatory seed script: fetches 5 agency RSS/Atom feeds, classifies actions as high/medium/low via word-boundary keyword matching, and publishes to regulatory:actions:v1 via runSeed. Issues: globalThis.fetch used as raw default parameter (AGENTS.md convention), FINRA feed uses plain HTTP, bootstrap hydration not wired in api/bootstrap.js.
tests/regulatory-seed-unit.test.mjs	Thorough unit tests covering entity decoding, HTML stripping, RSS/Atom parsing, deduplication, classification tiers, payload construction, and runSeed wiring; uses vm.runInContext to test pure functions without live network or Redis calls.

Sequence Diagram

sequenceDiagram
    participant Cron as Railway Cron
    participant Main as main()
    participant Utils as _seed-utils / runSeed
    participant Feeds as Regulatory Feeds (5)
    participant Redis as Upstash Redis

    Cron->>Main: trigger seed-regulatory-actions.mjs
    Main->>Utils: runSeed('regulatory','actions','regulatory:actions:v1', fetchFn, opts)
    Utils->>Utils: acquireLock('regulatory:actions')
    Utils->>Main: invoke fetchFn()
    Main->>Main: fetchRegulatoryActionPayload()
    Main->>Feeds: Promise.allSettled(fetchFeed × 5)
    Feeds-->>Main: RSS/Atom XML responses
    Main->>Main: parseFeed() → normalizeFeedItems()
    Main->>Main: dedupeAndSortActions()
    Main->>Main: classifyAction() × N (high/medium/low)
    Main->>Main: buildSeedPayload() → {actions, fetchedAt, recordCount, highCount, mediumCount}
    Main-->>Utils: resolved payload
    Utils->>Redis: atomicPublish('regulatory:actions:v1', payload, TTL=7200s)
    Utils->>Redis: writeFreshnessMetadata('regulatory','actions')
    Utils->>Redis: releaseLock()

Loading

_{Reviews (1): Last reviewed commit: "feat(regulatory): classify and publish r..." | Re-trigger Greptile}

[lspassos1]

Follow-up on the bootstrap thread: I am intentionally leaving regulatory:actions:v1 out of api/bootstrap.js here. That key is an internal seed input for seed-cross-source-signals.mjs, not a client-facing bootstrap source. The UI hydrates crossSourceSignals directly, and that key is already in bootstrap. Adding regulatory:actions:v1 would only increase bootstrap payload size with unused data; it would not change first-load behavior for the cross-source panel.

[koala73]

Review — PR #2567 (classify + Redis write)

Why this PR? Classifies fetched regulatory actions by keyword matching and writes to Redis.

Blocking

1. TTL = cron interval (1x). Must be >= 3x.
TTL_SECONDS = 7200 with a 2h cron means a single missed Railway cycle = data disappears. Every other seeder in the project uses >= 3x. Fix: TTL_SECONDS = 21600 (6h).

2. Missing regulatory:actions:v1 in server/_shared/cache-keys.ts.
Per the 4-file registration checklist. The health dashboard can't track this key without it.

3. Keyword model misses obvious enforcement headlines.
The classifier is title-only with a narrow vocabulary. Tested against live CFTC feed: headlines like "CFTC resolves action against..." and "permanent injunction" classify as low because the keywords don't cover enforcement verbs (resolve, enjoin, injunction, consent order, cease-and-desist, debarment, suspension, revocation). These are the exact "market-moving regulatory actions" the feature is meant to surface, but they never reach the cross-source signal path because #2568 drops all low items.

Fix: Expand the keyword list to include enforcement action verbs:

• high: injunction, cease and desist, consent order, debarment, suspension order, revocation
• medium: resolves action, settled charges, administrative proceeding, remedial action

Also consider classifying against description (not just title) if PR #2564 extracts it. Many agencies put the enforcement detail in the description, not the headline.

4. notice as a medium keyword over-classifies routine items.
notice matches every FINRA "Regulatory Notice" and "Information Notice" which are routine administrative publications, not enforcement. These then flow into #2568 as policy signals and participate in composite escalation, creating false positives.

Fix: Either remove notice from medium keywords, or add negative patterns (e.g., skip if title matches /^(Regulatory|Information|Technical) Notice/).

5. tier: 'low' conflates "genuinely low priority" with "unclassifiable".
An item that matches no keywords gets low, but so does a genuinely low-priority item. The project has a pattern for this (classification-default-unknown): default to unknown and let downstream consumers decide whether to include or exclude unknowns.

Suggestions

• Pre-compile keyword regexes at module load (some already are, ensure all are)
• Add // @ts-check

[koala73]

Replaced by #2691 which is properly rebased on main after #2564 merged. The classification layer now builds on the existing fetch/parse code instead of duplicating it.