Skip to content

Add Firebase Cleanup cron#3508

Merged
isTravis merged 30 commits intomainfrom
tr/firebase-clean
Mar 2, 2026
Merged

Add Firebase Cleanup cron#3508
isTravis merged 30 commits intomainfrom
tr/firebase-clean

Conversation

@isTravis
Copy link
Member

@isTravis isTravis commented Feb 26, 2026

Firebase Cleanup Tool

Implements automated cleanup of Firebase to reduce storage costs.

What it should do

  • Prunes old data from changes/, checkpoints/, checkpointMap/, checkpoint, and merges/ before the safe threshold
  • Safe threshold: min(latestCheckpointKey, latestReleaseHistoryKey) — preserves all steps needed for release creation (any discussions made on a release are fast-forwarded through steps to align on the new release, so we have to preserve steps since release in case there are discussions)
  • Fast-forwards discussions before pruning to prevent broken anchors
  • Removes orphaned drafts (Drafts with no corresponding Pub in postgres)
  • Removes orphaned Firebase paths (Firebase data with no associated Pub in postgres)

CLI Usage

# Dry-run all drafts (default)
pnpm run tools cleanupFirebase

# Execute cleanup
pnpm run tools cleanupFirebase --execute

# Test single pub
pnpm run tools cleanupFirebase --pubId <uuid>

# Set batch size
pnpm run tools cleanupFirebase --batchSize 100

Scheduled

Runs on a regular interval set via cron in tools/cron.ts.


Testing Checklist

1. Dry-run mode

  • Run pnpm run tools cleanupFirebase in dev
  • Verify it logs what would be deleted without making changes
  • Confirm Firebase data is unchanged after dry-run

2. Single pub test

  • Create a test pub with multiple edits/checkpoints
  • Run with --pubId <test-pub-id> --execute
  • Verify old changes/checkpoints are pruned
  • Verify latest checkpoint and document state are preserved

3. Discussion fast-forwarding

  • Create a pub with inline discussions at various anchor points
  • Make edits that would cause discussions to need fast-forwarding
  • Run cleanup with --execute
  • Verify all discussions still appear at correct positions
  • Open editor and confirm discussions are functional

4. Release preservation

  • Create a pub with one or more releases
  • Run cleanup
  • Verify can still create new releases (no "historyKey not found" errors)
  • Verify existing releases still accessible

5. Orphan cleanup

  • Identify a Draft ID that exists in Firebase but not Postgres (or create one for testing)
  • Run cleanup and verify orphan is detected
  • With --execute, verify orphan is removed from Firebase

6. Edge cases

  • Pub with no checkpoints (should skip gracefully)
  • Pub with no releases (should use checkpoint key only)
  • Legacy path format (pub-*/branch-*) — verify handled correctly
  • New path format (drafts/draft-*) — verify handled correctly

7. Production dry-run

  • Run dry-run on production
  • Review logged output for expected behavior
  • Estimate storage savings from logged deletion counts

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an automated Firebase cleanup tool and schedules it via cron to reduce Realtime Database storage usage/costs while attempting to preserve editor/release functionality.

Changes:

  • Registers a new cleanupFirebase CLI tool under pnpm run tools ....
  • Schedules a weekly production cron job to run Firebase cleanup with --execute.
  • Implements tools/cleanupFirebase.ts to prune old Firebase history data, fast-forward discussions, and remove orphaned drafts/paths.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 9 comments.

File Description
tools/index.js Adds cleanupFirebase to the tools command registry.
tools/cron.ts Schedules weekly production execution of the cleanup tool.
tools/cleanupFirebase.ts Implements the Firebase cleanup/pruning + orphan detection/removal logic.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 7 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@isTravis isTravis requested a review from Copilot February 26, 2026 21:44
@isTravis isTravis marked this pull request as ready for review February 26, 2026 21:45
@isTravis isTravis requested review from 3mcd and tefkah February 26, 2026 21:45
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 10 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@isTravis isTravis merged commit 962e324 into main Mar 2, 2026
1 of 2 checks passed
@isTravis isTravis deleted the tr/firebase-clean branch March 2, 2026 20:59
@isTravis isTravis restored the tr/firebase-clean branch March 3, 2026 18:44
@isTravis isTravis deleted the tr/firebase-clean branch March 3, 2026 18:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants