Skip to content

fix: Implement RSS feed circuit breaker#141

Open
seer-by-sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/rss-circuit-breaker
Open

fix: Implement RSS feed circuit breaker#141
seer-by-sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/rss-circuit-breaker

Conversation

@seer-by-sentry

Copy link
Copy Markdown
Contributor

Description

This PR addresses issue TUVIX-API-21, where the Tuvix API was logging a high volume of HTTP 522 (and other Cloudflare-related) errors due to repeatedly attempting to fetch from persistently failing RSS feed URLs.

Root Cause: The system lacked a mechanism to identify and temporarily disable feed sources that consistently failed to fetch, leading to continuous retries of broken URLs on every cron run.

Solution Implemented:
Introduced a circuit breaker pattern for RSS feed fetching:

  1. Schema Update: Added consecutive_failures (integer), last_error_at (timestamp), and fetch_disabled_at (timestamp, nullable) columns to the sources table in packages/api/src/db/schema.ts.
  2. Database Migration: Created a Drizzle migration (0012_add_source_failure_tracking.sql) to safely add these new columns and an index on fetch_disabled_at.
  3. **Failure Tracking Logic (packages/api/src/services/rss-fetcher.ts):
    • Success: On a successful feed fetch, the consecutive_failures count is reset to 0, and last_error_at and fetch_disabled_at are cleared.
    • Failure: When fetchSingleFeed encounters an error, consecutive_failures is incremented, last_error_at is updated, and lastFetched is advanced to prevent immediate re-queueing. If consecutive_failures reaches a threshold (currently 10), fetch_disabled_at is set, effectively disabling the feed.
    • Exclusion from Fetching: The getStaleSources query now excludes feeds where fetch_disabled_at is set. This prevents the system from attempting to fetch from known broken sources.
    • Cooldown Period: Disabled feeds are automatically re-enabled after a 24-hour cooldown period (if fetch_disabled_at is older than 24 hours) to allow for potential recovery of the external feed source.

This change significantly reduces the error noise from persistently failing feeds and optimizes resource usage by avoiding unnecessary fetch attempts.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test update
  • Build/CI update
  • Other (please describe):

Related Issues

Fixes #
Relates to #TUVIX-API-21

Changes Made

  • Introduced a circuit breaker pattern for RSS feed fetching.
  • Added consecutive_failures, last_error_at, and fetch_disabled_at columns to the sources table in packages/api/src/db/schema.ts.
  • Created a Drizzle migration (0012_add_source_failure_tracking.sql) to safely add these new columns and an index on fetch_disabled_at.
  • Implemented failure tracking logic in packages/api/src/services/rss-fetcher.ts to manage successful fetches, increment failures, disable feeds, and re-enable them after a cooldown period.
  • Modified the getStaleSources query to exclude feeds that are currently disabled.

Testing

  • Tested locally
  • Added/updated unit tests
  • All tests pass (pnpm test)
  • Type checking passes (pnpm type-check)
  • Linting passes (pnpm lint)

Screenshots/Videos

Documentation

  • Documentation has been updated (if needed)
  • Code comments added for complex logic
  • README updated (if needed)

Checklist

  • My code follows the project's coding standards
  • I have performed a self-review of my code
  • I have added tests that prove my fix is effective or that my feature works

Additional Notes

Fixes TUVIX-API-21

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants