feat: Ability to re-include older episodes when max_age is increased#872
Open
mm503 wants to merge 1 commit into
Open
feat: Ability to re-include older episodes when max_age is increased#872mm503 wants to merge 1 commit into
mm503 wants to merge 1 commit into
Conversation
mm503
force-pushed
the
fix/support_expand_max_age
branch
2 times, most recently
from
6094c01 to
f34a9b9
Compare
Increasing a feed's max_age now pulls in older episodes that were previously out of range, instead of silently ignoring them. This required decoupling discovery depth from page_size and making episode re-inclusion work off the database rather than the API window. - Discovery depth is driven by max_age: a normal update still does a shallow scan (page_size most-recent episodes), but when max_age is expanded beyond what was previously scanned, the YouTube builder performs a one-time deep scan that pages back to the max_age cutoff. A per-feed ScannedThrough high-water mark gates this so it costs nothing in steady state, and it is bounded by a 20-page backstop. Brand-new feeds stay shallow. - Previously cleaned (soft-deleted) episodes that match the filters again are re-queued for download off the database records directly, independent of the page_size API window. Only episodes that would survive the keep_last cleanup are re-queued, to avoid a re-download/re-clean loop. - Cleanup no longer clears an episode's title/description when soft-deleting it, so a re-included episode keeps its metadata. For episodes cleaned by older versions (which wiped metadata), the title/description are recovered from the current feed query. - The feed build skips a downloaded episode with an empty title instead of aborting the whole feed, so one malformed item can no longer take down the entire feed. - A startup warning is logged when clean.keep_last exceeds page_size and no max_age is configured. Currently implemented for YouTube.
mm503
force-pushed
the
fix/support_expand_max_age
branch
from
f34a9b9 to
6d5c2e5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hey @mxpv, I have a bit controversial PR for you. On several occasions, I needed to raise the
max_agesetting for a channel to go farther in history, and as you probably know, there is a design limitation that makes it a bit hard.Increasing a feed's max_age now pulls in older episodes that were previously out of range, instead of silently ignoring them. This required decoupling discovery depth from
page_sizeand making episode re-inclusion work off the database rather than the API window to save API credits.Changes:
max_age: a normal update still does a shallow scan (page_sizemost-recent episodes), but when max_age is expanded beyond what was previously scanned, the YouTube builder performs a one-time deep scan that pages back to themax_agecutoff. A per-feedScannedThroughhigh-water mark gates this so it costs nothing in steady state, and it is bounded by a 20-page backstop. Brand-new feeds stay shallow.page_sizeAPI window. Only episodes that would survive thekeep_lastcleanup are re-queued, to avoid a re-download/re-clean loop.clean.keep_lastexceedspage_sizeand nomax_ageis configured.It was a bit of a journey where I went through damaging some DB records on my side all the way to repairing them and having a behavior that doesn't shred through API credits unnecessarily.