fix(amp-worker-core): exclude hot files from compaction groups#1969
Open
fix(amp-worker-core): exclude hot files from compaction groups#1969
Conversation
Previously, the cooldown mechanism only prevented hot files (younger than cooldown_duration) from starting new compaction groups, but allowed them to be swept into existing groups started by cold files. This caused an incident where a recently-written backfill segment was compacted into a large file containing a reorg'd block, regressing dataset progress by millions of blocks. Hot files are now fully excluded from compaction groups — they can neither start nor join any group until their cooldown period elapses.
Theodus
changed the title
Collaborator
|
Current behaviour is intentional. In the principle that if we're compacting then we might as well try to pack the group as optimally as possible. If very recent files gen 0 should be excluded due to the reorg problem, that should imo have a specific mechanism for preventing compaction of recent files. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously, the cooldown mechanism only prevented hot files (younger than cooldown_duration) from starting new compaction groups, but allowed them to be swept into existing groups started by cold files. This caused an incident where a recently-written backfill segment was compacted into a large file containing a reorg'd block, regressing dataset progress by millions of blocks.
Hot files are now fully excluded from compaction groups — they can neither start nor join any group until their cooldown period elapses.