HDDS-9154. Design doc for snapshot diff optimization.#10335
HDDS-9154. Design doc for snapshot diff optimization.#10335SaketaChalamchala wants to merge 3 commits into
Conversation
SaketaChalamchala
added
snapshot
| ### 2.3. UpdateID Gating | ||
| **Baseline Issue:** The baseline performs full object comparisons including timestamps to detect modifications, which is susceptible to clock skew and is computationally expensive. | ||
| **Optimized Design:** Uses the `dbTxSequenceNumber` of the `fromSnapshot` as a strict gate. During the `toSnapshot` scan, entries are only considered candidates for diff if their `updateID > fromSnapshotDbTxSequenceNumber`. |
There was a problem hiding this comment.
updateId comes from Ratis trxnLogIndex , dbTxSequenceNumber is RocksDB concept. Pls double check if those are comparable?
There was a problem hiding this comment.
Thanks for pointing this out @smengcl. updateID and dbTxSequenceNumber are not comparable. It is more likely that dbTxSequenceNumber faster than updateID and we may miss some valid diff entries with this comparison updateID > fromSnapshotDbTxSequenceNumber.
A more suitable gate can be updateID > fromSnapshot's lastTransactionInfo.txnIndex. This may only need to be an optimization on the full diff path.
For DAG based diff, we can read the entry's sequence number using the raw SST iterators and compare with fromSnapshotDbTxSequenceNumber. This will always guarantee correctness regardless of any updateID bugs.
What do you think?
2 participants
What changes were proposed in this pull request?
This is a design doc for an optimized snapshot diff that relies on mostly sequential reads of the snapshot DBs and gets the diff directly from the delta files instead of the current implementation that does a lot of cross database random lookups.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9154
How was this patch tested?
N/A