refactor_create/seal_shard

[JacksonYao287]

1 change create/seal shard to log only , so no push_data happens for these two requests.
2 add a check for on_commit put_blob to avoid putting blob to a sealed shard.
3 move select_specific_chunk(mark chunk to inuse state) to on_commit of create_shard.
4 write shard footer for seal shard in baseline resync
5 add a sealed_lsn for shard meta blk, which is not compatible with previous version

[xiaoxichen]

can we simply use get_reserved_blks() ?
If we want to protect the case that reserved_bytes_in_chunk set to zero, make this logical into get_reserved_blks() or simply force it to reserve a few MB.

[JacksonYao287]

according to our current logic, only seal_shard can use the reserved space(default 4 blks), which can make sure the seal_shard can be done in any case.

if we allow create shard to use the reserved space, it will break the above consumption?
sorry, I am not quite clear about you suggestion, or let me ask this in another way, how to guarantee the blk can be successfully allocated if we use get_reserved_blks()?

[xiaoxichen]

I mean using the get_reserved_blks() which default to 16MB , if any chunk has less than get_reserved_blks() bytes we simply dont use it. Do not need to calculate the size of Shard Header.

[JacksonYao287]

here, I use const static on purpose to avoid it to be calculated every time. it will be calculated only once.

[xiaoxichen]

is it possbile we failed to get enough space in first emergent GC(which already cleared all written-but-not-referenced data)? Seems like we dont need a while(true).

[JacksonYao287]

theoretically, there are some very corner cases that we failed to get enough space in first emergent GC.

for example, if we have two stale push_data req and each of them will try to allocate 4 blks, and meanwhile, in the target chunk, there only left exact 4-blk free space, other blks are all valid.

1 the first stale push_data take 4 blks in the target chunk, which lead to the no_space_left when we try to allocate blk for shard header, and we trigger emergent gc to free 4 blks

2 the second stale push_data comes after emergent gc and take the 4 blks just freed, then when trying to allocate blk for shard header, the second no_space_left occurs.

[xiaoxichen]

Considering Leader will not create shard on the chunk if chunk avaialbe space is lower than given threshold, is this still possible?

[JacksonYao287]

Considering Leader will not create shard on the chunk if chunk avaialbe space is lower than given threshold

the available space of a chunk when committing create_shard might be different from that when being selected for creating_shard because of the staled push_data request(put_blob).

Actually, for both leader and follower, this corner case might happen. this root cause is that we don`t exactly know how many staled push_data request are in filght , and when they will try to allocate blk.

a while-true loop here will not bring too much cost, since almost all the alloc_blk will succeed in the first time.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 51.74129% with 97 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.21%. Comparing base (1746bcc) to head (c1d3e02).
⚠️ Report is 120 commits behind head on main.

Files with missing lines	Patch %	Lines
src/lib/homestore_backend/hs_shard_manager.cpp	51.66%	50 Missing and 8 partials ⚠️
src/lib/homestore_backend/hs_blob_manager.cpp	18.51%	19 Missing and 3 partials ⚠️
src/lib/homestore_backend/replication_message.hpp	46.15%	6 Missing and 1 partial ⚠️
...ib/homestore_backend/replication_state_machine.cpp	50.00%	4 Missing and 1 partial ⚠️
src/lib/homestore_backend/hs_pg_manager.cpp	0.00%	4 Missing ⚠️
...lib/homestore_backend/snapshot_receive_handler.cpp	95.23%	1 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #340      +/-   ##
==========================================
- Coverage   63.15%   58.21%   -4.94%     
==========================================
  Files          32       35       +3     
  Lines        1900     4301    +2401     
  Branches      204      519     +315     
==========================================
+ Hits         1200     2504    +1304     
- Misses        600     1533     +933     
- Partials      100      264     +164     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[JacksonYao287]

@xiaoxichen I have address your comments in the latest commit, ptal

[xiaoxichen]

lgtm.

Lets bake it in SH before merging. The thinking is we might have more tiny fixes to be merge recently , want to avoid taking this out to production.

[JacksonYao287]

lgtm.

Lets bake it in SH before merging. The thinking is we might have more tiny fixes to be merge recently , want to avoid taking this out to production.

sure, I will let you know after I have run storage hammer for several rounds and confident to merge it