New core index: deleted.index

[staltz]

💡 Idea

When we're streaming the log in jitdb, we can detect deleted records, but we don't really do anything about it, we just skip to the next records. What we could do is update a bitvector deleted.index to indicate that such record is deleted. Then, this can alleviate other code downstream, such that we don't have to get the record from the log, verify that it's a deleted record (which by the way uses the is-buffer-zero module), and then drop it from the query results. We can just do a bitvector AND with deleted.index.

Since deletes can happens in random order, we will also have to listen to a stream of deleted offsets from AAOL. Such API doesn't currently exist in AAOL but could be somewhat easily built, maybe as an enhancement of AAOL.onDeletesFlushed().

cc @arj03

[arj03]

True, this could be a good speedup, especially if you have many deleted messages. Could just be a pre-filter, we still have the check in get which is good to keep there in case things are not totally in sync.