notes

# Log of decisions (problems / solutions):

[done]

# Problem 1: search is slow, have to go through the whole file to find last key/value.
# Solution: build index in memory pointing to byte position of value.

# Problem 2: the client looses its index once the command is run.
# Solution 1: have an index loaded in a instance always available.
# Solution 2: add repl to interact with the db.

# Problem 3: many lines for the same key can make the file size greater than it should be
# Solution 1: offer an operation to compact it.

# Problem 5: compaction does not replace current data
# Solution 1: offer an option to update it.

# Problem 7: when new key/value is added by another process, local index is not updated
# Solution 1: add how many bytes were read, and build index from there.

# Problem 4: compact operation is slow
# Solution 1: offer a compact operation that writes bytes in bulk.

# Problem 11: tests use the same database as repl
# Solution 1: update tests to use a new one for each execution.
# Note: now repl can use test database as well.

# Problem 10: building index step is slow
# Solution 1: read the whole file once instead of reading byte per byte.

# Problem 6: is it really faster with index? how much?
# Solution 1: Benchmarked it. Below the results:
# It takes around 4 minutes to build index of a dataset of 1GB in Python.
# Switching to a similiar algorithm in C++, takes around 1 minute and 30 seconds.
# Searching after index is built in both languages take microseconds to return in the same dataset.

# Problem 9: dictionary footprint of index seems too high
# Solution 1: index was rewritten in C++.

# Problem 13: c++ is not as productive and error-free as I'd like
# Solution: I decided to rewrite in Rust and benchmark it against cpp. Below, the results:
------------------------------------------------------------------
type (n = 10000)     | write average (sec)  | read avg (sec)
------------------------------------------------------------------
cpp                  | 7.174320220947266e-05 | 5.5613517761230467e-05
rust                 | 4.976601600646973e-05 | 3.898470401763916e-05
------------------------------------------------------------------
# The Rust compiler really helps to get the code safer.

# Problem 14: clean up references to C++ source code
# Solution: removed C++ code as well as benchmark code

# Problem 12: compact operation was removed
# Solution: reimplemented in Rust.

# Problem 15: it should be possible to pass the database path to the library or the cli
# Solution: a new parameter was added to new instances of DBL class

[todo]

# Problem 8: multiple processes can write at the same time and mess with data in disk
# Solution 1: add lock for writes (allow many to read).