'''
A single file embedded database for fun (and not profit). Use at your own risk.
Often the simplest solution that works in the happiest of cases is a good place to start. This
implementation is that. Don't expect blazing performance. Arguably, this code is thread safe -
https://web.archive.org/web/20201108091210/http://effbot.org/pyfaq/what-kinds-of-global-value-mutation-are-thread-safe.htm
But it is advisable to use _lock_ and not rely on this detail. This is an implementation detail of
the VM and not a language specification, therefore, implementations can vary, and so can the
correctness of your code if ported. Another reference.
https://stackoverflow.com/questions/2291069/is-python-variable-assignment-atomic/55279169#55279169.
To have the peace of mind, please un-comment the `with self.lock` statements.
Let's talk about the simplicity of the implementation: 1. Easy to understand. With three user facing
functions - get, set and delete, anyone with little python experience can understand this code. 2.
Implementation is correct (for most cases. Why most ? See the limitation section), i.e. if the
functions (set or delete) returns, you are guaranteed to see the bytes in your database file. We
fsync to ensure it.
It is important to understand the limitations of this implementation:
1. The size of the database is
limited by the size of your RAM and not the size of the disk.
2. Depending on the size of the
database, init can take a long time. These days one can easily get 16G of RAM on a laptop. so think
loading 16G of data into memory.
3. Even if you are inserting a single key or deleting one, you are
writing a multi GB database onto the disk again and again. Therefore, the throughput of your simple
database will be limited by your disk bandwidth.
4. Unless you have a potent filesystem, like ZFS
(https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/), if the file gets
corrupted, you will not be able to get back the database or worse get incorrect data. To close
this loophole, the dump function should compute a checksum and write a footer to store it. The
load should validate the checksum when reading the file.
Since we have talked about limitations, for completeness, let's discuss how can we make it efficient
without making it complicated:
1. Improving the write speed:
a. Instead of writing the full database every time, write a WAL. So, the set and delete just
updates the WAL with what's updated/added or deleted. Now just with this you have hugely
complicated your design :).
1. With the WAL your load must now read the last snapshot of the database from the disk and
apply the WAL on top.
b. Since a WAL can be grow really long and contains old updates ( think a key's value updated
three time, the WAL will contain all three but only the last one is active), it must be
periodically truncated. Anyone who has written a storage engine knows this is easier said
than done. You will likely take WAL checkpoints. and in a background thread, apply the
immutable checkpoint over the database file. Therefore, everytime the database boots up, it
has less WAL to replay.
With the above two, you should be able to get a descent key update throughput. Mostly, you will
be limited by the concurrency limitations of the Python VM.
Now, a 15GB of k/v data is still 15 GB of data. The __init__ will still be slow. What should we
do ? Well, not read the whole file at once. hmm.. lazy loading - This will take us to a
completely different solution space. Writing serialized json is no longer an option. When asked
for a key, we should be able to quickly look it up and return the value or say it does not
exist. We will most likely choose B+ tree implementation (LMDB) or LSM tree implementation
(RocksDB or LevelDB). And this will be the point at which you will be willingly immerse yourself
into all the complexity of building a storage engine.
Maybe you know that your k/v store will never grow beyond 1GB and you do not need anything at
all ? Or you know that your usecase is going to grow to 10s of GBs and your will embrace the
complexity out of necessity.
'''
import json
import os
import shutil
from tempfile import NamedTemporaryFile
import threading
from typing import Optional, Type
class PyDbBasic:
key_not_str_err = TypeError('key must be of string type')
def __init__(self, location) -> None:
self.loc = os.path.expanduser(location)
self.db = {}
self.lock = threading.Lock()
self._load()
def set(self, key: str, val: Type) -> bool:
# with self.lock:
self.db[key] = val
return self._dump()
def get(self, key: str) -> Optional[Type]:
try:
return self.db[key]
except KeyError:
return None
def delete(self, key: str) -> bool:
if key not in self.db:
return False
# with self.lock:
del self.db[key]
self._dump()
def items(self):
return self.db.items()
def _dump(self):
'''
Write the db dictionary object to the file atomically.
The data is first written to a temporary file. Then the file is renamed to the name chosen
by the user. Each update (set or delete) calls this method which writes a file to disk - not
very efficient if you want to handle millions of writes per second. But reads are served
from memory and therefore, plenty fast.
Writes are extremely in-efficin
'''
with NamedTemporaryFile(mode='wt', delete=False) as f:
json.dump(self.db, f)
# This file can be large and we cannot risk the data only residing in the os buffers and
# not on the disk. Think what happens the after this function returns and before the os
# decides to flush this, there is power outage. There will be data loss.
# The next line safe-guards against it.
os.fsync(f.fileno())
if os.stat(f.name).st_size != 0:
shutil.move(f.name, self.loc)
return True
else:
return False
def _load(self):
"""
Load the database file from the location provided, if the file is present.
If not, we start with an empty instance of a dict. The file will be written to on first
call to set.
"""
if os.path.exists(self.loc):
with open(self.loc, 'rt') as f:
try:
self.db = json.load(f)
except ValueError:
if os.stat(self.loc).st_size == 0:
self.db = {}
else:
raise
else:
self.db = {}