Odysseia - LMDB

Odysseia - LMDB
Photo by benjamin lehman / Unsplash

Lightning Memory-Mapped Database (LMDB) is a software library that provides a high-performance embedded transactional database in the form of a key-value store. This is a perfect fit for storing data produced by a blockchain because it is space efficient and vastly superior in terms of performance compared to something like PostgreSQL but also comes with a variety of downsides for external clients.

Pros

  • Just works, doesn't need anything to be installed on the system like for a traditional RDBMS.
  • Blazingly fast due to being a key-value store that doesn't care about data types or relationships between rows.
  • Very space efficient because data can be stored as binary and safe up to 50% of the original size.

Cons

  • The data you store can be difficult to consume by a client because filtering like with SQL queries is not possible out of the box.
  • Without a cursor you end up iterating over all rows which can cause a performance bottleneck because there is no filtering or indices like with SQL.
  • Has less to no safe-guards compared to an RDBMS which can result in unexpected bugs like duplicate rows because it isn't caught by a unique clause.

All the cons are heavily dependant on your use-case because things like client consumption of your data might not even a concern to you. If it is you should think twice about using LMDB and if you do I would advise you to spin up an external service that indexes the data into an RDBMS or ElasticSearch which will keep the data available in a more easily digestible format for clients to quickly filter and access.

Data Consumption

If you are running a blockchain you'll need to expose all of the data in one way or another to clients so that they can build applications like wallets or analytical dashboards.

The issue with LMDB is that you have no way of filtering data besides jumping to a range of rows with the start and end arguments. This allows targeting ranges but that generally isn't too useful for a client unless it is making a fully copy of the blockchain through an API. If you need something more specific like getting data for a wallet you'll need to iterate over all rows and do the filtering in-memory at runtime which can take a long time if you have millions of rows.

The easy way out would be to let the community build some use-case specific tools around the core but they will generally end up being unmaintained or not follow best-practices which will make peoples lives harder when interacting with the data of your blockchain without building another tool that fits their exact use-case.

The way it's being solved in Odysseia is that an external indexing service will be built. This service will be completely detached from Odysseia itself and pull all data from the LMDB databases to store it in an RDBMS for easy consumption through a REST API. This allows Odysseia itself to make use of the most performant tech without having to consider how difficult it is for people to hook into the system and make use of the data.

Iterating Data

Every time you launch Odysseia it will perform a full reconstruction of the data that is stored to make sure it matches what is expected. This is done in order to avoid that a node is launched in a corrupted state but comes at a big performance cost with LMDB because of how the underlying system handles the bootstrapping and verification of the data.

With an RDBMS you can perform aggregate SQL queries that will give you a a quick summary of all the data you are looking for. With LMDB this isn't possible because it is a simple key-value store that can't query specific fields. This means that you'll need to iterate over all keys and grab what you need from the associated value. If you have tens of millions of rows this can slow you down if you are only interested in certain rows like for example transactions of a specific type.

The most obvious solution would be to separate the rows into type-specific databases and reduce the amount of rows you need to iterate over. This is easy to do with LMDB because you have a root database and child databases which can all operate independently of each other.

The issue with this solution is that you'll end up with segregated databases and if you for some reason need to iterate over all transactions you would need to iterate over all rows in all databases, then merge those rows and finally sort them by their sequence to ensure they are all in the correct order because you can no longer rely on the insertion order of the rows.

The approach that's taken in Odysseia is that we'll only loop over all blocks and transactions once and perform all tasks that need to be performed inside of those two loops. This means that we keep O(n) as low as possible which will keep memory use also as low as possible because we only need to deserialise every block and transaction once. This approach will probably be tweaked in the future as Odysseia runs against a database with tens of millions of blocks and transactions.

Lack of safe-guards

When using something like PostgreSQL you have all kinds of safe-guards like the data-type of a column or unique constraints. None of those exist but that has pros and cons and is a trade-off for performance vs. security.

The performance benefit is that data can be inserted instantaneously because there aren't hundreds of checks that need to be performed to ensure that data is allowed to be inserted. The lack of those checks means that the burden of ensuring duplicate rows are not created is on your application.

It's important to keep a balance when adding those checks in your application code. If you add too many of them you will negate the performance benefits of LMDB, at which point you might as well use an RDBMS. Ideally you would use something like a value object which will ensure the data is properly formatted before you even attempt to insert it because this object will already be part of your usual application flow and not add any overhead at the time of inserting the data.

As Odysseia is based on the 3.0 release of ARK Core we are already working with value objects that contain their contents serialised as a Buffer. We can take this buffer and insert it as-is because we store blocks and transactions as binary values to optimise how much disk space is consumed per row.