Observing immudb Internals – A Tour

⭐️ immudb on github!

The latest version of immudb, the open source immutable database, comes with many improvements around instrumentation and observability. For this release we’ve put a lot of effort into optimizing our btree (our internal data structure) implementation and were able to achieve 22% higher indexing performance and a 22% reduction in disk usage.

Working on performance is usually a tricky job. When we think about performance of our code it’s very likely that our intuition will not reveal the true bottlenecks, no matter how much we think we know them. Therefore, it is crucial to have good internal visibility into what’s happening in the system running on real production data.

immudb exposes many internal metrics through the prometheus metrics endpoint. You can easily query it with curl or in your browser:

$ curl -s http://localhost:9497/metrics | head
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.3355e-05
go_gc_duration_seconds{quantile="0.25"} 1.3615e-05
go_gc_duration_seconds{quantile="0.5"} 1.9991e-05
go_gc_duration_seconds{quantile="0.75"} 3.0348e-05
go_gc_duration_seconds{quantile="1"} 3.3859e-05
go_gc_duration_seconds_sum 0.000151623
go_gc_duration_seconds_count 7
# HELP go_goroutines Number of goroutines that currently exist.

Embedding prometheus monitoring in a golang application exposes a ton of internal go-related metrics out of the box. This itself is a gold mine for performance tuning. However we’ve also added dedicated metrics to get true visibility onto what’s going on inside immudb, especially when observing and tuning btree performance.

Of course querying metrics with a simple curl command is not a very practical solution. Hopefully we’ve got very robust tools to visualize such values in an understandable and visually appealing way.

Starting with immudb 1.2.3 we’ve added a Grafana dashboard with correct visualization of some key immudb metrics. This dashboard can be downloaded from https://github.com/codenotary/immudb/blob/master/tools/monitoring/grafana-dashboard.json. We will be updating this dashboard with additional metrics in future releases.

Let’s take a closer look at the available charts.

Database Size Section

This section shows general information about database size and growth over time.

Database Size / Database Growth

These two dashboards show the amount of disk space used by databases and its growth. It can be used to forecast disk usage and make sure it’s not getting out of control.

Stored Entries / Stored Entries Rate

These dashboards show the amount of new DB entries and the insertion rate. The amount of new entries is calculated since the recent database restart.

Indexer Statistics

immudb has a unique feature of asynchronous indexing. It can ingest data at a very high speed and index them in the background. Not-yet-indexed data can not be easily queried (it can still be read by specifying explicit transaction id), thus ensuring that the indexing can catch up with the data insertion rate is crucial for keeping the application healthy.

Indexed %

This graph shows the total percentage of all transactions in the database that has been indexed so far. In a healthy situation, this chart should remain at or close to 100%. If this value starts dropping down, that means that the data ingestion rate is higher than the indexing rate indicating that additional rate limiting should be added to db writers.

Indexing / Commit rate

This chart shows the rate of new transactions added to the database and the rate of indexing those transactions. If the indexing rate is smaller than the commit rate, this means that the database isn’t keeping up with the indexing. In applications where only synchronous writes are performed or where data can be immediately indexed, the indexing rate line (Idx) and commit rate line (Cmt) will overlap.

TRXs Left to Index

This chart shows the number of transactions waiting for indexing. This value should be close to zero and should have a decreasing tendency.

TRX Count

This chart shows the total number of transactions in the database.

Btree Cache Size / Btree Cache Hit %

Those two charts show internal statistics about immudb btree cache. In order to avoid reading large amounts of data on every btree operation, immudb keeps an in-memory cache of recently used btree nodes. The amount of nodes in the cache is shown in the first chart, it is capped at the maximum amount of cache entries.

The second chart shows how effective the cache is presenting the percentage of btree node lookups that were optimized with the cache. For small databases, it’s very likely that this hit ratio will be close to 100%, but it will drop down once the amount of data increases. There’s no single rule on what value we should expect here. In our internal tests even 40% cache hit ratios with workloads using keys with random distribution were still yielding very good performance results. To get higher cache utilization, applications should prefer working on keys close to themselves – such as using sequentially increasing numbers where newly inserted data will end up in the btree portion close to previously accessed entries.

Btree Insights

This section of the dashboard contains a detailed view into btree internals.

Btree Depth

This chart shows the depth of the tree. Since btrees are auto-balancing data structures, this depth will have a logarithmic tendency. The depth of the tree indicates what’s the amount of nodes traversed by each btree operation.

Btree Child Node Count Distributions

These graphs show the distribution of the amount of child nodes. In a healthy btree like the one below, the amount of child nodes should be focused around a single value (40 in the example). Also the amount of child nodes should be kept at sane levels – values below 10 or above few hundred are a good indication that the btree isn’t performing well and the application should consider using keys of different, more uniform and shorter lengths for its data.

Note: These statistics are gathered when traversing the btree, if there’s no activity in the database the distribution will be flat.

Flush Statistics

immudb keeps recent btree changes in memory to reduce the amount of data to be written to disk. In order to persist those changes, there’s a btree flush process called once a threshold of new and modified entries is reached.

These metrics are calculated for nodes (both inner and leaf ones) and separately for KV entries in the leaf nodes.

The flush rate shows the rate of written nodes / entries per second. It clearly shows where the flush process started and where it ended.

Flush progress metrics for each flush cycle starts at zero and reach the total amount of entries processed during such single flush operation. The next flush will repeat the process by starting from zero reaching the maximum value. By looking at those maximum values, we can see how much data needs to be written to disk during flush operations. During normal DB operation, it should be steady over time. An unbound growth of those maximums could indicate that the flush operation is too aggressive and the threshold should be adjusted.

Compaction Statistics

Similarly to flush statistics, immudb exposes the same set of values for full compaction.

Note: these values are gathered for overall compaction that fully rewrites the btree structure. immudb 1.2.3 introduced a new online compaction mode that gradually removes unused btree data during flush operation. This new compaction mode is not included in those charts.

Data Size for Btree Nodes

immudb internally uses append-only files to store data. That is also used for btree nodes. We don’t replace existing data on disk. Instead we append a new modified version at the end of the data stream. immudb 1.2.3 introduced new online compaction of btree data that deals with garbage that naturally accumulates at the beginning of such data stream over time. This chart can be used to ensure that the amount of data being used for btree nodes is being reduced whenever a cleanup operation is performed.

S3 Statistics

This section is in its early days but already shows some basic insights into what the data performance is using immudb with an AWS S3 backed data store. The first chart shows the histogram of a single s3 object read duration , the second one shows the inbound traffic needed for immudb operations. If those values are high, you should consider switching back to local disk and an alternative remote storage back-end such as EBS volumes.

We hope you enjoyed this tour through the instrumentation and internals of immudb. Join us on our Discord if you have any questions or are deploying immudb and need help. If you’re a data geek, a golang lover or enjoy working on some of the world’s most interesting problems join our team. As always don’t forget to ⭐️ immudb on github!

CNIL Metrics & Logs

Self-Hosted performance monitoring and compliant log analysis for VMware vSphere, container and much more.

immudb

Built on the fastest immutable ledger technology. Open Source and easy to use and integrate into existing application.

Codenotary Cloud

Trusted CI/CD, SBOM and artifact
protection with cryptographic proof.
One CLI to manage all.

Subscribe to Our Newsletter

Get the latest product updates, company news, and special offers delivered right to your inbox.

Subscribe to our newsletter

Use Case - Tamper-resistant Clinical Trials

Goal:

Blockchain PoCs were unsuccessful due to complexity and lack of developers.

Still the goal of data immutability as well as client verification is a crucial. Furthermore, the system needs to be easy to use and operate (allowing backup, maintenance windows aso.).

Implementation:

immudb is running in different datacenters across the globe. All clinical trial information is stored in immudb either as transactions or the pdf documents as a whole.

Having that single source of truth with versioned, timestamped, and cryptographically verifiable records, enables a whole new way of transparency and trust.

Use Case - Finance

Goal:

Store the source data, the decision and the rule base for financial support from governments timestamped, verifiable.

A very important functionality is the ability to compare the historic decision (based on the past rulebase) with the rulebase at a different date. Fully cryptographic verifiable Time Travel queries are required to be able to achieve that comparison.

Implementation:

While the source data, rulebase and the documented decision are stored in verifiable Blobs in immudb, the transaction is stored using the relational layer of immudb.

That allows the use of immudb’s time travel capabilities to retrieve verified historic data and recalculate with the most recent rulebase.

Use Case - eCommerce and NFT marketplace

Goal:

No matter if it’s an eCommerce platform or NFT marketplace, the goals are similar:

  • High amount of transactions (potentially millions a second)
  • Ability to read and write multiple records within one transaction
  • prevent overwrite or updates on transactions
  • comply with regulations (PCI, GDPR, …)


Implementation:

immudb is typically scaled out using Hyperscaler (i. e. AWS, Google Cloud, Microsoft Azure) distributed across the Globe. Auditors are also distributed to track the verification proof over time. Additionally, the shop or marketplace applications store immudb cryptographic state information. That high level of integrity and tamper-evidence while maintaining a very high transaction speed is key for companies to chose immudb.

Use Case - IoT Sensor Data

Goal:

IoT sensor data received by devices collecting environment data needs to be stored locally in a cryptographically verifiable manner until the data is transferred to a central datacenter. The data integrity needs to be verifiable at any given point in time and while in transit.

Implementation:

immudb runs embedded on the IoT device itself and is consistently audited by external probes. The data transfer to audit is minimal and works even with minimum bandwidth and unreliable connections.

Whenever the IoT devices are connected to a high bandwidth, the data transfer happens to a data center (large immudb deployment) and the source and destination date integrity is fully verified.

Use Case - DevOps Evidence

Goal:

CI/CD and application build logs need to be stored auditable and tamper-evident.
A very high Performance is required as the system should not slow down any build process.
Scalability is key as billions of artifacts are expected within the next years.
Next to a possibility of integrity validation, data needs to be retrievable by pipeline job id or digital asset checksum.

Implementation:

As part of the CI/CD audit functionality, data is stored within immudb using the Key/Value functionality. Key is either the CI/CD job id (i. e. Jenkins or GitLab) or the checksum of the resulting build or container image.

White Paper — Registration

We will also send you the research paper
via email.

CodeNotary — Webinar

White Paper — Registration

Please let us know where we can send the whitepaper on CodeNotary Trusted Software Supply Chain. 

Become a partner

Start Your Trial

Please enter contact information to receive an email with the virtual appliance download instructions.

Start Free Trial

Please enter contact information to receive an email with the free trial details.