OLTP vs. Traditional Databases
- The world is shifting towards more transactional processing, leading to the need for redesigned OLTP systems.
- Traditional open-source OLTP databases like MySQL and SQLite struggle with the current scale and volume of transactions.
- The current approach of "more with more" scalability leads to increased operational costs and complexity.
TigerBeetle: A High-Performance OLTP Database
- TigerBeetle is an open-source distributed financial transactions database optimized for OLTP.
- It aims to improve performance by three orders of magnitude using the same or less hardware.
- TigerBeetle uses a Double Entry accounting schema, allowing multiple business transactions in a single database transaction for improved performance and scalability.
- It employs a fixed-size data structure and static memory allocation to minimize memory bandwidth usage.
- TigerBeetle utilizes a replicated state machine architecture for data consistency across multiple machines in a cluster.
- It eliminates row-level locks by keeping hot transactional data local to each CPU core, reducing contention and enhancing performance.
- TigerBeetle scales diagonally by combining vertical scaling for compute and hot storage with horizontal scaling for colder remote storage.
- It employs an LSM Forest instead of an LSM Tree to optimize writes, reads, and memory for different data types.
- TigerBeetle uses a just-in-time compaction algorithm for eliminating right stalls and providing a hard guarantee on performance.
- It assumes storage faults do happen and employs various detection and recovery techniques to ensure data integrity.
- TigerBeetle's storage engine uses a deterministic compaction algorithm for efficient recovery from local faults.
- It integrates its storage engine with the consensus protocol for optimized network processing and reduced replication round trips.
- TigerBeetle utilizes Viewstamped Replication (VSR) for consensus, offering advantages over Raft.
- It extends the consensus log with a cryptographic hash chain for recovery even with corrupt logs in different places.
- TigerBeetle implements protocol-aware recovery to handle storage faults and prevent data loss.
- An optimization is in progress to reduce replication round trips by exploiting the fact that most operations will always get a quorum.
Performance and Safety
- TigerBeetle can process up to 988,000 transactions per second on NVMe with primary indexes and 200,000 to 500,000 transactions per second with 20 secondary indexes.
- It employs NASA's Power of 10 rules for safety-critical code, incorporating assertions and resource limits for reliability.
- TigerBeetle is designed as a deterministic distributed database, allowing for comprehensive testing and verification of its consensus protocol and storage engine.
- The simulation testing environment enables accelerated time simulation for years of testing in a matter of days.
- A live demo showcases a simulated Tiger Beetle cluster running on the Golden Gate Bridge, demonstrating faultless operation.
- Network faults, disk corruption, and cosmic ray simulations are used to test the database's resilience and correctness.
- TigerBeetle includes an "Easter egg" duck mode to simulate OLTP workloads under OLAP conditions.
