HeroDB

HeroDB is a modular database server that provides a unified Redis-compatible interface to interact with various storage backends. It acts as a protocol-aware proxy which allows to query different underlying storage engines.

Under the hood, HeroDB implements the RESP3 protocol, ensuring compatibility with existing tools like redis-cli. The architecture is designed to be extensible, enabling future additions of new storage backends while maintaining a consistent query interface.

Supported Backends

HeroDB currently supports three distinct storage backends, each tailored for specific use cases. HeroDB can host multiple database instances of different backend types simultaneously. You do not need to choose just one; you can have an in-memory cache, a persistent key-value store, and a vector database all running within the same server process, accessible via different database indices.

• InMemory (INMEMORY): A volatile, high-performance key-value store. Ideal for caching, temporary data, and testing. Data is lost when the server stops. More information can be found in the in-memory backend documentation.
• Redb (REDB): A persistent, embedded key-value store. Provides ACID transactions and crash safety. Suitable for durable data storage where a full SQL database is overkill. More information can be found in the redb backend documentation.
• LanceDB (LANCEDB): A persistent vector database. Optimized for storing and searching vector embeddings. Perfect for AI/ML applications requiring similarity search (KNN). More information can be found in the lancedb backend documentation.
• Sled (SLED): A high-performance, embedded key-value store. Offers a modern, lock-free architecture suitable for high-concurrency workloads. More information can be found in the sled backend documentation.
• Tantivy (TANTIVY): A full-text search engine library. Allows indexing and searching text documents. More information can be found in the tantivy backend documentation.

Command Structure & Namespacing

Commands in HeroDB are context-aware based on the currently selected database.

• Namespacing: Commands can optionally be prefixed with the backend's namespace (e.g., LANCEDB ADD vs ADD).
  • If you provide a namespace (e.g., LANCEDB ADD), HeroDB validates that it matches the current backend. If you try to run REDB SET on a LanceDB instance, it will fail.
  • If you omit the namespace (e.g., ADD), HeroDB automatically routes it to the correct handler for the active backend.
• Universal Commands: Some commands like PING, LIST, SELECT, and COMMANDS are global and work regardless of the selected backend.

Security & Instances

HeroDB supports a security model based on Visibility (Public/Private) and Capabilities (ReadOnly/ReadWrite). This allows you to secure your database instances with access keys and granular permissions.

For a detailed explanation of how database instances and the security model work, please refer to the Instance Documentation.

Data-at-Rest Encryption

HeroDB provides transparent data-at-rest encryption using XChaCha20-Poly1305 authenticated encryption. This ensures that sensitive data stored in key-value backends is protected with modern, secure cryptography.

Supported Backends

Encryption is supported for key-value backends only:

• ✅ InMemory - Full encryption support
• ✅ Redb - Full encryption support
• ✅ Sled - Full encryption support
• ❌ LanceDB - Not supported (vector search requires unencrypted data)
• ❌ Tantivy - Not supported (full-text search requires unencrypted data)

Key Features

• Per-Instance Encryption: Each database instance can be independently configured for encryption.
• Stateless Mode: Clients provide their encryption key per-session via the USEKEY command. The server never stores the key.
• Transparent Operation: Once a key is set, all SET/GET operations automatically encrypt/decrypt data.
• Authenticated Encryption: XChaCha20-Poly1305 provides both confidentiality and integrity verification.
• Secure Key Handling: Keys are zeroized from memory when dropped.

Quick Example

# Generate a new encryption key
> KEYGEN
"YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY="

# Set the key for this session
> USEKEY YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=
OK

# Select an encrypted instance (user instances start at 1)
> SELECT 1
OK

# Data is automatically encrypted/decrypted
> SET secret "my sensitive data"
OK

> GET secret
"my sensitive data"

Encryption Commands

Command	Description
KEYGEN	Generate a new random 256-bit encryption key
USEKEY <key>	Set the encryption key for the current session
CLEARKEY	Clear the encryption key from the session
KEYINFO	Show current encryption state

For more information please see the Encryption Documentation.

Administrative Database (DB0)

HeroDB uses an encrypted administrative database at index 0 (DB0) to store all metadata about database instances. This provides secure, persistent storage of:

1. Instance Metadata: Names, backend types, creation timestamps
2. Encryption Keys: Per-instance encryption keys for encrypted databases
3. Access Keys: Authentication keys and permissions for private instances

DB0 Features

• Encrypted at Rest: Entire DB0 file encrypted using XChaCha20-Poly1305
• Admin Secret Required: HeroDB cannot start without the --admin-secret password
• Automatic Bootstrap: DB0 is created automatically on first startup
• Persistent: All metadata survives restarts
• Reserved Index: Index 0 is reserved for DB0; user instances start at index 1

DB0 File Structure

<data-dir>/
├── 0.salt          # Salt for admin-secret key derivation (16 bytes)
├── 0.db.enc        # Encrypted administrative database
└── *.redb/sled     # User database instance files

Starting HeroDB

Both --data-dir and --admin-secret are required:

herodb --data-dir ./my_data --admin-secret "your-secure-password"

On first startup, DB0 is bootstrapped. On subsequent startups, DB0 is decrypted using the admin-secret to load all instance metadata.

API Access Methods

HeroDB provides three ways to access the JSON-RPC administrative API:

1. HTTP (default port 3000) - For remote access and web applications
2. Unix Domain Socket (default /tmp/herodb.sock) - For local clients with better performance
3. RESP Protocol (default port 5555) - Redis-compatible interface for data operations

Configuration

HeroDB can be configured using the following command-line arguments:

Argument	Short	Description	Required	Default
--port	-p	TCP port for the RESP server (Redis interface)	No	5555
--api-port		HTTP port for the JSON-RPC API server	No	3000
--socket-path		Unix socket path for JSON-RPC API server	No	/tmp/herodb.sock
--data-dir		Directory for persistent storage and DB0	Yes	-
--admin-secret		Password for encrypting/decrypting DB0	Yes	-

Getting Started

HeroDB requires an admin secret and data directory to start. By default, it starts with no user database instances (only DB0 exists at index 0).

1. Start the Server: Build and run with required arguments.

   cargo build --release
   ./target/release/herodb --data-dir ./my_data --admin-secret "secure-password"
   

   • RESP Server (Redis interface): Port 5555
   • HTTP API Server (Management interface): Port 3000
   • UDS API Server (Management interface): /tmp/herodb.sock
   • DB0: Automatically bootstrapped at ./my_data/0.db.enc

2. Health Check: Verify the API server is running.

   Via HTTP:

   curl -X POST -H "Content-Type: application/json" -d '{
     "jsonrpc": "2.0",
     "method": "health_check",
     "params": {},
     "id": 1
   }' http://localhost:3000/
   

   Via Unix Socket (requires socat):

   echo '{"jsonrpc":"2.0","method":"health_check","id":1}' | socat - UNIX-CONNECT:/tmp/herodb.sock
   

   Expected response:

   {"jsonrpc":"2.0","result":"OK","id":1}
   

3. Create a Database: Use the JSON-RPC API to create a new instance. For example, to create an In-Memory cache named my-cache:

   curl -X POST -H "Content-Type: application/json" -d '{
     "jsonrpc": "2.0",
     "method": "db_createInstance",
     "params": {
       "name": "my-cache",
       "backend_type": "InMemory"
     },
     "id": 1
   }' http://localhost:3000/
   

   To create a persistent Redb instance:

   curl -X POST -H "Content-Type: application/json" -d '{
     "jsonrpc": "2.0",
     "method": "db_createInstance",
     "params": {
       "name": "my-store",
       "backend_type": { "Redb": "my_data/store.redb" }
     },
     "id": 2
   }' http://localhost:3000/
   

   To create an encrypted instance (encryption is immutable):

   curl -X POST -H "Content-Type: application/json" -d '{
     "jsonrpc": "2.0",
     "method": "db_createInstance",
     "params": {
       "name": "secure-store",
       "backend_type": { "Redb": "my_data/secure.redb" },
       "encryption_enabled": true
     },
     "id": 3
   }' http://localhost:3000/
   

   Note

   : Encryption settings are immutable and can only be set at creation time.

4. Connect: Use any Redis client (e.g., redis-cli) to connect to the RESP port.

   redis-cli -p 5555
   

5. List Databases: Verify your new instance exists and find its index.

   LIST
   

   Output example:

   1) "1: my-cache (INMEMORY)"
   

   Note

   : User instances start at index 1. Index 0 is reserved for DB0 (administrative database).

6. Select: Switch to the database (using the index from LIST).

   SELECT 1
   

7. Query: Run commands supported by the backend.

   SET user:1 "HeroDB User"
   GET user:1
   

8. Security & Management: You can use the API to change an instance's visibility to Private and manage access keys.

   • Method: db_updateSecurity
   • Params: instance_index, visibility, add_keys, remove_keys

   See Instance Documentation for full API details.

9. Encryption (For Encrypted Instances): To use encryption with an encrypted instance:

   # Select the encrypted instance
   SELECT 2
   
   # Generate a key (or use an existing one)
   KEYGEN
   
   # Set the key for your session
   USEKEY <your-base64-key>
   
   # Now SET/GET operations are automatically encrypted
   SET secret "sensitive data"
   GET secret
   

   Important: Encryption settings are immutable. They must be set at instance creation and cannot be changed later.

   See Encryption Documentation and Encryption Walkthrough for full details.