lhumina_code/hero_lib

Fork 1

implement a logger we can reuse everywhere #133

New issue

Closed

opened 2026-05-01 03:47:18 +00:00 by despiegk · 4 comments

despiegk commented

2026-05-01 03:47:18 +00:00

Owner

in /Volumes/T7/code0/hero_lib/crates/core/src/logger

Hero Logs — Fast Append-Only Logging Spec

Goal

Create a very fast, simple, human-readable logging system.

No database.
No Tantivy.
No indexer for now.
Just reliable local log capture.

Log directory

~/hero/var/logs/

Each source has maximum 3 dotted parts:

src = "service.job.worker"

Mapped to:

~/hero/var/logs/service/job/worker.log

Examples:

router             → ~/hero/var/logs/router.log
router.http        → ~/hero/var/logs/router/http.log
router.http.proxy  → ~/hero/var/logs/router/http/proxy.log

If source has more than 3 parts, give error

Log entry

pub struct LogEntry {
    pub contextid: u32,
    pub job_id: u32,
    pub src: String,
    pub tags: Vec<String>,
    pub epoch: u32,
    pub loglevel: u8,
    pub content: String,
}

Line format

One log entry is one line.

Use tab-separated fields:

epoch<TAB>contextid<TAB>job_id<TAB>loglevel<TAB>tags<TAB>content\n

Example:

1714552210	0	0	5	service:router,mode:admin	router started on ui.sock

src is not written inside the line because it is represented by the file path.

Field rules

`epoch`

Unix timestamp in seconds.

u32 string

Example:

1714552210

`contextid`

Context identifier.

0 = default/admin/system

`job_id`

Job identifier.

0 = not job-related

`loglevel`

Single u8 value.

Suggested mapping:

0   debug
10  info
20  notice
30  warning
40  critical
254 error
255 reserved

`tags`

Comma-separated list.

key:value,key:value,key:value

Rules:

no spaces
no tabs
no newlines
comma separates tags
colon separates key/value

Example:

service:router,ctx:admin,kind:startup

Empty tags:

`content`

Human-readable message.

Rules:

no tabs
no raw newlines

Sanitizing:

\t  → space
\n  → [BR]
\r  → removed or [BR]

Writer architecture

Do not write directly from the caller thread.

Use:

caller thread
  → LogEntry
  → sanitize + format
  → send over channel

writer thread
  → receive lines
  → group by path
  → append with BufWriter<File>
  → periodic flush

Performance setting

Recommended defaults:

channel_size = 100_000
writer_buffer_per_file = 256 KB
flush_interval = 100 ms
max_open_files = 128
fsync = false

For higher safety:

flush_interval = 10 ms
fsync_on_shutdown = true

For maximum speed:

writer_buffer_per_file = 1 MB
flush_interval = 250 ms
fsync = false

File handling

Maintain an LRU cache of open writers:

HashMap<PathBuf, BufWriter<File>>

Open files with:

OpenOptions::new()
    .create(true)
    .append(true)
    .open(path)

Create parent directories automatically.

Rotation

Simple size-based rotation.

Default:

max_file_size = 256 MB
max_rotated_files = 8

Rotation format:

proxy.log
proxy.log.1
proxy.log.2
proxy.log.3

On rotation:

flush writer
close file
rename old files
open new proxy.log

Overflow behavior

Default: never block critical app logic forever.

Recommended modes:

DropNewest
Block
DropDebugFirst

Default:

DropDebugFirst

Meaning:

if channel is full:
  drop debug/info first
  keep warning/error if possible

At minimum, maintain counters:

dropped_debug: u64
dropped_info: u64
dropped_error: u64

Shutdown

On graceful shutdown:

stop accepting new logs
drain channel
flush all BufWriters
optionally fsync
close files

Minimal API

pub struct LoggerConfig {
    pub root: PathBuf,
    pub channel_size: usize,
    pub flush_interval_ms: u64,
    pub writer_buffer_size: usize,
    pub max_open_files: usize,
    pub max_file_size: u64,
    pub max_rotated_files: usize,
}

pub struct LogEntry {
    pub contextid: u32,
    pub job_id: u32,
    pub src: String,
    pub tags: Vec<String>,
    pub epoch: u32,
    pub loglevel: u8,
    pub content: String,
}

pub trait LogSink {
    fn log(&self, entry: LogEntry);
    fn flush(&self);
    fn shutdown(&self);
}

Final recommended setting

root = ~/hero/var/logs
format = TSV line
src = file path
buffer = 256 KB per file
flush = every 100 ms
rotation = 256 MB
fsync = only on shutdown
database = none
search = none

in /Volumes/T7/code0/hero_lib/crates/core/src/logger # Hero Logs — Fast Append-Only Logging Spec ## Goal Create a very fast, simple, human-readable logging system. No database. No Tantivy. No indexer for now. Just reliable local log capture. --- ## Log directory ```text ~/hero/var/logs/ ``` Each source has maximum 3 dotted parts: ```text src = "service.job.worker" ``` Mapped to: ```text ~/hero/var/logs/service/job/worker.log ``` Examples: ```text router → ~/hero/var/logs/router.log router.http → ~/hero/var/logs/router/http.log router.http.proxy → ~/hero/var/logs/router/http/proxy.log ``` If source has more than 3 parts, give error --- ## Log entry ```rust pub struct LogEntry { pub contextid: u32, pub job_id: u32, pub src: String, pub tags: Vec<String>, pub epoch: u32, pub loglevel: u8, pub content: String, } ``` --- ## Line format One log entry is one line. Use tab-separated fields: ```text epoch<TAB>contextid<TAB>job_id<TAB>loglevel<TAB>tags<TAB>content\n ``` Example: ```text 1714552210 0 0 5 service:router,mode:admin router started on ui.sock ``` `src` is not written inside the line because it is represented by the file path. --- ## Field rules ### `epoch` Unix timestamp in seconds. ```text u32 string ``` Example: ```text 1714552210 ``` --- ### `contextid` Context identifier. ```text 0 = default/admin/system ``` --- ### `job_id` Job identifier. ```text 0 = not job-related ``` --- ### `loglevel` Single `u8` value. Suggested mapping: ```text 0 debug 10 info 20 notice 30 warning 40 critical 254 error 255 reserved ``` --- ### `tags` Comma-separated list. ```text key:value,key:value,key:value ``` Rules: ```text no spaces no tabs no newlines comma separates tags colon separates key/value ``` Example: ```text service:router,ctx:admin,kind:startup ``` Empty tags: ```text - ``` --- ### `content` Human-readable message. Rules: ```text no tabs no raw newlines ``` Sanitizing: ```text \t → space \n → [BR] \r → removed or [BR] ``` --- ## Writer architecture Do not write directly from the caller thread. Use: ```text caller thread → LogEntry → sanitize + format → send over channel writer thread → receive lines → group by path → append with BufWriter<File> → periodic flush ``` --- ## Performance setting Recommended defaults: ```text channel_size = 100_000 writer_buffer_per_file = 256 KB flush_interval = 100 ms max_open_files = 128 fsync = false ``` For higher safety: ```text flush_interval = 10 ms fsync_on_shutdown = true ``` For maximum speed: ```text writer_buffer_per_file = 1 MB flush_interval = 250 ms fsync = false ``` --- ## File handling Maintain an LRU cache of open writers: ```rust HashMap<PathBuf, BufWriter<File>> ``` Open files with: ```rust OpenOptions::new() .create(true) .append(true) .open(path) ``` Create parent directories automatically. --- ## Rotation Simple size-based rotation. Default: ```text max_file_size = 256 MB max_rotated_files = 8 ``` Rotation format: ```text proxy.log proxy.log.1 proxy.log.2 proxy.log.3 ``` On rotation: ```text flush writer close file rename old files open new proxy.log ``` --- ## Overflow behavior Default: never block critical app logic forever. Recommended modes: ```text DropNewest Block DropDebugFirst ``` Default: ```text DropDebugFirst ``` Meaning: ```text if channel is full: drop debug/info first keep warning/error if possible ``` At minimum, maintain counters: ```rust dropped_debug: u64 dropped_info: u64 dropped_error: u64 ``` --- ## Shutdown On graceful shutdown: ```text stop accepting new logs drain channel flush all BufWriters optionally fsync close files ``` --- ## Minimal API ```rust pub struct LoggerConfig { pub root: PathBuf, pub channel_size: usize, pub flush_interval_ms: u64, pub writer_buffer_size: usize, pub max_open_files: usize, pub max_file_size: u64, pub max_rotated_files: usize, } pub struct LogEntry { pub contextid: u32, pub job_id: u32, pub src: String, pub tags: Vec<String>, pub epoch: u32, pub loglevel: u8, pub content: String, } pub trait LogSink { fn log(&self, entry: LogEntry); fn flush(&self); fn shutdown(&self); } ``` --- ## Final recommended setting ```text root = ~/hero/var/logs format = TSV line src = file path buffer = 256 KB per file flush = every 100 ms rotation = 256 MB fsync = only on shutdown database = none search = none ```

despiegk referenced this issue from lhumina_code/hero_proc

2026-05-01 03:49:36 +00:00

[arch] Log pipeline performance — measure producers, size SQLite, or move logs out (root cause behind #80) #85

despiegk commented

2026-05-01 03:54:12 +00:00

Author

Owner

Hero Logs — Implementation Spec for Issue #133

Objective

Implement an append-only file logger at crates/core/src/logger/ exposing a LogSink trait whose default implementation maps a dotted source path (e.g. router.http.proxy) to a path under ~/hero/var/logs/ and writes one TSV-formatted line per entry. The writer runs on a dedicated std::thread, owns a bounded MPSC channel, an LRU cache of BufWriter<File> handles, and performs size-based rotation, periodic flushing, and graceful shutdown. The module must build cleanly inside herolib_core with no new optional features and no tokio dependency.

Requirements

Logs go to ~/hero/var/logs/ with src dotted-path mapping to file path. Max 3 dotted parts:
- router -> ~/hero/var/logs/router.log
- router.http -> ~/hero/var/logs/router/http.log
- router.http.proxy -> ~/hero/var/logs/router/http/proxy.log
- More than 3 parts -> error.
LogEntry fields: contextid: u32, job_id: u32, src: String, tags: Vec<String>, epoch: u32, loglevel: u8, content: String.
One line per entry, TSV: epoch\tcontextid\tjob_id\tloglevel\ttags\tcontent\n. src is NOT in the line.
epoch is u32 unix seconds. contextid == 0 means default/admin/system. job_id == 0 means not job-related.
loglevel mapping: 0=debug, 10=info, 20=notice, 30=warning, 40=critical, 254=error, 255=reserved.
Tags: comma-separated key:value pairs, no spaces/tabs/newlines, empty Vec serializes as -.
Content sanitization: tabs -> space, \n -> [BR], \r removed or [BR].
Architecture: caller thread builds LogEntry, sanitizes + formats, sends pre-rendered record to a bounded channel; a single writer thread groups by path, writes through a BufWriter<File> (append + create), flushes periodically.
Defaults: channel_size=100_000, writer_buffer_per_file=256 KiB, flush_interval=100 ms, max_open_files=128, fsync=false.
LRU cache of open writers; OpenOptions { create: true, append: true }; auto-create parent directories.
Size-based rotation: max_file_size=256 MiB, max_rotated_files=8. On rotation: flush, close, rename chain, reopen.
Overflow modes: DropNewest, Block, DropDebugFirst (default DropDebugFirst). Counters: dropped_debug, dropped_info, dropped_error.
Shutdown: stop accepting, drain channel, flush all, optionally fsync, close.
Minimal API:
- LoggerConfig { root, channel_size, flush_interval_ms, writer_buffer_size, max_open_files, max_file_size, max_rotated_files }
- LogEntry { contextid, job_id, src, tags, epoch, loglevel, content }
- trait LogSink { fn log(&self, entry: LogEntry); fn flush(&self); fn shutdown(&self); }

Files to Modify / Create

File	Description
`crates/core/src/logger/mod.rs`	Module root; declares submodules and re-exports public API. Module-level rustdoc with example.
`crates/core/src/logger/errors.rs`	`LoggerError` enum (thiserror) and `Result<T>` alias.
`crates/core/src/logger/level.rs`	`LogLevel` enum + `as_u8` / `from_u8` with the 0/10/20/30/40/254/255 mapping.
`crates/core/src/logger/config.rs`	`LoggerConfig` + defaults; helper to resolve `~/hero/var/logs/`; `OverflowMode` enum.
`crates/core/src/logger/entry.rs`	`LogEntry`, `epoch_now()`, sanitization, `format_line`, `src_to_path`.
`crates/core/src/logger/writer.rs`	Internal writer-thread state: LRU cache, size counters, flushing, rotation.
`crates/core/src/logger/sink.rs`	`LogSink` trait + `Logger` (default file-backed implementation).
`crates/core/src/lib.rs`	Add `pub mod logger;` and a one-line rustdoc bullet.
`crates/core/tests/logger.rs`	Integration test: routing, sanitization, rotation, drop counters, shutdown.

No new mandatory Cargo deps required.

Implementation Plan

Step 1 - Module skeleton and public surface stubs

Files: errors.rs, level.rs, config.rs, mod.rs, lib.rs.
Adds LoggerError, Result, LogLevel, OverflowMode, LoggerConfig with documented defaults; declares submodules; pub mod logger; in lib.rs.

Dependencies: none.

Step 2 - `LogEntry`, sanitization, formatting, src->path mapping

Files: entry.rs, mod.rs.
Adds LogEntry, epoch_now(), sanitize_content, sanitize_tag, format_tags, format_line, src_to_path. Inline unit tests for all.

Dependencies: Step 1.

Step 3 - Writer thread (LRU cache, rotation, flush)

Files: writer.rs.
Adds WriterMsg enum and WriterState with the LRU cache, rotation chain, periodic flush via recv_timeout, optional fsync on shutdown, shutdown ack. Errors are logged via eprintln! but never panic the thread. Inline unit tests using tempfile::TempDir.

Dependencies: Steps 1-2.

Step 4 - `LogSink` trait + `Logger` implementation

Files: sink.rs, mod.rs.
Adds LogSink trait (Send + Sync), Logger struct holding bounded SyncSender, JoinHandle, atomic counters, shutdown flag. Implements three overflow modes; constructors Logger::new(cfg) and Logger::with_overflow(cfg, mode) returning Arc<Self>. Drop calls shutdown().

Dependencies: Steps 1-3.

Step 5 - Integration tests

Files: crates/core/tests/logger.rs.
Tests: 1-part / 2-part / 3-part path routing, >3 parts -> dropped_error counter, rotation chain limit, content sanitization, idempotent shutdown, tag formatting and - empty case.

Dependencies: Steps 1-4.

Step 6 - Module documentation

Files: mod.rs, lib.rs.
Module-level rustdoc with defaults table, mapping rules, and a no_run example. Bullet for the logger in the crate-level doc.

Dependencies: Steps 1-4.

Acceptance Criteria

pub mod logger; in crates/core/src/lib.rs.
Public surface: LogSink, Logger, LogEntry, LoggerConfig, LoggerError, Result, LogLevel, OverflowMode.
LogEntry has the seven specified fields with the exact types.
LoggerConfig defaults match the issue: channel_size=100_000, flush_interval_ms=100, writer_buffer_size=256*1024, max_open_files=128, max_file_size=256*1024*1024, max_rotated_files=8, fsync=false, default root=~/hero/var/logs.
src mapping: 1/2/3 parts produce the documented paths; >3 parts is rejected.
Output line is exactly epoch\tcontextid\tjob_id\tloglevel\ttags\tcontent\n; src does not appear in the line.
Empty tags renders as -; non-empty tags joined with ,.
Content sanitization: tabs -> space, \n and lone \r -> [BR].
Loglevel constants map 0/10/20/30/40/254/255 -> debug/info/notice/warning/critical/error/reserved.
Writer runs on a dedicated std::thread named hero-logger; no tokio dependency added.
LRU cache evicts beyond max_open_files; evicted writers are flushed before drop.
Files opened with create(true).append(true); parent dirs auto-created.
Size-based rotation produces name.log.1 ... name.log.{max_rotated_files}; oldest dropped.
Three overflow modes implemented; default DropDebugFirst.
dropped_debug, dropped_info, dropped_error counters readable from Logger.
shutdown() is idempotent, drains pending, flushes, optionally fsyncs, joins.
Integration tests cover routing, rotation, sanitization, drop counters, and shutdown semantics.
cargo build -p herolib_core and cargo test -p herolib_core succeed.

Notes

std::thread over tokio: the issue says "writer thread" and herolib_core does not depend on tokio in default features. Using std::thread plus std::sync::mpsc::sync_channel avoids forcing an async runtime on every caller.
Bounded channel: mpsc::sync_channel(N) provides the back-pressure semantics required by Block and DropNewest.
DropDebugFirst: with a single queue, we cannot reorder already-queued items. Chosen semantics: when the channel is full, drop incoming debug records (incrementing dropped_debug); block on incoming non-debug records. Biases the queue toward higher severity over time.
epoch auto-fill: if entry.epoch == 0, substitute epoch_now() at format time. 0 is unambiguous as a sentinel.
Tag handling: sanitize (strip whitespace and commas) but do not enforce structural key:value validation.
LRU policy: move-to-front on touch, evict from back on overflow. O(n) move is fine for max_open_files=128.
Rotation atomicity: std::fs::rename is best-effort. A crash mid-rotation can leave a gap. Acceptable for a non-critical log.
Default root: dirs::home_dir().join("hero").join("var").join("logs").
No new Cargo deps.
Stale code: the previously deleted README/log_instructions.md described an MQTT design unrelated to this spec. Starting fresh.

# Hero Logs — Implementation Spec for Issue #133 ## Objective Implement an append-only file logger at `crates/core/src/logger/` exposing a `LogSink` trait whose default implementation maps a dotted source path (e.g. `router.http.proxy`) to a path under `~/hero/var/logs/` and writes one TSV-formatted line per entry. The writer runs on a dedicated `std::thread`, owns a bounded MPSC channel, an LRU cache of `BufWriter<File>` handles, and performs size-based rotation, periodic flushing, and graceful shutdown. The module must build cleanly inside `herolib_core` with no new optional features and no tokio dependency. ## Requirements - Logs go to `~/hero/var/logs/` with `src` dotted-path mapping to file path. Max 3 dotted parts: - `router` -> `~/hero/var/logs/router.log` - `router.http` -> `~/hero/var/logs/router/http.log` - `router.http.proxy` -> `~/hero/var/logs/router/http/proxy.log` - More than 3 parts -> error. - `LogEntry` fields: `contextid: u32`, `job_id: u32`, `src: String`, `tags: Vec<String>`, `epoch: u32`, `loglevel: u8`, `content: String`. - One line per entry, TSV: `epoch\tcontextid\tjob_id\tloglevel\ttags\tcontent\n`. `src` is NOT in the line. - `epoch` is u32 unix seconds. `contextid == 0` means default/admin/system. `job_id == 0` means not job-related. - `loglevel` mapping: `0=debug, 10=info, 20=notice, 30=warning, 40=critical, 254=error, 255=reserved`. - Tags: comma-separated `key:value` pairs, no spaces/tabs/newlines, empty `Vec` serializes as `-`. - Content sanitization: tabs -> space, `\n` -> `[BR]`, `\r` removed or `[BR]`. - Architecture: caller thread builds `LogEntry`, sanitizes + formats, sends pre-rendered record to a bounded channel; a single writer thread groups by path, writes through a `BufWriter<File>` (append + create), flushes periodically. - Defaults: `channel_size=100_000`, `writer_buffer_per_file=256 KiB`, `flush_interval=100 ms`, `max_open_files=128`, `fsync=false`. - LRU cache of open writers; `OpenOptions { create: true, append: true }`; auto-create parent directories. - Size-based rotation: `max_file_size=256 MiB`, `max_rotated_files=8`. On rotation: flush, close, rename chain, reopen. - Overflow modes: `DropNewest`, `Block`, `DropDebugFirst` (default `DropDebugFirst`). Counters: `dropped_debug`, `dropped_info`, `dropped_error`. - Shutdown: stop accepting, drain channel, flush all, optionally fsync, close. - Minimal API: - `LoggerConfig { root, channel_size, flush_interval_ms, writer_buffer_size, max_open_files, max_file_size, max_rotated_files }` - `LogEntry { contextid, job_id, src, tags, epoch, loglevel, content }` - `trait LogSink { fn log(&self, entry: LogEntry); fn flush(&self); fn shutdown(&self); }` ## Files to Modify / Create | File | Description | |------|-------------| | `crates/core/src/logger/mod.rs` | Module root; declares submodules and re-exports public API. Module-level rustdoc with example. | | `crates/core/src/logger/errors.rs` | `LoggerError` enum (thiserror) and `Result<T>` alias. | | `crates/core/src/logger/level.rs` | `LogLevel` enum + `as_u8` / `from_u8` with the 0/10/20/30/40/254/255 mapping. | | `crates/core/src/logger/config.rs` | `LoggerConfig` + defaults; helper to resolve `~/hero/var/logs/`; `OverflowMode` enum. | | `crates/core/src/logger/entry.rs` | `LogEntry`, `epoch_now()`, sanitization, `format_line`, `src_to_path`. | | `crates/core/src/logger/writer.rs` | Internal writer-thread state: LRU cache, size counters, flushing, rotation. | | `crates/core/src/logger/sink.rs` | `LogSink` trait + `Logger` (default file-backed implementation). | | `crates/core/src/lib.rs` | Add `pub mod logger;` and a one-line rustdoc bullet. | | `crates/core/tests/logger.rs` | Integration test: routing, sanitization, rotation, drop counters, shutdown. | No new mandatory Cargo deps required. ## Implementation Plan ### Step 1 - Module skeleton and public surface stubs Files: `errors.rs`, `level.rs`, `config.rs`, `mod.rs`, `lib.rs`. Adds `LoggerError`, `Result`, `LogLevel`, `OverflowMode`, `LoggerConfig` with documented defaults; declares submodules; `pub mod logger;` in `lib.rs`. Dependencies: none. ### Step 2 - `LogEntry`, sanitization, formatting, src->path mapping Files: `entry.rs`, `mod.rs`. Adds `LogEntry`, `epoch_now()`, `sanitize_content`, `sanitize_tag`, `format_tags`, `format_line`, `src_to_path`. Inline unit tests for all. Dependencies: Step 1. ### Step 3 - Writer thread (LRU cache, rotation, flush) Files: `writer.rs`. Adds `WriterMsg` enum and `WriterState` with the LRU cache, rotation chain, periodic flush via `recv_timeout`, optional fsync on shutdown, shutdown ack. Errors are logged via `eprintln!` but never panic the thread. Inline unit tests using `tempfile::TempDir`. Dependencies: Steps 1-2. ### Step 4 - `LogSink` trait + `Logger` implementation Files: `sink.rs`, `mod.rs`. Adds `LogSink` trait (`Send + Sync`), `Logger` struct holding bounded `SyncSender`, `JoinHandle`, atomic counters, shutdown flag. Implements three overflow modes; constructors `Logger::new(cfg)` and `Logger::with_overflow(cfg, mode)` returning `Arc<Self>`. `Drop` calls `shutdown()`. Dependencies: Steps 1-3. ### Step 5 - Integration tests Files: `crates/core/tests/logger.rs`. Tests: 1-part / 2-part / 3-part path routing, >3 parts -> dropped_error counter, rotation chain limit, content sanitization, idempotent shutdown, tag formatting and `-` empty case. Dependencies: Steps 1-4. ### Step 6 - Module documentation Files: `mod.rs`, `lib.rs`. Module-level rustdoc with defaults table, mapping rules, and a `no_run` example. Bullet for the logger in the crate-level doc. Dependencies: Steps 1-4. ## Acceptance Criteria - [ ] `pub mod logger;` in `crates/core/src/lib.rs`. - [ ] Public surface: `LogSink`, `Logger`, `LogEntry`, `LoggerConfig`, `LoggerError`, `Result`, `LogLevel`, `OverflowMode`. - [ ] `LogEntry` has the seven specified fields with the exact types. - [ ] `LoggerConfig` defaults match the issue: `channel_size=100_000`, `flush_interval_ms=100`, `writer_buffer_size=256*1024`, `max_open_files=128`, `max_file_size=256*1024*1024`, `max_rotated_files=8`, `fsync=false`, default `root=~/hero/var/logs`. - [ ] `src` mapping: 1/2/3 parts produce the documented paths; >3 parts is rejected. - [ ] Output line is exactly `epoch\tcontextid\tjob_id\tloglevel\ttags\tcontent\n`; `src` does not appear in the line. - [ ] Empty `tags` renders as `-`; non-empty tags joined with `,`. - [ ] Content sanitization: tabs -> space, `\n` and lone `\r` -> `[BR]`. - [ ] Loglevel constants map 0/10/20/30/40/254/255 -> debug/info/notice/warning/critical/error/reserved. - [ ] Writer runs on a dedicated `std::thread` named `hero-logger`; no tokio dependency added. - [ ] LRU cache evicts beyond `max_open_files`; evicted writers are flushed before drop. - [ ] Files opened with `create(true).append(true)`; parent dirs auto-created. - [ ] Size-based rotation produces `name.log.1` ... `name.log.{max_rotated_files}`; oldest dropped. - [ ] Three overflow modes implemented; default `DropDebugFirst`. - [ ] `dropped_debug`, `dropped_info`, `dropped_error` counters readable from `Logger`. - [ ] `shutdown()` is idempotent, drains pending, flushes, optionally fsyncs, joins. - [ ] Integration tests cover routing, rotation, sanitization, drop counters, and shutdown semantics. - [ ] `cargo build -p herolib_core` and `cargo test -p herolib_core` succeed. ## Notes - `std::thread` over tokio: the issue says "writer thread" and `herolib_core` does not depend on tokio in default features. Using `std::thread` plus `std::sync::mpsc::sync_channel` avoids forcing an async runtime on every caller. - Bounded channel: `mpsc::sync_channel(N)` provides the back-pressure semantics required by `Block` and `DropNewest`. - `DropDebugFirst`: with a single queue, we cannot reorder already-queued items. Chosen semantics: when the channel is full, drop incoming debug records (incrementing `dropped_debug`); block on incoming non-debug records. Biases the queue toward higher severity over time. - `epoch` auto-fill: if `entry.epoch == 0`, substitute `epoch_now()` at format time. `0` is unambiguous as a sentinel. - Tag handling: sanitize (strip whitespace and commas) but do not enforce structural `key:value` validation. - LRU policy: move-to-front on touch, evict from back on overflow. O(n) move is fine for `max_open_files=128`. - Rotation atomicity: `std::fs::rename` is best-effort. A crash mid-rotation can leave a gap. Acceptable for a non-critical log. - Default root: `dirs::home_dir().join("hero").join("var").join("logs")`. - No new Cargo deps. - Stale code: the previously deleted README/log_instructions.md described an MQTT design unrelated to this spec. Starting fresh.

despiegk commented

2026-05-01 04:07:32 +00:00

Author

Owner

Hero Logs - Implementation Spec for Issue #133 (Revised)

Supersedes the earlier spec comment. The change: job_id is removed from LogEntry and from the TSV line. Callers encode any job identifier they want inside src. The line becomes 5 columns instead of 6.

Objective

Requirements

Logs go to ~/hero/var/logs/ with src dotted-path mapping to file path. Max 3 dotted parts:
- router -> ~/hero/var/logs/router.log
- router.http -> ~/hero/var/logs/router/http.log
- router.http.proxy -> ~/hero/var/logs/router/http/proxy.log
- More than 3 parts -> error.
LogEntry fields: contextid: u32, src: String, tags: Vec<String>, epoch: u32, loglevel: u8, content: String. (No job_id - callers put it in src if they want it.)
One line per entry, TSV: epoch\tcontextid\tloglevel\ttags\tcontent\n. src is NOT in the line; job_id no longer exists.
epoch is u32 unix seconds. contextid == 0 means default/admin/system.
loglevel mapping: 0=debug, 10=info, 20=notice, 30=warning, 40=critical, 254=error, 255=reserved.
Tags: comma-separated key:value pairs, no spaces/tabs/newlines, empty Vec serializes as -.
Content sanitization: tabs -> space, \n -> [BR], \r removed or [BR].
Architecture: caller thread builds LogEntry, sanitizes + formats, sends pre-rendered record to a bounded channel; a single writer thread groups by path, writes through a BufWriter<File> (append + create), flushes periodically.
Defaults: channel_size=100_000, writer_buffer_per_file=256 KiB, flush_interval=100 ms, max_open_files=128, fsync=false.
LRU cache of open writers; OpenOptions { create: true, append: true }; auto-create parent directories.
Size-based rotation: max_file_size=256 MiB, max_rotated_files=8. On rotation: flush, close, rename chain, reopen.
Overflow modes: DropNewest, Block, DropDebugFirst (default DropDebugFirst). Counters: dropped_debug, dropped_info, dropped_error.
Shutdown: stop accepting, drain channel, flush all, optionally fsync, close.
Minimal API:
- LoggerConfig { root, channel_size, flush_interval_ms, writer_buffer_size, max_open_files, max_file_size, max_rotated_files } (plus fsync: bool)
- LogEntry { contextid, src, tags, epoch, loglevel, content }
- trait LogSink { fn log(&self, entry: LogEntry); fn flush(&self); fn shutdown(&self); }

Files to Modify / Create

File	Description
`crates/core/src/logger/mod.rs`	Module root; submodule decls and re-exports. Module-level rustdoc with example.
`crates/core/src/logger/errors.rs`	`LoggerError` enum (thiserror) and `Result<T>` alias.
`crates/core/src/logger/level.rs`	`LogLevel` enum + `as_u8` / `from_u8` for the 0/10/20/30/40/254/255 mapping.
`crates/core/src/logger/config.rs`	`LoggerConfig` + defaults; default-root resolver; `OverflowMode` enum.
`crates/core/src/logger/entry.rs`	`LogEntry`, `epoch_now()`, sanitization, `format_line` (5 columns), `src_to_path`.
`crates/core/src/logger/writer.rs`	Internal writer-thread state: LRU cache, size counters, flushing, rotation.
`crates/core/src/logger/sink.rs`	`LogSink` trait + `Logger` (default file-backed implementation).
`crates/core/src/lib.rs`	Add `pub mod logger;` and a one-line rustdoc bullet.
`crates/core/tests/logger.rs`	Integration tests: routing, sanitization, rotation, drop counters, shutdown.

No new mandatory Cargo deps.

Implementation Plan

Step 1 - Module skeleton and public surface stubs

Dependencies: none.

Step 2 - `LogEntry`, sanitization, formatting, src->path mapping

Files: entry.rs, mod.rs.
Adds LogEntry (six fields, no job_id), epoch_now(), sanitize_content, sanitize_tag, format_tags, format_line producing exactly five tab-separated columns + newline, src_to_path. Inline unit tests for all.

Dependencies: Step 1.

Step 3 - Writer thread (LRU cache, rotation, flush)

Files: writer.rs.
Adds WriterMsg and WriterState with the LRU cache, rotation chain, periodic flush via recv_timeout, optional fsync on shutdown, shutdown ack. Errors logged via eprintln!, never panic. Inline unit tests using tempfile::TempDir.

Dependencies: Steps 1-2.

Step 4 - `LogSink` trait + `Logger` implementation

Files: sink.rs, mod.rs.
Adds LogSink: Send + Sync trait, Logger struct with bounded SyncSender, JoinHandle, atomic counters, shutdown flag. Three overflow modes; Logger::new(cfg)/Logger::with_overflow(cfg, mode) returning Arc<Self>. Drop calls shutdown().

Dependencies: Steps 1-3.

Step 5 - Integration tests

Files: crates/core/tests/logger.rs.
Tests: 1/2/3-part path routing, >3 parts -> dropped_error, rotation chain limit, content sanitization, idempotent shutdown, tag formatting and - empty case, exact 5-column TSV line.

Dependencies: Steps 1-4.

Step 6 - Module documentation

Files: mod.rs, lib.rs.
Module-level rustdoc with defaults table, mapping rules, and a no_run example. Bullet for the logger in the crate-level doc.

Dependencies: Steps 1-4.

Acceptance Criteria

pub mod logger; in crates/core/src/lib.rs.
Public surface: LogSink, Logger, LogEntry, LoggerConfig, LoggerError, Result, LogLevel, OverflowMode.
LogEntry has exactly: contextid: u32, src: String, tags: Vec<String>, epoch: u32, loglevel: u8, content: String. No job_id.
LoggerConfig defaults match the issue: channel_size=100_000, flush_interval_ms=100, writer_buffer_size=256*1024, max_open_files=128, max_file_size=256*1024*1024, max_rotated_files=8, fsync=false, default root=~/hero/var/logs.
src mapping: 1/2/3 parts produce the documented paths; >3 parts is rejected.
Output line is exactly epoch\tcontextid\tloglevel\ttags\tcontent\n (five columns); src does not appear in the line.
Empty tags renders as -; non-empty tags joined with ,.
Content sanitization: tabs -> space, \n and lone \r -> [BR].
Loglevel constants map 0/10/20/30/40/254/255 -> debug/info/notice/warning/critical/error/reserved.
Writer runs on a dedicated std::thread named hero-logger; no tokio dependency added.
LRU cache evicts beyond max_open_files; evicted writers are flushed before drop.
Files opened with create(true).append(true); parent dirs auto-created.
Size-based rotation produces name.log.1 ... name.log.{max_rotated_files}; oldest dropped.
Three overflow modes implemented; default DropDebugFirst.
dropped_debug, dropped_info, dropped_error counters readable from Logger.
shutdown() is idempotent, drains pending, flushes, optionally fsyncs, joins.
Integration tests cover routing, rotation, sanitization, drop counters, and shutdown semantics.
cargo build -p herolib_core and cargo test -p herolib_core succeed.

Notes

job_id removed: src is freeform (still 1-3 dotted parts). Callers may encode service.job.<id> themselves; the logger does not interpret it.
std::thread over tokio: the issue says "writer thread" and herolib_core does not depend on tokio in default features.
Bounded channel: mpsc::sync_channel(N) provides back-pressure for Block and DropNewest.
DropDebugFirst: when the channel is full, drop incoming debug records (incrementing dropped_debug); block on incoming non-debug records. Biases the queue toward higher severity over time.
epoch auto-fill: if entry.epoch == 0, substitute epoch_now() at format time. 0 is unambiguous as a sentinel.
Tag handling: sanitize (strip whitespace and commas); no key:value structural validation.
LRU policy: move-to-front on touch, evict from back on overflow. O(n) is fine for default 128 entries.
Rotation atomicity: std::fs::rename is best-effort; a crash mid-rotation can leave a gap. Acceptable.
Default root: dirs::home_dir().join("hero").join("var").join("logs").
No new Cargo deps.

# Hero Logs - Implementation Spec for Issue #133 (Revised) > Supersedes the earlier spec comment. The change: `job_id` is removed from `LogEntry` and from the TSV line. Callers encode any job identifier they want inside `src`. The line becomes 5 columns instead of 6. ## Objective Implement an append-only file logger at `crates/core/src/logger/` exposing a `LogSink` trait whose default implementation maps a dotted source path (e.g. `router.http.proxy`) to a path under `~/hero/var/logs/` and writes one TSV-formatted line per entry. The writer runs on a dedicated `std::thread`, owns a bounded MPSC channel, an LRU cache of `BufWriter<File>` handles, and performs size-based rotation, periodic flushing, and graceful shutdown. The module must build cleanly inside `herolib_core` with no new optional features and no tokio dependency. ## Requirements - Logs go to `~/hero/var/logs/` with `src` dotted-path mapping to file path. Max 3 dotted parts: - `router` -> `~/hero/var/logs/router.log` - `router.http` -> `~/hero/var/logs/router/http.log` - `router.http.proxy` -> `~/hero/var/logs/router/http/proxy.log` - More than 3 parts -> error. - `LogEntry` fields: `contextid: u32`, `src: String`, `tags: Vec<String>`, `epoch: u32`, `loglevel: u8`, `content: String`. (No `job_id` - callers put it in `src` if they want it.) - One line per entry, TSV: `epoch\tcontextid\tloglevel\ttags\tcontent\n`. `src` is NOT in the line; `job_id` no longer exists. - `epoch` is u32 unix seconds. `contextid == 0` means default/admin/system. - `loglevel` mapping: `0=debug, 10=info, 20=notice, 30=warning, 40=critical, 254=error, 255=reserved`. - Tags: comma-separated `key:value` pairs, no spaces/tabs/newlines, empty `Vec` serializes as `-`. - Content sanitization: tabs -> space, `\n` -> `[BR]`, `\r` removed or `[BR]`. - Architecture: caller thread builds `LogEntry`, sanitizes + formats, sends pre-rendered record to a bounded channel; a single writer thread groups by path, writes through a `BufWriter<File>` (append + create), flushes periodically. - Defaults: `channel_size=100_000`, `writer_buffer_per_file=256 KiB`, `flush_interval=100 ms`, `max_open_files=128`, `fsync=false`. - LRU cache of open writers; `OpenOptions { create: true, append: true }`; auto-create parent directories. - Size-based rotation: `max_file_size=256 MiB`, `max_rotated_files=8`. On rotation: flush, close, rename chain, reopen. - Overflow modes: `DropNewest`, `Block`, `DropDebugFirst` (default `DropDebugFirst`). Counters: `dropped_debug`, `dropped_info`, `dropped_error`. - Shutdown: stop accepting, drain channel, flush all, optionally fsync, close. - Minimal API: - `LoggerConfig { root, channel_size, flush_interval_ms, writer_buffer_size, max_open_files, max_file_size, max_rotated_files }` (plus `fsync: bool`) - `LogEntry { contextid, src, tags, epoch, loglevel, content }` - `trait LogSink { fn log(&self, entry: LogEntry); fn flush(&self); fn shutdown(&self); }` ## Files to Modify / Create | File | Description | |------|-------------| | `crates/core/src/logger/mod.rs` | Module root; submodule decls and re-exports. Module-level rustdoc with example. | | `crates/core/src/logger/errors.rs` | `LoggerError` enum (thiserror) and `Result<T>` alias. | | `crates/core/src/logger/level.rs` | `LogLevel` enum + `as_u8` / `from_u8` for the 0/10/20/30/40/254/255 mapping. | | `crates/core/src/logger/config.rs` | `LoggerConfig` + defaults; default-root resolver; `OverflowMode` enum. | | `crates/core/src/logger/entry.rs` | `LogEntry`, `epoch_now()`, sanitization, `format_line` (5 columns), `src_to_path`. | | `crates/core/src/logger/writer.rs` | Internal writer-thread state: LRU cache, size counters, flushing, rotation. | | `crates/core/src/logger/sink.rs` | `LogSink` trait + `Logger` (default file-backed implementation). | | `crates/core/src/lib.rs` | Add `pub mod logger;` and a one-line rustdoc bullet. | | `crates/core/tests/logger.rs` | Integration tests: routing, sanitization, rotation, drop counters, shutdown. | No new mandatory Cargo deps. ## Implementation Plan ### Step 1 - Module skeleton and public surface stubs Files: `errors.rs`, `level.rs`, `config.rs`, `mod.rs`, `lib.rs`. Adds `LoggerError`, `Result`, `LogLevel`, `OverflowMode`, `LoggerConfig` with documented defaults; declares submodules; `pub mod logger;` in `lib.rs`. Dependencies: none. ### Step 2 - `LogEntry`, sanitization, formatting, src->path mapping Files: `entry.rs`, `mod.rs`. Adds `LogEntry` (six fields, no job_id), `epoch_now()`, `sanitize_content`, `sanitize_tag`, `format_tags`, `format_line` producing exactly five tab-separated columns + newline, `src_to_path`. Inline unit tests for all. Dependencies: Step 1. ### Step 3 - Writer thread (LRU cache, rotation, flush) Files: `writer.rs`. Adds `WriterMsg` and `WriterState` with the LRU cache, rotation chain, periodic flush via `recv_timeout`, optional fsync on shutdown, shutdown ack. Errors logged via `eprintln!`, never panic. Inline unit tests using `tempfile::TempDir`. Dependencies: Steps 1-2. ### Step 4 - `LogSink` trait + `Logger` implementation Files: `sink.rs`, `mod.rs`. Adds `LogSink: Send + Sync` trait, `Logger` struct with bounded `SyncSender`, `JoinHandle`, atomic counters, shutdown flag. Three overflow modes; `Logger::new(cfg)`/`Logger::with_overflow(cfg, mode)` returning `Arc<Self>`. `Drop` calls `shutdown()`. Dependencies: Steps 1-3. ### Step 5 - Integration tests Files: `crates/core/tests/logger.rs`. Tests: 1/2/3-part path routing, >3 parts -> dropped_error, rotation chain limit, content sanitization, idempotent shutdown, tag formatting and `-` empty case, exact 5-column TSV line. Dependencies: Steps 1-4. ### Step 6 - Module documentation Files: `mod.rs`, `lib.rs`. Module-level rustdoc with defaults table, mapping rules, and a `no_run` example. Bullet for the logger in the crate-level doc. Dependencies: Steps 1-4. ## Acceptance Criteria - [ ] `pub mod logger;` in `crates/core/src/lib.rs`. - [ ] Public surface: `LogSink`, `Logger`, `LogEntry`, `LoggerConfig`, `LoggerError`, `Result`, `LogLevel`, `OverflowMode`. - [ ] `LogEntry` has exactly: `contextid: u32`, `src: String`, `tags: Vec<String>`, `epoch: u32`, `loglevel: u8`, `content: String`. No `job_id`. - [ ] `LoggerConfig` defaults match the issue: `channel_size=100_000`, `flush_interval_ms=100`, `writer_buffer_size=256*1024`, `max_open_files=128`, `max_file_size=256*1024*1024`, `max_rotated_files=8`, `fsync=false`, default `root=~/hero/var/logs`. - [ ] `src` mapping: 1/2/3 parts produce the documented paths; >3 parts is rejected. - [ ] Output line is exactly `epoch\tcontextid\tloglevel\ttags\tcontent\n` (five columns); `src` does not appear in the line. - [ ] Empty `tags` renders as `-`; non-empty tags joined with `,`. - [ ] Content sanitization: tabs -> space, `\n` and lone `\r` -> `[BR]`. - [ ] Loglevel constants map 0/10/20/30/40/254/255 -> debug/info/notice/warning/critical/error/reserved. - [ ] Writer runs on a dedicated `std::thread` named `hero-logger`; no tokio dependency added. - [ ] LRU cache evicts beyond `max_open_files`; evicted writers are flushed before drop. - [ ] Files opened with `create(true).append(true)`; parent dirs auto-created. - [ ] Size-based rotation produces `name.log.1` ... `name.log.{max_rotated_files}`; oldest dropped. - [ ] Three overflow modes implemented; default `DropDebugFirst`. - [ ] `dropped_debug`, `dropped_info`, `dropped_error` counters readable from `Logger`. - [ ] `shutdown()` is idempotent, drains pending, flushes, optionally fsyncs, joins. - [ ] Integration tests cover routing, rotation, sanitization, drop counters, and shutdown semantics. - [ ] `cargo build -p herolib_core` and `cargo test -p herolib_core` succeed. ## Notes - `job_id` removed: `src` is freeform (still 1-3 dotted parts). Callers may encode `service.job.<id>` themselves; the logger does not interpret it. - `std::thread` over tokio: the issue says "writer thread" and `herolib_core` does not depend on tokio in default features. - Bounded channel: `mpsc::sync_channel(N)` provides back-pressure for `Block` and `DropNewest`. - `DropDebugFirst`: when the channel is full, drop incoming debug records (incrementing `dropped_debug`); block on incoming non-debug records. Biases the queue toward higher severity over time. - `epoch` auto-fill: if `entry.epoch == 0`, substitute `epoch_now()` at format time. `0` is unambiguous as a sentinel. - Tag handling: sanitize (strip whitespace and commas); no `key:value` structural validation. - LRU policy: move-to-front on touch, evict from back on overflow. O(n) is fine for default 128 entries. - Rotation atomicity: `std::fs::rename` is best-effort; a crash mid-rotation can leave a gap. Acceptable. - Default root: `dirs::home_dir().join("hero").join("var").join("logs")`. - No new Cargo deps.

despiegk commented

2026-05-01 04:17:13 +00:00

Author

Owner

Test Results

cargo test -p herolib_core

Suite	Result
Library unit tests	196 passed, 0 failed
`tests/logger.rs` integration	8 passed, 0 failed
Doc tests	28 passed, 0 failed
Total	232 passed, 0 failed, 0 regressions

New logger unit tests (15 in `crates/core/src/logger/`)

level::round_trip_known_values
level::unknown_maps_to_reserved
entry::src_one_part
entry::src_two_parts
entry::src_three_parts
entry::src_too_deep
entry::src_empty_or_dotted
entry::src_rejects_path_separators
entry::sanitize_content_replaces_tabs_and_newlines
entry::empty_tags_render_dash
entry::multi_tags_join_with_comma
entry::tags_are_sanitized
entry::format_line_shape_five_columns
entry::format_line_autofills_epoch
writer::opens_and_appends
writer::lru_eviction
writer::rotation_chain
writer::shutdown_flushes_and_acks

Integration tests (`crates/core/tests/logger.rs`)

one_part_src_writes_root_log - exact TSV output verified byte-for-byte
three_part_src_creates_nested_dirs - parent directory auto-creation
too_deep_src_increments_dropped_error - >3 dotted parts rejected, counter incremented
rotation_chain_caps_at_max_rotated_files - .log.1/.log.2/.log.3 exist, .log.4 does not
content_sanitization_produces_single_line - tabs/CRLF/LF/CR all collapsed correctly
shutdown_is_idempotent_and_blocks_new_logs - second shutdown is a no-op; post-shutdown log bumps dropped_error
tag_formatting_and_empty_dash - empty tags render -, internal whitespace and commas sanitized
drop_newest_drops_under_pressure - bounded channel under burst correctly increments dropped_debug

## Test Results `cargo test -p herolib_core` | Suite | Result | |---|---| | Library unit tests | 196 passed, 0 failed | | `tests/logger.rs` integration | 8 passed, 0 failed | | Doc tests | 28 passed, 0 failed | | **Total** | **232 passed, 0 failed, 0 regressions** | ### New logger unit tests (15 in `crates/core/src/logger/`) - `level::round_trip_known_values` - `level::unknown_maps_to_reserved` - `entry::src_one_part` - `entry::src_two_parts` - `entry::src_three_parts` - `entry::src_too_deep` - `entry::src_empty_or_dotted` - `entry::src_rejects_path_separators` - `entry::sanitize_content_replaces_tabs_and_newlines` - `entry::empty_tags_render_dash` - `entry::multi_tags_join_with_comma` - `entry::tags_are_sanitized` - `entry::format_line_shape_five_columns` - `entry::format_line_autofills_epoch` - `writer::opens_and_appends` - `writer::lru_eviction` - `writer::rotation_chain` - `writer::shutdown_flushes_and_acks` ### Integration tests (`crates/core/tests/logger.rs`) - `one_part_src_writes_root_log` - exact TSV output verified byte-for-byte - `three_part_src_creates_nested_dirs` - parent directory auto-creation - `too_deep_src_increments_dropped_error` - >3 dotted parts rejected, counter incremented - `rotation_chain_caps_at_max_rotated_files` - `.log.1`/`.log.2`/`.log.3` exist, `.log.4` does not - `content_sanitization_produces_single_line` - tabs/CRLF/LF/CR all collapsed correctly - `shutdown_is_idempotent_and_blocks_new_logs` - second shutdown is a no-op; post-shutdown log bumps `dropped_error` - `tag_formatting_and_empty_dash` - empty tags render `-`, internal whitespace and commas sanitized - `drop_newest_drops_under_pressure` - bounded channel under burst correctly increments `dropped_debug`

despiegk commented

2026-05-01 04:18:53 +00:00

Author

Owner

Implementation Summary

Append-only file logger landed at crates/core/src/logger/. Public surface re-exported from herolib_core::logger.

Files created

crates/core/src/logger/mod.rs - module root, public re-exports, module-level docs with mapping table and TSV format spec.
crates/core/src/logger/errors.rs - LoggerError (thiserror) + Result<T> alias. Variants: InvalidSrc, SrcTooDeep { src, parts }, Io, ChannelClosed, AlreadyShutdown.
crates/core/src/logger/level.rs - LogLevel enum and LOG_LEVEL_* constants for 0/10/20/30/40/254/255.
crates/core/src/logger/config.rs - LoggerConfig with documented defaults (channel_size=100_000, flush_interval_ms=100, writer_buffer_size=256 KiB, max_open_files=128, max_file_size=256 MiB, max_rotated_files=8, fsync=false), default_root() resolving to ~/hero/var/logs, and OverflowMode (DropNewest, Block, DropDebugFirst; default DropDebugFirst).
crates/core/src/logger/entry.rs - LogEntry (six fields: contextid, src, tags, epoch, loglevel, content), epoch_now(), sanitize_content, sanitize_tag, format_tags, format_line (5-column TSV), src_to_path enforcing the 1/2/3-part rule.
crates/core/src/logger/writer.rs - dedicated std::thread writer. LRU cache (HashMap<PathBuf, BufWriter<File>> + VecDeque<PathBuf>), per-file size tracking, periodic flush via recv_timeout, size-based rotation chain, drain-on-shutdown, optional fsync.
crates/core/src/logger/sink.rs - LogSink: Send + Sync trait and the Logger struct. Bounded mpsc::sync_channel for back-pressure, atomic counters (dropped_debug, dropped_info, dropped_error), CAS-guarded idempotent shutdown(), Drop calls shutdown().

File modified

crates/core/src/lib.rs - pub mod logger; + a one-line entry in the crate-level doc.

Public API

pub use logger::{
    LogEntry, LogLevel, LogSink, Logger, LoggerConfig, LoggerError,
    OverflowMode, Result,
    LOG_LEVEL_DEBUG, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_WARNING,
    LOG_LEVEL_CRITICAL, LOG_LEVEL_ERROR, LOG_LEVEL_RESERVED,
    epoch_now, format_line, src_to_path, default_root,
};

Wire format

Five tab-separated columns, ending with \n:

epoch<TAB>contextid<TAB>loglevel<TAB>tags<TAB>content\n

src is encoded by the file path and is never inside the line. job_id is not part of the model - callers encode it inside src if they want it.

Acceptance criteria

pub mod logger; in crates/core/src/lib.rs.
Public surface: LogSink, Logger, LogEntry, LoggerConfig, LoggerError, Result, LogLevel, OverflowMode.
LogEntry has exactly the six specified fields with the exact types.
LoggerConfig defaults match the issue.
src mapping: 1/2/3 parts produce the documented paths; >3 parts is rejected.
Output line is exactly epoch\tcontextid\tloglevel\ttags\tcontent\n; src does not appear in the line.
Empty tags renders as -; non-empty tags joined with ,.
Content sanitization: tabs -> space; \n, \r, and \r\n -> [BR].
Loglevel constants map 0/10/20/30/40/254/255 -> debug/info/notice/warning/critical/error/reserved.
Writer runs on a dedicated std::thread named hero-logger; no tokio dependency added.
LRU cache evicts beyond max_open_files; evicted writers are flushed.
Files opened with create(true).append(true); parent dirs auto-created.
Size-based rotation chain name.log.1...name.log.{max_rotated_files}; oldest dropped.
Three overflow modes implemented; default DropDebugFirst.
dropped_debug, dropped_info, dropped_error counters readable from Logger.
shutdown() is idempotent, drains pending records, flushes, optionally fsyncs, joins.
Integration tests cover routing, rotation, sanitization, drop counters, and shutdown semantics.
cargo build -p herolib_core and cargo test -p herolib_core succeed.

Notes / decisions

job_id removed per follow-up clarification on this issue. src remains 1-3 dotted parts; callers encode any job identifier they want there.
No new Cargo dependencies. thiserror, serde, dirs (target-gated), and std cover everything; tempfile (already a dev-dep) is used in the writer's inline tests and the integration test.
DropDebugFirst semantics with a single bounded queue: when the channel is full, drop incoming loglevel == 0 records and bump dropped_debug; non-debug records fall back to Block so they are never silently lost.
epoch == 0 is treated as a sentinel and replaced with epoch_now() at format time. Real Unix timestamps in the system are always far above zero.
Rotation uses std::fs::rename per slot - a crash mid-rotation can leave a numbering gap, which is acceptable for non-critical logs.
Default root is dirs::home_dir().join("hero/var/logs") on native targets and /tmp/hero/var/logs on wasm32.

## Implementation Summary Append-only file logger landed at `crates/core/src/logger/`. Public surface re-exported from `herolib_core::logger`. ### Files created - `crates/core/src/logger/mod.rs` - module root, public re-exports, module-level docs with mapping table and TSV format spec. - `crates/core/src/logger/errors.rs` - `LoggerError` (`thiserror`) + `Result<T>` alias. Variants: `InvalidSrc`, `SrcTooDeep { src, parts }`, `Io`, `ChannelClosed`, `AlreadyShutdown`. - `crates/core/src/logger/level.rs` - `LogLevel` enum and `LOG_LEVEL_*` constants for 0/10/20/30/40/254/255. - `crates/core/src/logger/config.rs` - `LoggerConfig` with documented defaults (channel_size=100_000, flush_interval_ms=100, writer_buffer_size=256 KiB, max_open_files=128, max_file_size=256 MiB, max_rotated_files=8, fsync=false), `default_root()` resolving to `~/hero/var/logs`, and `OverflowMode` (`DropNewest`, `Block`, `DropDebugFirst`; default `DropDebugFirst`). - `crates/core/src/logger/entry.rs` - `LogEntry` (six fields: `contextid`, `src`, `tags`, `epoch`, `loglevel`, `content`), `epoch_now()`, `sanitize_content`, `sanitize_tag`, `format_tags`, `format_line` (5-column TSV), `src_to_path` enforcing the 1/2/3-part rule. - `crates/core/src/logger/writer.rs` - dedicated `std::thread` writer. LRU cache (`HashMap<PathBuf, BufWriter<File>>` + `VecDeque<PathBuf>`), per-file size tracking, periodic flush via `recv_timeout`, size-based rotation chain, drain-on-shutdown, optional fsync. - `crates/core/src/logger/sink.rs` - `LogSink: Send + Sync` trait and the `Logger` struct. Bounded `mpsc::sync_channel` for back-pressure, atomic counters (`dropped_debug`, `dropped_info`, `dropped_error`), CAS-guarded idempotent `shutdown()`, `Drop` calls `shutdown()`. ### File modified - `crates/core/src/lib.rs` - `pub mod logger;` + a one-line entry in the crate-level doc. ### Public API ```rust pub use logger::{ LogEntry, LogLevel, LogSink, Logger, LoggerConfig, LoggerError, OverflowMode, Result, LOG_LEVEL_DEBUG, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_WARNING, LOG_LEVEL_CRITICAL, LOG_LEVEL_ERROR, LOG_LEVEL_RESERVED, epoch_now, format_line, src_to_path, default_root, }; ``` ### Wire format Five tab-separated columns, ending with `\n`: ``` epoch<TAB>contextid<TAB>loglevel<TAB>tags<TAB>content\n ``` `src` is encoded by the file path and is never inside the line. `job_id` is not part of the model - callers encode it inside `src` if they want it. ### Acceptance criteria - [x] `pub mod logger;` in `crates/core/src/lib.rs`. - [x] Public surface: `LogSink`, `Logger`, `LogEntry`, `LoggerConfig`, `LoggerError`, `Result`, `LogLevel`, `OverflowMode`. - [x] `LogEntry` has exactly the six specified fields with the exact types. - [x] `LoggerConfig` defaults match the issue. - [x] `src` mapping: 1/2/3 parts produce the documented paths; >3 parts is rejected. - [x] Output line is exactly `epoch\tcontextid\tloglevel\ttags\tcontent\n`; `src` does not appear in the line. - [x] Empty `tags` renders as `-`; non-empty tags joined with `,`. - [x] Content sanitization: tabs -> space; `\n`, `\r`, and `\r\n` -> `[BR]`. - [x] Loglevel constants map 0/10/20/30/40/254/255 -> debug/info/notice/warning/critical/error/reserved. - [x] Writer runs on a dedicated `std::thread` named `hero-logger`; no tokio dependency added. - [x] LRU cache evicts beyond `max_open_files`; evicted writers are flushed. - [x] Files opened with `create(true).append(true)`; parent dirs auto-created. - [x] Size-based rotation chain `name.log.1`...`name.log.{max_rotated_files}`; oldest dropped. - [x] Three overflow modes implemented; default `DropDebugFirst`. - [x] `dropped_debug`, `dropped_info`, `dropped_error` counters readable from `Logger`. - [x] `shutdown()` is idempotent, drains pending records, flushes, optionally fsyncs, joins. - [x] Integration tests cover routing, rotation, sanitization, drop counters, and shutdown semantics. - [x] `cargo build -p herolib_core` and `cargo test -p herolib_core` succeed. ### Notes / decisions - `job_id` removed per follow-up clarification on this issue. `src` remains 1-3 dotted parts; callers encode any job identifier they want there. - No new Cargo dependencies. `thiserror`, `serde`, `dirs` (target-gated), and `std` cover everything; `tempfile` (already a dev-dep) is used in the writer's inline tests and the integration test. - `DropDebugFirst` semantics with a single bounded queue: when the channel is full, drop incoming `loglevel == 0` records and bump `dropped_debug`; non-debug records fall back to `Block` so they are never silently lost. - `epoch == 0` is treated as a sentinel and replaced with `epoch_now()` at format time. Real Unix timestamps in the system are always far above zero. - Rotation uses `std::fs::rename` per slot - a crash mid-rotation can leave a numbering gap, which is acceptable for non-critical logs. - Default root is `dirs::home_dir().join("hero/var/logs")` on native targets and `/tmp/hero/var/logs` on `wasm32`.

despiegk referenced this issue from a commit

2026-05-01 04:42:22 +00:00

feat(core/logger): append-only file logger with src-path routing

despiegk referenced this issue from a commit

2026-05-01 04:42:22 +00:00

feat(core/logger): query API, source listing, perf tests, README

despiegk closed this issue

2026-05-01 05:11:08 +00:00

despiegk referenced this issue from a commit

2026-05-01 05:16:35 +00:00

feat(core/logger): cleanup APIs for retention

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

lhumina_code/hero_lib#133

No description provided.

Rows
Columns

implement a logger we can reuse everywhere #133

Hero Logs — Fast Append-Only Logging Spec

Goal

Log directory

Log entry

Line format

Field rules

epoch

contextid

job_id

loglevel

tags

content

Writer architecture

Performance setting

File handling

Rotation

Overflow behavior

Shutdown

Minimal API

Final recommended setting

Hero Logs — Implementation Spec for Issue #133

Objective

Requirements

Files to Modify / Create

Implementation Plan

Step 1 - Module skeleton and public surface stubs

Step 2 - LogEntry, sanitization, formatting, src->path mapping

Step 3 - Writer thread (LRU cache, rotation, flush)

Step 4 - LogSink trait + Logger implementation

Step 5 - Integration tests

Step 6 - Module documentation

Acceptance Criteria

Notes

Hero Logs - Implementation Spec for Issue #133 (Revised)

Objective

Requirements

Files to Modify / Create

Implementation Plan

Step 1 - Module skeleton and public surface stubs

Step 2 - LogEntry, sanitization, formatting, src->path mapping

Step 3 - Writer thread (LRU cache, rotation, flush)

Step 4 - LogSink trait + Logger implementation

Step 5 - Integration tests

Step 6 - Module documentation

Acceptance Criteria

Notes

Test Results

New logger unit tests (15 in crates/core/src/logger/)

Integration tests (crates/core/tests/logger.rs)

Implementation Summary

Files created

File modified

Public API

Wire format

Acceptance criteria

Notes / decisions

`epoch`

`contextid`

`job_id`

`loglevel`

`tags`

`content`

Step 2 - `LogEntry`, sanitization, formatting, src->path mapping

Step 4 - `LogSink` trait + `Logger` implementation

Step 2 - `LogEntry`, sanitization, formatting, src->path mapping

Step 4 - `LogSink` trait + `Logger` implementation

New logger unit tests (15 in `crates/core/src/logger/`)

Integration tests (`crates/core/tests/logger.rs`)