lhumina_code/hero_lib
20
0
Fork 1
Code Issues 16 Pull requests 1 Projects Releases 2 Packages 1 Wiki Activity Actions

add features in crate webserver #122

New issue
Open
opened 2026-03-22 06:24:44 +00:00 by despiegk · 3 comments
despiegk commented 2026-03-22 06:24:44 +00:00
Owner
Copy link

crates/webserver

We're developing a web server, and it needs to have integrated DNS server, so implemented the NS server, with support for the multiple records, needs to be the submodule to the create web server. Call a DNS. There is a full-bloom CCULite database backend. That probably needs to be as a separate model soup module of the Great Web server, and in there, but all the information to do with the proxy, as well as visitors. So we need to track all the visitors. That's really important. Um, but also like the proxy rules, they need to be very flexible, like on port, prefix, post-fixed, you know, names. And then we need to lock all the requests. We need to have a generic login form. Which can be rebranded through a template. We need to be able to store that template so that we can say on this URL or this port, we use, if it's like a slash login, we give you a default login page where and then users can log in. It needs to have proper odd server support, so we need to be able to configure that too. So we can, people can log in into this proxy. They are known into that database. They can login in multiple modes. It can be with email only, without a check, with email and a check. Or it needs to be pluggable towards the future. For now, the standard is just email recognition itself. We give some, we can insert JavaScript code so that we track on the proxy, so whatever the user goes through, we track which locations they're on for how long, so we can check which user is how long onto something. We recognize the user by means of local storage in the browser so that if they get to the proxy again and they get onto a certain location and it's only um, email as an example, so we have different authentication levels, but if it's only email without confirmation, and that's already known, there is no reason for the user to do that again. Because it's a proxy, you can go to the back end. Many different things, right? And we just store all the requests time where the user was going to, and then it can go to the back end to any different unique based and location. It does HTPS, let's encrypt DNS, we can configure the DNS names as well. So it's a full-blown DNS pack and There's an open RPC interface fully specs in line with the server. And all of that is implemented as a library, so we can reuse it. Anywhere, it can use macros if needed, documented well. And make an example script so we can also test it well and use the browser MCP to test it.

crates/webserver We're developing a web server, and it needs to have integrated DNS server, so implemented the NS server, with support for the multiple records, needs to be the submodule to the create web server. Call a DNS. There is a full-bloom CCULite database backend. That probably needs to be as a separate model soup module of the Great Web server, and in there, but all the information to do with the proxy, as well as visitors. So we need to track all the visitors. That's really important. Um, but also like the proxy rules, they need to be very flexible, like on port, prefix, post-fixed, you know, names. And then we need to lock all the requests. We need to have a generic login form. Which can be rebranded through a template. We need to be able to store that template so that we can say on this URL or this port, we use, if it's like a slash login, we give you a default login page where and then users can log in. It needs to have proper odd server support, so we need to be able to configure that too. So we can, people can log in into this proxy. They are known into that database. They can login in multiple modes. It can be with email only, without a check, with email and a check. Or it needs to be pluggable towards the future. For now, the standard is just email recognition itself. We give some, we can insert JavaScript code so that we track on the proxy, so whatever the user goes through, we track which locations they're on for how long, so we can check which user is how long onto something. We recognize the user by means of local storage in the browser so that if they get to the proxy again and they get onto a certain location and it's only um, email as an example, so we have different authentication levels, but if it's only email without confirmation, and that's already known, there is no reason for the user to do that again. Because it's a proxy, you can go to the back end. Many different things, right? And we just store all the requests time where the user was going to, and then it can go to the back end to any different unique based and location. It does HTPS, let's encrypt DNS, we can configure the DNS names as well. So it's a full-blown DNS pack and There's an open RPC interface fully specs in line with the server. And all of that is implemented as a library, so we can reuse it. Anywhere, it can use macros if needed, documented well. And make an example script so we can also test it well and use the browser MCP to test it.
despiegk commented 2026-03-22 06:30:04 +00:00
Author
Owner
Copy link

Implementation Spec for Issue #122: Add Features in Crate Webserver

Objective

Expand the existing herolib_webserver crate from a UDS-to-HTTP/HTTPS proxy library into a comprehensive, reusable web server library with integrated DNS, SQLite storage, visitor tracking, flexible proxy rules, authentication, login forms, HTTPS with Let's Encrypt, and an OpenRPC interface.

Requirements

  • R1: Integrated DNS server submodule supporting A, AAAA, CNAME, MX, TXT, NS, SRV record types
  • R2: SQLite database backend for storing proxy rules, visitor records, request logs, user accounts, sessions
  • R3: Visitor tracking — identify visitors via browser local storage tokens, record URLs visited, page duration
  • R4: Flexible proxy rules — configurable by port, prefix, postfix, domain name, stored in SQLite
  • R5: Request logging — log all HTTP requests with timestamp, path, method, status, user association
  • R6: Generic login form — rebrandable HTML template, mountable at configurable paths per port
  • R7: Auth server — pluggable AuthProvider trait; initial implementation is email-only (with optional confirmation)
  • R8: User session persistence — returning users recognized via local storage token to skip re-auth
  • R9: HTTPS with Let's Encrypt — extend existing TLS for multi-domain support and DNS integration
  • R10: DNS name configuration — manage DNS zones/records via API and SQLite storage
  • R11: OpenRPC interface — fully specified JSON-RPC API for all server management operations
  • R12: Library architecture — all features as reusable library modules with feature flags
  • R13: Example scripts and testing

Implementation Plan (10 Steps)

Step 1: Update Cargo.toml with feature flags and dependencies

Add feature flags (db, dns, auth, tracking, openrpc, full) and deps (rusqlite, hickory-server, hickory-proto, uuid, chrono).

Step 2: Implement SQLite database submodule (src/db/)

Schema, migrations, models, and CRUD operations for proxy_rules, visitors, request_logs, users, sessions, dns_records.

Step 3: Implement DNS server submodule (src/dns/)

UDP/TCP DNS listener using hickory-server, SQLite-backed record resolution with upstream fallback.

Step 4: Implement auth submodule (src/auth/)

AuthProvider trait, email-only provider, session management, axum middleware, login form handlers with embedded HTML templates.

Step 5: Implement visitor tracking and request logging (src/tracking/, src/logging/)

Tower middleware for visitor identification (cookie/local-storage), request logging to SQLite, JS snippet for client-side tracking.

Step 6: Extend proxy rules for domain/postfix routing

Add RuleEngine backed by SQLite, domain-matching middleware, DynamicRules bind mode.

Step 7: Implement OpenRPC interface (src/rpc/)

JSON-RPC 2.0 router with methods for proxy rules, DNS, users, sessions, visitors, logs, server status. Spec served at /rpc/discover.

Step 8: Implement top-level WebServer builder (src/server.rs)

Fluent builder composing all modules, axum Router with layered middleware, serve() and spawn() methods.

Step 9: Extend HTTPS/Let's Encrypt for multi-domain and DNS integration

Multi-domain TLS, DNS-01 challenge support via integrated DNS server.

Step 10: Write examples and integration tests

examples/basic_server.rs, examples/dns_server.rs, tests/server_integration.rs.

Acceptance Criteria

  • All existing proxy/TLS tests pass
  • cargo check -p herolib_webserver --features full compiles
  • Default features (proxy + tls only) still compile — no regressions
  • SQLite DB created and migrated on open
  • Proxy rules CRUD via Rust API and OpenRPC
  • DNS server resolves configured records, falls back to upstream
  • Visitor tracking assigns/persists visitor IDs
  • Request logging records all HTTP requests
  • Login form served and processes email auth
  • Session tokens persist (cookie + local storage)
  • OpenRPC spec served at /rpc/discover
  • make fmt and make lint pass
  • All public APIs have doc comments
  • Examples compile

Notes

  • rusqlite with bundled compiles C SQLite from source — if CI forbids C deps, will use redb or limbo as alternatives
  • DNS port 53 requires elevated privileges; alternative ports supported for dev/testing
  • Existing SocketProxy API remains unchanged (backward compatible)
  • Each capability is behind its own feature flag for modularity
## Implementation Spec for Issue #122: Add Features in Crate Webserver ### Objective Expand the existing `herolib_webserver` crate from a UDS-to-HTTP/HTTPS proxy library into a comprehensive, reusable web server library with integrated DNS, SQLite storage, visitor tracking, flexible proxy rules, authentication, login forms, HTTPS with Let's Encrypt, and an OpenRPC interface. ### Requirements - **R1**: Integrated DNS server submodule supporting A, AAAA, CNAME, MX, TXT, NS, SRV record types - **R2**: SQLite database backend for storing proxy rules, visitor records, request logs, user accounts, sessions - **R3**: Visitor tracking — identify visitors via browser local storage tokens, record URLs visited, page duration - **R4**: Flexible proxy rules — configurable by port, prefix, postfix, domain name, stored in SQLite - **R5**: Request logging — log all HTTP requests with timestamp, path, method, status, user association - **R6**: Generic login form — rebrandable HTML template, mountable at configurable paths per port - **R7**: Auth server — pluggable `AuthProvider` trait; initial implementation is email-only (with optional confirmation) - **R8**: User session persistence — returning users recognized via local storage token to skip re-auth - **R9**: HTTPS with Let's Encrypt — extend existing TLS for multi-domain support and DNS integration - **R10**: DNS name configuration — manage DNS zones/records via API and SQLite storage - **R11**: OpenRPC interface — fully specified JSON-RPC API for all server management operations - **R12**: Library architecture — all features as reusable library modules with feature flags - **R13**: Example scripts and testing ### Implementation Plan (10 Steps) #### Step 1: Update Cargo.toml with feature flags and dependencies Add feature flags (`db`, `dns`, `auth`, `tracking`, `openrpc`, `full`) and deps (`rusqlite`, `hickory-server`, `hickory-proto`, `uuid`, `chrono`). #### Step 2: Implement SQLite database submodule (`src/db/`) Schema, migrations, models, and CRUD operations for proxy_rules, visitors, request_logs, users, sessions, dns_records. #### Step 3: Implement DNS server submodule (`src/dns/`) UDP/TCP DNS listener using hickory-server, SQLite-backed record resolution with upstream fallback. #### Step 4: Implement auth submodule (`src/auth/`) `AuthProvider` trait, email-only provider, session management, axum middleware, login form handlers with embedded HTML templates. #### Step 5: Implement visitor tracking and request logging (`src/tracking/`, `src/logging/`) Tower middleware for visitor identification (cookie/local-storage), request logging to SQLite, JS snippet for client-side tracking. #### Step 6: Extend proxy rules for domain/postfix routing Add `RuleEngine` backed by SQLite, domain-matching middleware, `DynamicRules` bind mode. #### Step 7: Implement OpenRPC interface (`src/rpc/`) JSON-RPC 2.0 router with methods for proxy rules, DNS, users, sessions, visitors, logs, server status. Spec served at `/rpc/discover`. #### Step 8: Implement top-level WebServer builder (`src/server.rs`) Fluent builder composing all modules, axum Router with layered middleware, `serve()` and `spawn()` methods. #### Step 9: Extend HTTPS/Let's Encrypt for multi-domain and DNS integration Multi-domain TLS, DNS-01 challenge support via integrated DNS server. #### Step 10: Write examples and integration tests `examples/basic_server.rs`, `examples/dns_server.rs`, `tests/server_integration.rs`. ### Acceptance Criteria - [ ] All existing proxy/TLS tests pass - [ ] `cargo check -p herolib_webserver --features full` compiles - [ ] Default features (proxy + tls only) still compile — no regressions - [ ] SQLite DB created and migrated on open - [ ] Proxy rules CRUD via Rust API and OpenRPC - [ ] DNS server resolves configured records, falls back to upstream - [ ] Visitor tracking assigns/persists visitor IDs - [ ] Request logging records all HTTP requests - [ ] Login form served and processes email auth - [ ] Session tokens persist (cookie + local storage) - [ ] OpenRPC spec served at `/rpc/discover` - [ ] `make fmt` and `make lint` pass - [ ] All public APIs have doc comments - [ ] Examples compile ### Notes - `rusqlite` with `bundled` compiles C SQLite from source — if CI forbids C deps, will use `redb` or `limbo` as alternatives - DNS port 53 requires elevated privileges; alternative ports supported for dev/testing - Existing `SocketProxy` API remains unchanged (backward compatible) - Each capability is behind its own feature flag for modularity
despiegk commented 2026-03-22 07:02:34 +00:00
Author
Owner
Copy link

Test Results

  • Unit tests: 60 passed, 0 failed
  • Integration tests: 6 passed, 0 failed
  • Doc tests: 27 passed, 0 failed, 3 ignored
  • Total: 93 passed, 0 failed
  • cargo check --features full: clean (0 warnings)
  • cargo check (default features): clean
  • cargo fmt --check: clean

Implementation Summary

New Modules Created (all feature-gated)

Module Feature Description
db/ db SQLite storage — schema, migrations, CRUD for proxy_rules, visitors, request_logs, users, sessions, dns_records
dns/ dns Integrated DNS server — UDP+TCP listener, SQLite-backed resolver, upstream forwarding, zone management
auth/ auth Authentication — AuthProvider trait, email-only provider, session manager, axum middleware, login form with embedded HTML
tracking/ tracking Visitor tracking middleware — cookie/header-based visitor ID, JS snippet for client-side tracking
logging/ tracking Request logging middleware — records method, path, status, duration, visitor/user to SQLite
rpc/ openrpc OpenRPC JSON-RPC 2.0 API — 17 methods for proxy rules, DNS, users, sessions, visitors, logs, server status
server.rs proxy WebServer builder — composes all modules into a single server with fluent API

New Feature Flags

db, dns, auth, tracking, openrpc, full (enables all)

Files Modified

  • Cargo.toml — new features and dependencies
  • src/lib.rs — module declarations, updated docs
  • src/error.rs — new error variants (Database, Dns, Auth, Config)
  • src/proxy/mod.rs — added rules submodule and DynamicRules support
  • src/proxy/config.rs — DynamicRules bind mode, db field on builder
  • src/proxy/router.rs — serve_dynamic() for database-backed routing

Examples

  • examples/basic_server.rs — WebServer with auth + tracking + RPC
  • examples/dns_server.rs — WebServer with DNS + zone management
## Test Results - **Unit tests**: 60 passed, 0 failed - **Integration tests**: 6 passed, 0 failed - **Doc tests**: 27 passed, 0 failed, 3 ignored - **Total**: 93 passed, 0 failed - `cargo check --features full`: clean (0 warnings) - `cargo check` (default features): clean - `cargo fmt --check`: clean ## Implementation Summary ### New Modules Created (all feature-gated) | Module | Feature | Description | |--------|---------|-------------| | `db/` | `db` | SQLite storage — schema, migrations, CRUD for proxy_rules, visitors, request_logs, users, sessions, dns_records | | `dns/` | `dns` | Integrated DNS server — UDP+TCP listener, SQLite-backed resolver, upstream forwarding, zone management | | `auth/` | `auth` | Authentication — `AuthProvider` trait, email-only provider, session manager, axum middleware, login form with embedded HTML | | `tracking/` | `tracking` | Visitor tracking middleware — cookie/header-based visitor ID, JS snippet for client-side tracking | | `logging/` | `tracking` | Request logging middleware — records method, path, status, duration, visitor/user to SQLite | | `rpc/` | `openrpc` | OpenRPC JSON-RPC 2.0 API — 17 methods for proxy rules, DNS, users, sessions, visitors, logs, server status | | `server.rs` | `proxy` | `WebServer` builder — composes all modules into a single server with fluent API | ### New Feature Flags `db`, `dns`, `auth`, `tracking`, `openrpc`, `full` (enables all) ### Files Modified - `Cargo.toml` — new features and dependencies - `src/lib.rs` — module declarations, updated docs - `src/error.rs` — new error variants (Database, Dns, Auth, Config) - `src/proxy/mod.rs` — added `rules` submodule and `DynamicRules` support - `src/proxy/config.rs` — `DynamicRules` bind mode, `db` field on builder - `src/proxy/router.rs` — `serve_dynamic()` for database-backed routing ### Examples - `examples/basic_server.rs` — WebServer with auth + tracking + RPC - `examples/dns_server.rs` — WebServer with DNS + zone management
despiegk commented 2026-03-22 07:08:54 +00:00
Author
Owner
Copy link

Implementation committed: 73e9def62ac769cda3cad07ccda184ba0228617b

Browse: 73e9def62a

Implementation committed: `73e9def62ac769cda3cad07ccda184ba0228617b` Browse: https://forge.ourworld.tf/lhumina_code/hero_lib/commit/73e9def62ac769cda3cad07ccda184ba0228617b
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
development
development-google-cli-wrapper
main
v0.3.14
v0.3.5
v0.1.0
v0.0.1-rc1
v0.3.4
v0.3.3
v0.3.2
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lhumina_code/hero_lib#122
No description provided.