Observability on a DuckDB Backbone: Arc, LogLens, and the New Era of In-DB Analytics for Logs

Observability just got a DuckDB boost. Arc unifies metrics, logs, traces, and events in one SQL-first shelf, using DuckDB as the engine with MessagePack, Arrow, and Parquet under the hood. It’s humming—about 2.45M RPS for metrics and around 1M RPS for logs, events, and traces ^[1]. Arc is OSS under AGPL-3.0; there’s a VS Code extension to explore Arc, plus Apache Superset integration and progress on Telegraf and a Grafana datasource ^[1].

Arc stack and capabilities - Arc uses DuckDB as the SQL engine, with MessagePack, Arrow, and Parquet under the hood ^[1]. - Retention policies, deletes, and continuous queries that aggregate data as it arrives ^[1]. - VS Code extension, Apache Superset integration; Telegraf and a Grafana datasource in the works ^[1].

LogLens: fast structured-log queries - LogLens is a fast CLI tool written in Rust for structured logs ^[2]. - It blends a simple SQL-like query language with parallel processing via rayon and fast file access via memmap2 ^[2]. - It’s closed-source, freemium, with a Free tier and a Pro tier; a $79/year license covers updates and includes a perpetual fallback ^[2]. - Website: getloglens.com and docs at getloglens.com/docs ^[2].

Implications for observability workloads - A DuckDB backbone lets DB teams centralize metrics, logs, traces, and events, enabling ad-hoc analysis in one place ^[1]. - LogLens speeds up exploration of large structured logs with SQL-like queries, signaling a new pattern for in-DB analytics on logs ^[2]. - Licensing realities matter: Arc is OSS under AGPL-3.0 ^[1], while LogLens is closed-source freemium ^[2].

Watch next: broader adoption of this stack, with more Grafana/Telegraf integrations and deeper in-DB analytics for logs.

References