Skip to content
Logchef

Architecture

Architectural Overview

LogChef is architected as a specialized query and visualization layer on top of ClickHouse. Its design emphasizes a clear separation of concerns:

  • Query Engine: Core focus on transforming user queries into optimized ClickHouse SQL
  • No Ingestion Pipeline: The architecture intentionally excludes log collection, focusing exclusively on the query interface
  • ClickHouse Integration: Deep integration with ClickHouse’s query capabilities while maintaining schema flexibility

This architectural approach allows LogChef to leverage the existing ecosystem of log collection tools while providing a specialized interface for exploring logs once they’re in ClickHouse.

System Overview

LogChef Architecture Diagram

Technology Stack

Backend

  • Go: LogChef’s core backend is written in Go, providing high performance, concurrency, and efficient resource utilization
  • SQLite: Lightweight database used for metadata storage of users, teams, sources, and saved queries
  • ClickHouse: High-performance columnar database optimized for analytical queries on log data

Frontend

  • Vue.js: Modern JavaScript framework used to build the reactive user interface
  • Tailwind CSS: Utility-first CSS framework for styling the UI components

Core Components

1. Query Engine

  • Converts simple search syntax to optimized ClickHouse SQL
  • Manages query execution across multiple sources
  • Supports both simple search syntax and raw SQL for complex queries

2. Authentication Service

  • Integrates with OIDC providers (like Keycloak, Zitadel etc)
  • Manages user sessions and authorization
  • Enforces role-based access control

3. Source Manager

  • Manages connections to remote ClickHouse instances
  • Handles source registration and validation
  • Provides connection pooling mechanisms

Data Flow

  1. Log Ingestion (external to LogChef):

    • Various collectors (Vector, Filebeat, etc.) send logs to ClickHouse
    • Each collector handles its own schema mapping and transformations

  2. Log Querying (LogChef’s domain):

    • Users construct queries via the UI (simple syntax or SQL)
    • LogChef translates simple syntax to optimized ClickHouse SQL
    • Queries are executed against the appropriate ClickHouse source(s)
    • Results are processed, formatted, and displayed in the UI

Data Storage

SQLite Metadata Store

SQLite manages all system configuration and relationships:

  • Sources: Connection details to remote ClickHouse databases
  • Users: Account information and authentication data
  • Teams: Organizational units with role-based access
  • Saved Queries: Team-specific saved queries (supports both simple syntax and raw SQL)

ClickHouse Log Storage

LogChef connects to multiple remote ClickHouse databases as sources:

  • Schema Flexibility: Sources can:

    • Use the default OTEL schema as-is
    • Customize the built-in OpenTelemetry (OTEL) schema
    • Use custom schemas

  • Requirements: Only a timestamp field (DateTime/DateTime64) is mandatory

  • Schema Agnostic: Beyond timestamp, any column structure is supported

Deployment Considerations

  • Single Binary: LogChef runs as a lightweight single binary with minimal resource requirements
  • Stateless Operation: Core application is stateless for horizontal scaling (only SQLite metadata is persistent)
  • Proxying: Can be deployed behind reverse proxies like Nginx or Caddy

This architecture ensures:

  • Fast log querying across multiple data sources
  • Efficient metadata management
  • Scalable log storage and retrieval
  • Robust access controls
  • Clean and responsive user experience