Prometheus Metrics

Logchef exposes Prometheus metrics covering instance usage, performance, and health: query patterns, authentication events, and source connectivity.

Metrics Endpoint

All metrics are available at the /metrics endpoint in Prometheus format:

GET http://localhost:8125/metrics

Configure your Prometheus server to scrape this endpoint for monitoring and alerting.

Metric Categories

Logchef emits metrics across several categories:

• HTTP Metrics: Request/response patterns and performance
• Query Metrics: Database query execution and performance
• Authentication Metrics: Login attempts and session management
• Authorization Metrics: Access control failures and patterns
• ClickHouse Metrics: Database connection and operation health

HTTP Metrics

Monitor web interface performance and usage patterns.

Metric	Description	Type	Labels
logchef_http_requests_total	Total HTTP requests	Counter	method, endpoint, status, user_role
logchef_http_request_duration_seconds	HTTP request latency distribution	Histogram	method, endpoint
logchef_http_response_size_bytes	HTTP response size distribution	Histogram	method, endpoint
logchef_http_errors_total	HTTP errors by type	Counter	method, endpoint, error_type, status, user_role
logchef_http_active_requests	Currently active HTTP requests	Gauge	-

Key Use Cases:

• Identify slow endpoints with logchef_http_request_duration_seconds
• Monitor error rates by endpoint and role
• Track API usage patterns by endpoint
• Set up alerts for high error rates or latency spikes

Query Metrics

Track database query execution and performance patterns.

Metric	Description	Type	Labels
logchef_query_total	Total queries executed	Counter	source_id, source_name, database, table, query_type, result
logchef_query_duration_seconds	Query execution time distribution	Histogram	source_id, source_name, database, table
logchef_query_rows_returned	Distribution of rows returned	Histogram	source_id, source_name, database, table
logchef_query_timeouts_total	Query timeouts by source	Counter	source_id, source_name, database, table, query_type
logchef_query_errors_total	Query errors by type	Counter	source_id, source_name, database, table, error_type

Query Types: select, show, describe, explain, insert, create, drop, alter, other

Error Types: timeout, connection, syntax, permission, not_found, other

Key Use Cases:

• Monitor query performance per source/table
• Track query patterns and popular sources
• Set up alerts for high query failure rates

Authentication Metrics

Monitor login attempts and session management.

Metric	Description	Type	Labels
logchef_auth_attempts_total	Authentication attempts	Counter	method, result, user_role
logchef_session_operations_total	Session validations	Counter	operation, result, user_role

Auth Methods: token, oauth, oauth_mcp

Operations: validate

Key Use Cases:

• Monitor authentication failure rates
• Track session validation failures
• Identify suspicious login patterns
• Alert on failed authentication spikes

Authorization Metrics

Track access control failures and permission violations.

Metric	Description	Type	Labels
logchef_authorization_failures_total	Authorization failures	Counter	endpoint, reason, user_role

Failure Reasons: insufficient_role, insufficient_token_scope, oauth_admin_route

Key Use Cases:

• Monitor access control violations
• Identify roles hitting unauthorized endpoints
• Track permission boundary effectiveness

ClickHouse Metrics

Monitor database connection health and operation performance.

Metric	Description	Type	Labels
logchef_clickhouse_connection_status	ClickHouse connection health (0=down, 1=up)	Gauge	source_id, source_name, database, table, host
logchef_clickhouse_connection_validation_total	Connection validation attempts	Counter	source_id, source_name, database, table, host, result

Key Use Cases:

• Monitor ClickHouse connection health per source
• Set up alerts for database connectivity issues

Example Queries

Monitor Query Performance by Source

# Average query duration per source
rate(logchef_query_duration_seconds_sum[5m]) / rate(logchef_query_duration_seconds_count[5m])

# Query success rate by source
rate(logchef_query_total{result="success"}[5m]) / rate(logchef_query_total[5m])

Track Activity

# Query volume by source
topk(10, sum by (source_name) (rate(logchef_query_total[1h])))

# Authentication failure rate by role
sum by (user_role) (rate(logchef_auth_attempts_total{result="failure"}[5m]))

Monitor HTTP Performance

# Slowest endpoints (95th percentile)
histogram_quantile(0.95, rate(logchef_http_request_duration_seconds_bucket[5m]))

# Error rate by endpoint
rate(logchef_http_errors_total[5m]) / rate(logchef_http_requests_total[5m])

ClickHouse Health Monitoring

# Sources with connection issues
logchef_clickhouse_connection_status == 0

# Connection failure rate
rate(logchef_clickhouse_connection_validation_total{result="failure"}[5m])

Alerting Examples

High Query Failure Rate

- alert: HighQueryFailureRate
  expr: rate(logchef_query_total{result="failure"}[5m]) / rate(logchef_query_total[5m]) > 0.1
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "High query failure rate detected"
    description: "Query failure rate is {{ $value | humanizePercentage }} for source {{ $labels.source_name }}"

Authentication Failures

- alert: HighAuthFailureRate
  expr: rate(logchef_auth_attempts_total{result="failure"}[5m]) > 5
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "High authentication failure rate"
    description: "{{ $value }} failed authentication attempts per second"

ClickHouse Connection Down

- alert: ClickHouseConnectionDown
  expr: logchef_clickhouse_connection_status == 0
  for: 30s
  labels:
    severity: critical
  annotations:
    summary: "ClickHouse connection down"
    description: "Connection to {{ $labels.source_name }} ({{ $labels.host }}) is down"

Dashboard Recommendations

Dashboards worth building:

1. Overview Dashboard: Key metrics, query volume, error rates
2. Performance Dashboard: Query latency, HTTP response times
3. Security Dashboard: Authentication patterns, authorization failures
4. Source Health Dashboard: ClickHouse connection status, per-source performance

Next steps

• Build the dashboards above inside Logchef itself: see Dashboards
• Review Database Backends & High Availability if you’re scaling to multiple replicas