Alerting

Logchef evaluates SQL queries on a schedule and sends notifications when thresholds are exceeded. Alerts fire via email (SMTP) and webhooks — no external alerting system required.

Quick Start

1. Navigate to your team/source → Alerts → New Alert
2. Write a filter condition: severity_text = "ERROR"
3. Set threshold: greater than 100
4. Set lookback: 5 minutes, frequency: 60 seconds
5. Add recipients and/or webhook URLs
6. Save

Logchef generates the SQL automatically (SELECT count(*) ... WHERE severity_text = 'ERROR' AND timestamp >= now() - toIntervalSecond(300)), evaluates it every 60 seconds, and sends notifications when the count exceeds 100.

Query Modes

LogchefQL mode — Write a filter condition, pick an aggregate function. Time filter is added automatically.

severity_text = "ERROR"
status_code >= 500 and service = "api"
body ~ "timeout"

SQL mode — Full ClickHouse SQL. Must return a single numeric column named value.

SELECT avg(JSONExtractFloat(log_attributes, 'response_time_ms')) as value
FROM logs
WHERE service = 'api-gateway'
  AND timestamp >= now() - toIntervalSecond(600)

Alert Configuration

Field	Description
Name	Human-readable identifier
Severity	info, warning, or critical
Query	LogchefQL condition or ClickHouse SQL
Threshold	Value + operator (>, >=, <, <=, ==, !=)
Frequency	Evaluation interval in seconds
Lookback	Time range for the query
Recipients	Team members to email
Webhook URLs	HTTP endpoints to POST payloads to

Notifications

Email (SMTP)

Configure SMTP in Administration → System Settings → Alerts:

Setting	Example
SMTP Host	smtp.example.com
SMTP Port	587
SMTP Security	starttls
SMTP From	alerts@example.com

Webhooks

Add webhook URLs per alert. Logchef POSTs a JSON payload containing:

• Alert name, severity, status (triggered / resolved)
• The query result value and threshold
• Labels (team, source, custom key-value pairs)
• Annotations (description, runbook URL, query text)

Works with Slack incoming webhooks, PagerDuty, or any HTTP endpoint.

Labels and Annotations

Add custom labels for routing and annotations for context:

{
  "labels": { "env": "production", "service": "payment-api" },
  "annotations": { "runbook": "https://wiki.example.com/high-error-rate" }
}

Reliability

• Failed deliveries retry with exponential backoff (500ms → 1s → 2s)
• Delivery outcomes recorded in alert history
• Resolution notifications sent when conditions clear

Dashboard

The alerts list shows live status for each rule:

• Toggle alerts on/off directly from the list
• Red pulsing dot = firing, green dot = resolved
• Duplicate existing alerts for similar conditions
• View full evaluation history per alert

SMTP Configuration (First Boot)

Seed initial settings via config.toml — after first boot, use the Admin UI:

[alerts]
enabled = true
evaluation_interval = "1m"
smtp_host = "smtp.example.com"
smtp_port = 587
smtp_from = "alerts@example.com"
smtp_security = "starttls"

Troubleshooting

Alerts not firing: Check the alert is enabled, test the query manually, verify the threshold makes sense for your data volume.

Alerts not delivered: Verify SMTP settings, check that recipients or webhook URLs are configured on the alert, review delivery status in alert history.

False positives: Increase the threshold, extend the lookback window, or use avg() instead of count() for smoother signals.