Alerting

Logchef evaluates alert queries on a schedule and sends notifications when thresholds are exceeded. Alerts fire via email (SMTP) and webhooks, so you don’t need a separate alerting system.

Quick Start

1. Navigate to your team/source → Alerts → New Alert
2. Choose a query mode supported by the source
3. Write a filter or native query
4. Set threshold: greater than 100
5. Set lookback: 5 minutes, frequency: 60 seconds
6. Add recipients and/or webhook URLs
7. Save

For ClickHouse and VictoriaLogs sources, Logchef can generate the executable alert query automatically from a LogchefQL condition. Native mode remains available when you want to write SQL or LogsQL directly.

Query Modes

Condition mode: Write a LogchefQL filter condition and pick an aggregate function (count, sum, avg, min, max). For anything beyond count, also pick the field to aggregate over: schema-backed suggestions show all fields for VictoriaLogs sources and numeric columns for ClickHouse. Logchef generates the backend-native executable query and adds the alert window automatically.

severity_text = "ERROR"
status_code >= 500 and service = "api"
body ~ "timeout"

Native mode: Write the source’s native alert query. The query must return a single numeric value.

SELECT avg(JSONExtractFloat(log_attributes, 'response_time_ms')) as value
FROM logs
WHERE service = 'api-gateway'
  AND timestamp >= now() - toIntervalSecond(600)

For VictoriaLogs sources, native mode uses LogsQL and evaluates through stats_query. Logchef applies the configured lookback window automatically, so the query can stay focused on the filter and stats logic.

level:="error" | stats count() as value

Alert Configuration

Field	Description
Name	Human-readable identifier
Severity	info, warning, or critical
Query	LogchefQL condition, ClickHouse SQL, or VictoriaLogs LogsQL depending on source
Threshold	Value + operator (>, >=, <, <=, ==, !=)
Frequency	Evaluation interval in seconds
Lookback	Time range for the query
Recipients	Team members to email
Webhook URLs	HTTP endpoints to POST payloads to

Notifications

Email (SMTP)

Configure SMTP in Administration → System Settings → Alerts:

Setting	Example
SMTP Host	smtp.example.com
SMTP Port	587
SMTP Security	starttls
SMTP From	alerts@example.com

Webhooks

Add webhook URLs per alert. Logchef POSTs a JSON payload containing:

• Alert name, severity, status (triggered / resolved)
• The query result value and threshold
• Labels (team, source, custom key-value pairs)
• Annotations (description, runbook URL, query text)

Works with Slack incoming webhooks, PagerDuty, or any HTTP endpoint.

Labels and Annotations

Add custom labels for routing and annotations for context:

{
  "labels": { "env": "production", "service": "payment-api" },
  "annotations": { "runbook": "https://wiki.example.com/high-error-rate" }
}

Reliability

• Failed deliveries retry with exponential backoff (500ms → 1s → 2s)
• Delivery outcomes recorded in alert history
• Resolution notifications sent when conditions clear

Dashboard

The alerts list shows live status for each rule:

• Toggle alerts on/off directly from the list
• Red pulsing dot = firing, green dot = resolved
• Duplicate existing alerts for similar conditions
• View full evaluation history per alert

SMTP Configuration (First Boot)

Seed initial settings via config.toml. After first boot, use the Admin UI:

[alerts]
enabled = true
evaluation_interval = "1m"
smtp_host = "smtp.example.com"
smtp_port = 587
smtp_from = "alerts@example.com"
smtp_security = "starttls"

enabled is the server-wide alerting switch. With alerts.enabled = false (or LOGCHEF_ALERTS__ENABLED=false) Logchef hides all alerting UI and every alert endpoint returns 503. That’s useful for exploration-only deployments, or to keep alert evaluation on a single instance when running multiple replicas against a shared Postgres backend.

Troubleshooting

Alerts not firing: Check the alert is enabled, test the query manually, verify the threshold makes sense for your data volume, and confirm the selected query language matches the source type.

Alerts not delivered: Verify SMTP settings, check that recipients or webhook URLs are configured on the alert, review delivery status in alert history.

False positives: Increase the threshold, extend the lookback window, or use avg() instead of count() for smoother signals.

For VictoriaLogs-specific guidance, see Using VictoriaLogs with Logchef.