System

System is the first place to look when something isn’t right.

Section	Shows
Components	The health of PostgreSQL, ClickHouse and the flow collector
Exporters	Per exporter: datagrams, records, drops, sequence loss, export lag, clock skew, sampling rate and source
Ingest	Queue drops, records held while a new exporter’s sampling rate is unknown, and the kernel’s receive buffer
ClickHouse writer	Rows written and dropped, insert latency (median and 99th percentile)
Detection	Hosts tracked, drops, evaluation time, and the baseline state (the hour loaded, how many hosts have one)
BGP	Each session’s state, uptime and last error
Datasets	Each dataset’s last refresh, row count and any error
Storage	Disk use per table and a days-until-full projection

It also tells you when something needs doing:

• A new release is out, with its notes and the upgrade commands. See Upgrading.
• The compose file is older than the image needs, or ClickHouse runs an outdated configuration. See Common problems.
• A reverse proxy isn’t trusted: its address, and the HN_HTTP_TRUSTED_PROXIES line to add.
• The UDP receive buffer couldn’t be raised, and the sysctl to set on the host.

Admins can generate a support file here: see Support file.

Health checks and alerts

Heimnull checks itself every minute and can notify you (the Heimnull health event, see Notifications) when:

• a ClickHouse disk has less than 10% free;
• ClickHouse reaches its memory limit;
• merges fall behind (a partition with more than 300 parts);
• ClickHouse runs an older configuration than the compose file, or the compose file is older than the image needs;
• an exporter’s SNMP polls have failed for 10 minutes.

A problem is reported after two failing checks in a row, again every 6 hours while it lasts, and as recovered after 10 minutes clear. Each message says what to do.

For monitoring from outside, see API and metrics.