Reference
Storage and retention
What Heimnull stores where, how long each kind of data is kept by default, how much disk it takes, and how searches are kept from hurting ingest.
Where data lives
Section titled “Where data lives”| Store | Holds | Volume |
|---|---|---|
| ClickHouse | Flow records, rollups, attack samples and series, per-minute peaks, SNMP counters, ingest statistics | heimnull_hn-ch |
| PostgreSQL | Users, configuration, attacks, mitigations, the audit log, notification history, the license, caches of lookups | heimnull_hn-pg |
| Datasets | Downloaded ASN, geolocation and address-range datasets | heimnull_hn-datasets |
Retention
Section titled “Retention”Settings → Retention sets how many days each kind of data is kept. A change applies at once: data older than a shortened retention is deleted right away, after a confirmation.
| Data | Default | Used for |
|---|---|---|
| Raw flows | 30 days | Flow records, host detail, recent ranges in Explore, re-classify |
| 1-minute rollups | 365 days | Interfaces, applications, ports, countries, TCP flags |
| 1-minute detail rollups | 90 days | ASes, prefixes, your hosts (many distinct values) |
| 1-hour rollups | Forever | Long-range charts |
| Hourly host detail | 365 days | Each host’s applications, ASes and countries beyond the raw retention |
| SNMP counters | 365 days | Interfaces, the accuracy check, Planning |
| Ingest statistics | 90 days | Per-exporter counters every 10 seconds |
| Attack samples | 365 days | Each attack’s sample flows and 1-second series |
| Per-host peaks | 90 days | The tuning preview and baselines |
| Hostgroup peaks | 365 days | The same, per hostgroup |
0 keeps data forever (raw flows need at least 1 day). The attack and mitigation records themselves are in PostgreSQL, apart from these.
Explore picks the most detailed data that covers the range you ask for: raw flows within their retention, then 1-minute rollups, then 1-hour rollups. Questions only raw flows can answer (flow records, conversations) are limited to the raw retention.
Disk use
Section titled “Disk use”- Raw flows take 6 to 36 bytes per stored record, compressed: about 3.5 to 8.5 GB a day for a router at 10 Gbit/s average with 1-in-512 packet sampling.
- 1-minute rollups take tens of GB at the default retention; 1-hour rollups a few GB a year.
- We recommend at least 500 GB of NVMe for 30 days of raw flows with headroom.
System → Storage shows each table’s size and a days-until-full projection: it’s the guide for adding storage. Raw flows fill the disk first, so shortening their retention is the quickest relief. A ClickHouse disk under 10% free triggers a system_health notification.
Searches and ingest
Section titled “Searches and ingest”Searches can’t slow down ingest or detection:
- They run as a read-only ClickHouse user with its own limits: 30 seconds for a flow search, 10 for dashboards, 1.5 GB of memory, 4 threads, and at most 100,000 result rows (larger results are cut off). Inserts use a separate user.
- ClickHouse’s CPU ceiling (
HN_CH_CPUS) keeps the heaviest search from taking the CPUs the collector needs. - Every query is parameterized: nothing you type is ever pasted into SQL.
A search that hits its limit fails with a message; ingest and detection carry on. Narrow the range or the filter, or use a rollup-backed view.
If ClickHouse is down
Section titled “If ClickHouse is down”The collector keeps up to 1.5 million records in memory and inserts them when ClickHouse is back; beyond that, it drops the oldest batches, counts them, and notifies (ingest_problem). Detection, Live and mitigation don’t depend on ClickHouse and carry on.