Common problems

Start at the System page: it shows each component’s health, every exporter’s ingest, the BGP sessions and datasets, and often says what to do. If this page doesn’t solve it, generate a support file and send it to us.

To see the containers’ state and logs:

cd /opt/heimnull
sudo docker compose ps
sudo docker compose logs --tail 100 heimnull

Installing

ClickHouse won’t start

clickhouse keeps restarting, docker compose up --wait never finishes, and its log or exit code says “Illegal instruction” or 132.

The CPU has no AVX2, which ClickHouse’s amd64 build needs. Check with grep -o -m1 avx2 /proc/cpuinfo. On Proxmox, set the VM’s CPU type to host or x86-64-v3, then stop and start the VM (a reboot from inside isn’t enough). On another hypervisor, pass AVX2 through. Physical CPUs without AVX2 (Xeon E5-2600 v2 and older) can’t run Heimnull. Heimnull’s own log names the problem at every start too.

env-template printed warnings

Each WARNING: line names something that won’t work or will run short: no AVX2, too little memory, too few CPUs, or Docker Desktop. See Requirements. .env is written anyway; the warnings only go to the terminal.

“Permission denied” writing .env

The redirect in sudo docker run … > .env runs in your own shell, which can’t write to root’s /opt/heimnull. Run the whole command inside root’s shell:

sudo sh -c 'umask 077; docker run --rm registry.heimnull.com/heimnull:latest env-template > .env'

Nothing arrives with Docker Desktop

Docker Desktop runs containers in its own VM, so Heimnull’s “host network” is that VM’s and your routers’ datagrams never reach it. Use Docker Engine on a Linux host.

Exporters

A router doesn’t appear

Check, in order:

1. Configure → Exporters: it may be there as pending, waiting for approval. Pending exporters’ flows are dropped.
2. The firewall on the Heimnull host, and anything between: the flow port, UDP, from the router’s address.
3. The router’s source address. Heimnull identifies an exporter by it. Set a fixed source address (a loopback) on the router.
4. Packets arriving at all: sudo tcpdump -n -c 10 udp port 2055 (or 4739, 6343) on the Heimnull host.

Every flow port accepts every format, so a “wrong” port isn’t the problem as long as Heimnull listens on it.

Records arrive but nothing is stored

• No template yet. NetFlow v9 and IPFIX records can’t be decoded until the router sends its template. After a Heimnull restart, some routers take minutes: MikroTik’s template refresh counts packets, not time. Set a time-based template refresh (see the router guides).
• No sampling rate yet. Heimnull holds a new exporter’s records for up to 2 minutes while its sampling rate is unknown. A Juniper MX announces its rate about every 80 seconds.

Rates are too high or too low

Compare the flows with SNMP: on the Interfaces page (with SNMP set up on the exporter), each interface shows flow-estimated traffic against its SNMP counters. A ratio far from 1 points at the cause:

• The sampling rate. Configure → Exporters shows the rate Heimnull uses and where it came from (the records, an options template, your setting). If the router doesn’t send it, set it. If the router sends a wrong one, use Override rate.
• Sampling on more than one interface counts the same packet twice: sample on the WAN only, or mark the other interfaces’ samples as copies (see Juniper MX).
• Interface roles. Without roles, traffic with neither address in your networks is “transit” and isn’t counted as inbound or outbound. Give the WAN interfaces the Transit or Peering role, and check Possibly missing networks on the Networks page.
• An sFlow switch over its sampling limit: the EX drops samples above 300 a second per line card, and every rate then reads low. See Juniper EX.

MikroTik: gigabits on an idle link (RouterOS 6)

RouterOS 6 writes its packet sampling rate with the bytes in the wrong order, so 1 in 10 arrives as more than 160 million. A few packets to the router then show as gigabits.

• Turn packet sampling off on RouterOS 6: /ip traffic-flow set packet-sampling=no. It’s unreliable on that release.
• Upgrade Heimnull to v3.1.1 or later, which recognizes the swapped value and reads it the right way round. The exporter’s page says when it does.
• If the exporter still shows a sampling rate other than 1 once sampling is off, set its Override rate to 1.

Stored hours keep the totals they were stored with. See MikroTik RouterOS 6.

MikroTik: traffic far below the interface counters

FastTrack sends established connections around most of RouterOS’s packet processing, and traffic-flow then misses much of the forwarded traffic, especially long downloads. Disable the fasttrack-connection firewall rules, and check the router’s CPU afterward. See FastTrack.

MikroTik: traffic reads about 7% low

RouterOS reports its sampling ratio rounded down: an interval of 5 and a space of 49 (1 in 10.8) arrives as 10. Use an interval of 1 and a space of N−1 (space 99 for 1 in 100), which makes the reported rate exact.

The Live page lags 15 to 30 seconds

Routers that export from a flow cache (MikroTik, J-Flow, most NetFlow v9) send each flow only when it’s been idle for the inactive timeout or has run for the active timeout. Heimnull counts the traffic when it arrives, and shades the seconds still filling in. Lower the router’s inactive timeout (5 to 10 seconds) and keep the active timeout at 30 seconds or less. Packet sampling (Juniper MX inline monitoring, sFlow) has no such lag.

Drops under load

System → Ingest shows queue drops, the ClickHouse writer’s dropped rows, and detection drops. For drops in the kernel before Heimnull sees the packets, check the UDP receive buffer errors:

netstat -su | grep -i 'receive buffer'

Mitigation

A mitigation didn’t appear on the router

1. Mitigations → the mitigation’s page: its history says if it was refused and why (outside your networks, whitelisted, the cap on active mitigations, the emergency stop, a FlowSpec rule with nothing to match), and what each session actually received.
2. Configure → BGP: the session is established, Announce mitigations to this session is on, the blackhole community is set, and for FlowSpec, the family is on and the router offered it when the session opened (the page flags families the router lacks).
3. The router’s import policy: it accepts host routes inside your networks with your community. The router guides have the checks.
4. The license: in Free mode, nothing is announced. Configure → License shows what’s in force.

FlowSpec always escalates to a blackhole

For inbound attacks, escalation looks at the rate delivered past the router. If your router samples only on input, before the FlowSpec filter, Heimnull only ever sees the arriving rate, which FlowSpec doesn’t change. Sample the output of your internal units too, and turn on Sampled on both sides for the exporter. See Juniper MX.

Web interface and access

Everyone shows the proxy’s address

Login throttling and the audit log show your reverse proxy’s address for every user: the proxy isn’t trusted. Add its address to HN_HTTP_TRUSTED_PROXIES in .env, and run sudo docker compose up -d --wait. A proxy in a Docker container connects from its bridge network, not 127.0.0.1. The System page names an untrusted proxy and the line to add. See HTTPS.

“Origin” errors when saving

Heimnull refuses changes whose Origin header names another host. Make sure the reverse proxy passes the Host header unchanged.

The admin password is lost

There are no default credentials. Reset any account’s password from the server:

cd /opt/heimnull
sudo docker compose exec heimnull api reset-password <username>

It prints a new random password once, re-enables the account, ends its sessions and records it in the audit log.

Server

The disk is filling up

System → Storage shows the disk use per table and how many days are left. Raw flows fill it first: shorten Raw flows under Settings → Retention. The change applies at once: older data is deleted. Rollups, attacks and peaks keep their own retention. See Storage and retention.

“The compose file is older than this image needs”

docker compose pull updates the image, not docker-compose.yml. Fetch the file again and apply it:

cd /opt/heimnull
sudo curl -fsS -o docker-compose.yml https://registry.heimnull.com/docker-compose.yml
sudo docker compose up -d --wait

“ClickHouse runs an outdated configuration”

sudo docker compose up -d --force-recreate clickhouse

“The license server hasn’t been reachable”

Allow outbound HTTPS (TCP 443) to license.heimnull.com, then press Check now on Configure → License. The license keeps working for 30 days after the last good check-in. See Licensing.