CSF, lfd, and Imunify360: why your firewall is killing itself

The page came in at 03:14. A cPanel node on cpanel-host had stopped accepting new connections to wp-login on three sites, then started accepting them again, then stopped. The firewall was alive. The firewall was dead. The firewall was alive again. systemctl status lfd returned the line every cPanel operator eventually meets:

lfd.service: Main process exited, code=killed, status=9/KILL

If you have arrived at this post by Googling lfd killed signal 9, you are almost certainly running CSF and Imunify360 on the same machine, your csf.deny is far larger than it should be, and the on-call engineer who set this up did not anticipate what happens when two firewalls both believe they own the iptables INPUT chain.

This is the postmortem for that scenario. Five real symptoms drawn from our incident log, with the diagnostic commands we ran on the box, the config changes that closed the ticket, and an honest description of what ServerGuard's use case can and cannot do for it today.

The 3am symptom that always means the same thing

The first thing you see is not an alert. It is the absence of one. lfd is what fires the brute-force and login-failure notifications on a CSF-managed cPanel server. When lfd itself is the process that died, the alert pipeline goes with it. The Telegram bot stays silent. The email never arrives. The first signal is usually a customer ticket, or a monitoring probe that notices the SSH banner is taking seven seconds instead of two.

journalctl is the second stop:

journalctl -u lfd -n 200 --no-pager

May 10 03:14:02 cpanel-host systemd[1]: Started ConfigServer Firewall & Security - lfd.
May 10 03:14:11 cpanel-host lfd[28814]: daemon started on PID 28814
May 10 03:18:42 cpanel-host systemd[1]: lfd.service: Main process exited, code=killed, status=9/KILL
May 10 03:18:42 cpanel-host systemd[1]: lfd.service: Failed with result 'signal'.
May 10 03:18:42 cpanel-host systemd[1]: lfd.service: Consumed 4.812s CPU time.
May 10 03:18:43 cpanel-host systemd[1]: lfd.service: Scheduled restart job, restart counter is at 7.
May 10 03:18:48 cpanel-host systemd[1]: Started ConfigServer Firewall & Security - lfd.
May 10 03:23:21 cpanel-host systemd[1]: lfd.service: Main process exited, code=killed, status=9/KILL

Three things in this log matter. First, the service is being killed every four to five minutes. Second, restart counter is at 7 means this has been going on for at least half an hour and systemd is about to give up. Third, code=killed status=9/KILL is not the kernel OOM killer. dmesg would say so if it were. This is systemd enforcing a cgroup memory limit on lfd.service itself.

dmesg -T | grep -i 'killed process'

(no output)

That empty output is the diagnostic. If dmesg had an OOM line, the fix would be sysctl-level. It does not. The kill is coming from inside the unit.

The two firewall layers most cPanel servers run

A modern cPanel server typically has two security layers stacked on top of each other, and both of them write iptables rules.

CSF (ConfigServer Security & Firewall) is the user-facing iptables manager that hosting agencies have used since 2007. It owns /etc/csf/csf.conf, the csf.allow and csf.deny flat files, and a companion daemon called lfd (Login Failure Daemon) that watches /var/log/secure, /var/log/exim_mainlog, and the cPanel access logs for brute-force patterns. When lfd decides an IP is hostile, it calls csf -d and a new entry appears in csf.deny.

Imunify360 is the cPanel-blessed WAF and proactive defence layer. CloudLinux ships it as part of Imunify Suite; it provides modsec rules, malware scanning, the proactive-defence PHP wrapper, and an active firewall component that maintains its own deny list and writes its own iptables rules. It is the recommended security product in cPanel's own marketplace.

Both of them touch the iptables INPUT chain. CSF inserts into chains named LOCALINPUT and DENYIN. Imunify360 inserts into chains named f2b-imunify360-default and friends. In the absence of conflict each one stays in its own lane. In the presence of conflict (which is every busy server we have ever run) the lanes overlap and you get the symptoms below.

Why running both is the default and also the problem

cPanel's documentation describes CSF and Imunify360 as integrated. That is true at the marketing level. They both honour the same allow list. They both expose UI in WHM. They both call the same iptables binary.

What "integrated" does not mean is "coordinated." There is no shared lock, no shared deny list of record, and no shared sync schedule. CSF runs csf --denyf on its own timer to flush expired entries. Imunify360 runs imunify360-agent firewall sync on its own timer to reconcile its database with the live ruleset. When those timers overlap, both processes call iptables simultaneously. Neither tool acquires the iptables xtables lock with --wait consistently. CSF will, Imunify360 will sometimes, and the result is rules that get clobbered, duplicated, or written in the wrong order.

We have seen this on every cPanel node we run that has both layers turned on. The first symptom is usually subtle: a deploy SSH session gets dropped for ten seconds and then comes back. The second symptom is the same as the first, but happens often enough that someone files a ticket. The third symptom is lfd dying.

The rest of this post is about the four symptoms we have isolated and the configuration changes that resolve each one.

Symptom 1: lfd OOM-killed by systemd

The lfd daemon, on startup and every reload, parses csf.deny into memory to build its in-process IP-match structures. On a clean install csf.deny is a few hundred lines and the parse takes microseconds. Six weeks into a busy server's life, csf.deny can look like this:

wc -l /etc/csf/csf.deny && du -h /etc/csf/csf.deny

89381 /etc/csf/csf.deny
12M	/etc/csf/csf.deny

89,381 lines. Twelve megabytes. That is a real number from incident 015 in our log, on a cPanel node that had been live for forty-three days. The growth came from Imunify360's deny list being mirrored into csf.deny on every CSF sync, and from CSF's own duplicate-checking logic, which is O(n) over the file and falls over at this scale.

The parse is not memory-efficient. Watch it run under systemd-cgtop during an lfd restart on this file:

systemd-cgtop --depth=2 -n 5 | grep lfd

lfd.service                                   1   97.4    412.0M
lfd.service                                   1   98.1    537.8M
lfd.service                                   1   98.6    694.3M
lfd.service                                   1   98.2    812.1M

Eight hundred megabytes of resident memory just to parse a deny list. On many cPanel images, lfd.service ships with a MemoryMax=512M cgroup limit, or a MemoryHigh=384M soft ceiling. Once peak parse memory crosses the limit, systemd sends SIGKILL. The service restarts. The parse runs again. The service dies again.

The quick fix is a systemd drop-in that lifts the ceiling enough for the current csf.deny size, with headroom:

sudo systemctl edit lfd.service

[Service]
# Override systemd's default cgroup limits on lfd. The parse cost on
# a 12MB csf.deny peaks around 850MB; 1.5G gives a 70% safety margin.
# Re-evaluate this number whenever csf.deny changes order of magnitude.
MemoryHigh=1G
MemoryMax=1500M

sudo systemctl daemon-reload
sudo systemctl restart lfd
sudo systemctl status lfd

● lfd.service - ConfigServer Firewall & Security - lfd
     Loaded: loaded (/usr/lib/systemd/system/lfd.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/lfd.service.d
             └─override.conf
     Active: active (running) since Sat 2026-05-10 03:42:18 UTC; 12s ago
   Main PID: 31204 (lfd - sleeping)
      Tasks: 2 (limit: 18914)
     Memory: 832.0M (high: 1.0G max: 1.5G)
        CPU: 1.281s

The daemon stays up. But the underlying csf.deny is still twelve megabytes, the parse is still slow, and you have not actually fixed the problem. You have made it tolerable. The real fix is symptom 2.

Symptom 2: csf.deny growing without bound

CSF was designed in an era where csf.deny had hundreds of entries. It maintains the file with two configurable ceilings in csf.conf:

grep -E '^DENY_(IP|TEMP_IP)_LIMIT' /etc/csf/csf.conf

DENY_IP_LIMIT = "200"
DENY_TEMP_IP_LIMIT = "100"

Those are the defaults on a clean install. Most production servers never see them because nobody touches csf.conf after the initial setup wizard, and the defaults are set high enough not to alarm you. What is missing is enforcement when an external process (Imunify360) appends to csf.deny directly through csf -d calls.

Look at the file content near the bottom:

tail -20 /etc/csf/csf.deny

204.194.51.195 # lfd: (PERMBLOCK) 204.194.51.195 has had more than 4 temp blocks - Mon May 5 21:40:11 2026
172.191.49.89 # lfd: (XMLRPC) Failed XMLRPC login from 172.191.49.89 - Mon May 5 22:11:02 2026
91.137.27.140 # lfd: (PERMBLOCK) 91.137.27.140 has had more than 4 temp blocks - Tue May 6 04:18:55 2026
87.121.84.114 # lfd: (SSHD) Failed SSH login from 87.121.84.114 - Tue May 6 09:33:21 2026
130.12.181.92 # lfd: (SSHD) Failed SSH login from 130.12.181.92 - Tue May 6 09:33:22 2026
# Imunify360 ...
# Imunify360 ...
# Imunify360 ...
# ... 88,000 more lines ...

The first five lines are normal lfd-driven blocks against attackers who have hammered every cPanel node on the public internet for years. The lines below are Imunify360 mirroring its own deny list into csf.deny because someone enabled the IMUNIFY360_INTEGRATION flag and never tuned the rotation. Each mirrored line is a duplicate of an iptables rule Imunify360 has already inserted. CSF then reads csf.deny, looks at each entry, asks iptables whether it already exists, and only adds it if it does not. That dedup pass is O(n) against the live ruleset and O(n) against the file: with 89,381 lines and a comparable iptables ruleset, it takes minutes.

Two changes shrink the file and stop it growing. First, drop the limits to something that actually fits the parse budget:

# /etc/csf/csf.conf: edited values only
DENY_IP_LIMIT = "5000"
DENY_TEMP_IP_LIMIT = "2000"

5,000 permanent denies and 2,000 temporary ones is generous for an agency-scale cPanel server. CSF will rotate older entries out as new ones come in, parse times stay under a second, and lfd stops flapping.

Second, turn off the bidirectional mirror with Imunify360. In /etc/csf/csf.conf set LF_TRIGGER and IMUNIFY360_INTEGRATION to the values that match the architecture you actually want. We will discuss that further down. For most servers the right answer is to let Imunify360 own its own deny list and let CSF own only what lfd generates locally.

After both changes, csf -r rebuilds the ruleset:

sudo csf -r

DROP  all opt -- in !lo out *  0.0.0.0/0  -> 0.0.0.0/0
csf: arrays loaded
csf: rules loaded
csf: 5000 entries from csf.deny
csf: 0 IPv6 entries from csf.deny
csf: complete

Five thousand entries. Twelve megabytes is now a few hundred kilobytes. The next lfd restart parses in milliseconds.

Symptom 3: silent failure of csf -df

This one bit us during incident 006 and is worth its own section because it is invisible until you look closely. csf -df removes an IP from csf.deny and from the live iptables ruleset. It returns exit code 0 on success. It also returns exit code 0 when handed a whitespace-padded IP that does not match any entry, and most copy-and- paste workflows produce whitespace-padded IPs.

Watch what happens here:

# This is what an operator typically runs after copying an IP from a
# log line, where the copy includes a trailing newline or a space.
csf -df " 203.0.113.42"

csf: no matching IP found

echo $?

0

Exit code zero. No matching IP. The IP is still blocked. The operator moves on. Twenty minutes later the customer calls back and says they still cannot reach their site. The IP was never actually removed because csf -df parsed " 203.0.113.42" (with the leading space) as a literal needle and did not find it in csf.deny (which stores it without the space).

The fix is to trim before passing. The shortest one-liner that does this safely is:

csf -df "$(echo " 203.0.113.42" | xargs)"

Removing rule...
DROP       all  --  203.0.113.42         anywhere
csf: 203.0.113.42 removed from /etc/csf/csf.deny and iptables

xargs with no arguments collapses whitespace and drops trailing newlines. The IP gets passed clean. The remove succeeds. The exit code still says 0, and this time it actually means it.

We wrap this in a tiny shell function on every cPanel server we run:

# /root/.bashrc or /etc/profile.d/csf-helpers.sh
csf-deny-remove() {
  local ip
  ip="$(echo "$1" | xargs)"
  [[ -z "$ip" ]] && { echo "usage: csf-deny-remove <ip>"; return 1; }
  csf -df "$ip"
}

csf-deny-remove " 203.0.113.42 " then does the right thing without ceremony.

Symptom 4: race conditions on iptables INPUT

This is the symptom that took the longest to diagnose because it is intermittent by design. Imunify360 reloads its firewall ruleset on a five-minute timer. CSF runs its deny-sync logic on a separate timer. Roughly every fifteen minutes the two timers align. When they do, both processes call iptables-restore against the same chains at the same time.

iptables has a lock file (/run/xtables.lock) and a --wait flag that tells callers to block until the lock is free. If both callers respect the lock, the writes serialise and nothing breaks. If only one of them respects it (and Imunify360's wrapper calls iptables without --wait in some code paths) the second writer's rules silently overwrite the first's.

The way we caught it was a tcpdump on the admin IP during a window where SSH connectivity from 198.51.100.10 was dropping for three to five seconds at a time:

sudo tcpdump -nni any 'host 198.51.100.10 and tcp port 22' -tttt

2026-05-10 04:00:02.114 IP 198.51.100.10.51220 > cpanel-host.22: Flags [S], seq 1
2026-05-10 04:00:05.221 IP 198.51.100.10.51220 > cpanel-host.22: Flags [S], seq 1
2026-05-10 04:00:11.443 IP 198.51.100.10.51220 > cpanel-host.22: Flags [S], seq 1
2026-05-10 04:00:14.612 IP 198.51.100.10.51220 > cpanel-host.22 Flags [S.], seq 0, ack 1

Three SYN retransmits, twelve seconds of dead air, then the connection gets through. Repeat at 04:05:00, 04:10:00, 04:15:00. The pattern is exactly the Imunify360 timer cadence. During each window the rule that allows our admin IP got temporarily clobbered while iptables was mid-restore.

The fix that closed the ticket was twofold. First, stagger the schedules so they cannot collide:

# /etc/systemd/system/csf-deny-sync.timer.d/stagger.conf
[Timer]
# Imunify360 reloads on the :00, :05, :10, etc. mark.
# CSF deny-sync runs on the :02, :07, :12 mark. Never overlapping.
OnCalendar=*:02,07,12,17,22,27,32,37,42,47,52,57

Second, make sure lfd itself uses iptables -w for every write. CSF 6.x respects this through IPTABLES_OPT in csf.conf:

# /etc/csf/csf.conf
# Force --wait on every iptables call. This is the single most
# important config line on a server that runs CSF alongside any
# other firewall manager.
IPTABLES_OPT = "--wait 30"

After csf -r and systemctl restart lfd, the next 24-hour tcpdump on the admin IP showed zero retransmits.

The fix we ship now

When the on-call queue surfaces this scenario today (lfd flapping, csf.deny over a megabyte, intermittent admin-IP drops) we apply three changes in order.

The first is the lfd cgroup ceiling. Without it, the daemon will not stay up long enough to apply the next two changes.

# /etc/systemd/system/lfd.service.d/override.conf
[Service]
MemoryHigh=1G
MemoryMax=1500M

The second is the deny-list cap. Without it, csf.deny will grow again as Imunify360 keeps mirroring blocks.

# /etc/csf/csf.conf: edited values
DENY_IP_LIMIT = "5000"
DENY_TEMP_IP_LIMIT = "2000"
IPTABLES_OPT = "--wait 30"

The third is the timer stagger. Without it, the iptables races come back during the next high-traffic window.

# /etc/systemd/system/csf-deny-sync.timer.d/stagger.conf
[Timer]
OnCalendar=*:02,07,12,17,22,27,32,37,42,47,52,57

csf -r to rebuild the ruleset, systemctl daemon-reload && systemctl restart lfd to apply the new cgroup limit, and the next day's tcpdump should be quiet.

Why we don't recommend "just disable Imunify360"

Half of the forum answers on this topic say "uninstall Imunify360 and the problem goes away." That is technically correct and operationally wrong. Imunify360's WAF rules catch CVE-class WordPress plugin exploits within hours of disclosure. Its proactive-defence layer catches PHP backdoors that CSF cannot see. CSF operates at the network layer; Imunify360 operates inside the PHP runtime. Disabling it is a regression for any server hosting WordPress or other PHP-heavy applications.

The other half of the forum answers say "uninstall CSF and let Imunify360 own the firewall." That is closer to right and is the direction we recommend on new builds. But on existing servers, CSF's lfd is wired into operator muscle memory: every dashboard, every runbook, every Slack alert assumes CSF is the source of truth for IP blocks. Removing it is a multi-week project for any agency that has been on cPanel for more than a year.

The honest middle path is to keep both, but configure them so they do not collide. The three changes above do that. The next section is about the longer-term re-architecture for teams that want to do it properly.

The slow path: clean re-architecture

Once a server is stable on the three quick fixes, the next step is to move toward an architecture where each tool owns the chain it is best at. The shape of that architecture, in our experience, is:

• Imunify360 owns iptables INPUT for IP-level blocks. It has the better feed (CloudLinux's network of customer servers reports attackers in real time) and the better dedup logic.
• CSF runs in LF_TRIGGER mode for lfd only. The CSF iptables manager is disabled (TESTING = "1" is not the right knob; LF_DAEMON = "1" with TCP_IN/TCP_OUT set permissively is). The daemon still watches logs, still detects brute force, but instead of writing iptables rules directly, it calls a hook that asks Imunify360 to add the block.
• CSF's strengths (Connection Tracking, port-scan detection, suspicious process detection, the integrity checker) stay on. These do not touch iptables.

This is dangerous enough that it requires a maintenance window, a rollback plan, and a tested hook script. We do not recommend doing it under fire at 03:14. The three quick fixes buy you the time to plan this properly during a routine maintenance window.

If you are the operator coming back to this post six weeks after the quick fixes (lfd is stable, csf.deny is bounded, the iptables races are gone, and you are ready to clean it up) the broader topic of cPanel firewall lockouts you can cause yourself covers the things to test before flipping CSF into log-only mode. The SSH brute force monitoring use case covers the lfd triggers that need to keep working under the new architecture.

How ServerGuard handles this

ServerGuard's use case for this scenario sits at the overlap of brute-force defence and web-server health. What it does today:

• Detects lfd flapping by polling the systemd unit state every 30 seconds and correlating against journalctl for code=killed status=9/KILL lines. Once it has seen three kills in fifteen minutes, it raises an incident.
• Reads csf.deny size, current iptables rule count, and the systemd timer schedule for imunify360-agent.timer and any CSF deny-sync timer. It correlates the kill timestamps against the timer windows.
• Applies the lfd cgroup ceiling autonomously as a Safe action. The override drop-in is idempotent and trivially reversible, so it does not require human approval.
• Applies the timer stagger autonomously as a Safe action, for the same reason.

What it does not do today:

• It does not change DENY_IP_LIMIT or DENY_TEMP_IP_LIMIT in csf.conf automatically. Lowering those values from 200 (default) to 5000 is uncontroversial; lowering from a custom-set higher value to 5000 could lose deny entries the operator deliberately kept. It surfaces the change as a Moderate action with a one-click approval flow on Telegram or the dashboard.
• It does not perform the clean re-architecture (CSF read-only mode, Imunify360 as sole iptables owner). That is upcoming and will land as a Dangerous action with a multi-step approval and a tested rollback. We do not ship dangerous use cases without confidence in the rollback.

The honest summary: the quick fix is fully automated. The cleanup is guided but human-approved. The re-architecture is roadmap.

A 5-minute audit you can run right now

If you want to know whether your server has this problem brewing before lfd starts flapping at 03:14, the eight commands below give you the answer:

# 1. Is lfd healthy?
systemctl is-active lfd && systemctl status lfd --no-pager | head -10
 
# 2. Has it been killed recently?
journalctl -u lfd --since "24 hours ago" | grep -c 'status=9/KILL'
 
# 3. How big is csf.deny?
wc -l /etc/csf/csf.deny && du -h /etc/csf/csf.deny
 
# 4. Are the deny limits sane?
grep -E '^DENY_(IP|TEMP_IP)_LIMIT' /etc/csf/csf.conf
 
# 5. Is iptables --wait being used?
grep -E '^IPTABLES_OPT' /etc/csf/csf.conf
 
# 6. Is Imunify360 active and running its own firewall?
imunify360-agent version && imunify360-agent firewall status
 
# 7. Do CSF and Imunify360 timers overlap?
systemctl list-timers --all | grep -E 'imunify|csf'
 
# 8. What does the live ruleset look like?
iptables -L INPUT -n --line-numbers | wc -l

A healthy server returns: lfd active, zero kills in 24h, csf.deny under 1MB, deny limits set explicitly (not default), IPTABLES_OPT contains --wait, Imunify360 active, timers offset by at least two minutes, iptables INPUT chain under 6,000 rules. If any of those fail, the three-change fix in this post applies.

For the full annotated reference of every command, including the output you should expect on a healthy versus unhealthy server, see the csf and lfd command reference and the Imunify360 firewall integration reference.

Closing the loop

If you got here from lfd killed signal 9 or csf.deny too large, the fix is the three-change patch above and the audit. If your server is past that point (multi-megabyte csf.deny, multiple lfd kills per hour, intermittent connectivity drops correlating to timer windows) the fix still works, but the cleanup is going to take a maintenance window.

ServerGuard watches for exactly this pattern on every cPanel node we monitor. The detection runs every 30 seconds; the safe actions apply autonomously; the moderate actions wait for a one-click approval. If you would like the audit script from this post packaged as a single shell file, join the waitlist and we will send it in your welcome email along with a sample SGuard incident report from a real cPanel server.

The next operator who Googles lfd killed signal 9 at 03:14 should not have to read fifteen forum threads to find the right answer. We hope this post is now the one they find.