Security : Linux : Floodmon (SYN flood attacks monitoring, alert and mitigation)
Floodmon is a small Perl daemon used to monitor a server and to detect SYN flood attacks, to alert the administrator and to mitigate the attack in order to allow legitimate connections to succeed (HTTP, POP3, SMTP...). It includes a passive and active defense system able to adjust itself according to the attack magnitude.
Floodmon works mainly by tweaking the TCP/IP stack : activation/deactivation, queues size, timeout and retransmissions etc.
All modifications are done in real time in the '/proc' pseudo file system throughout the SYN flood attack. It can also null-route entire subnets (/8, /16 or /24) at a high rate. It has 4 protection levels, each one having its own configuration and can be quite aggressive if needed. It can alert the administrator of a possible attack by email and/or SMS, email detailed reports and even include a capture of the SYN segments for analysis. It can also display some useful network stats.
II - Setup :
The daemon, floodmon, must be installed in '/usr/sbin/' directory, chmoded to 0700 and only accessible to 'root' (chown root:root). Its configuration file 'floodmon.conf' has to be edited and moved to '/etc/'. Two other scripts are also included, one for munin-node and one to start the daemon aat boot time (see 'README' for more informations).
Floodmon creates a log file '/var/log/floodmon.log' and backup all user system configuration in '/var/cache/floodmon/' before doing any modification.
III - Programs & Perl modules needed :
Perl modules :
Time::HiRes : only used for the '--debug' option to display the duration of each operation. It can be downloaded on CPAN website, or :
- Debian : # apt-get install libtime-hires-perl
- RH/CentOS : # yum install perl-Time-HiRes
MIME::QuotedPrint : used to send alerts by email. Usually installed by default on most systems with the 'MIME-Base64' module, otherwise download it on CPAN.
Net::Pcap : used only if DUMP_SYNPACKETS is activated. It allows captures of SYN packets which will be sent with alerts and reports by email. It can be downloaded on CPAN or :
- Debian : # apt-get install libnet-pcap-perl
- RH/CentOS : # yum install perl-Net-Pcap
Programs :
iproute : floodmon uses the ip command from the iproute package to null-route IPs blocks.
- Debian : # apt-get install iproute
- RH/CentOS : # yum install iproute
sendmail : used to send emails. Usually installed by default.
wget : used to send SMS. Usually installed by default.
Floodmon does not need any other command or third party program. It doesn't use iptables or the netstat command either (it gets all connection informations from '/proc/net' and '/proc/sys/net', just like netstat does).
IV - Parameters :
floodmon :
Usage :
--help : display this menu.
--debug : run in debug mode (no daemon).
--daemon : run in daemon mode (default).
--reload : reload configuration file (/etc/floodmon.conf).
--stop : stop and restore user configuration (default).
--stop-save : stop but doesn't restore user configuration
(TCP/IP stack, routing table).
--stats : display network statistics.
--munin-node : daemon statistics to use with munin-node.
--sms-test : test SMS alerts.
--capture <num> : capture <num> SYN packets.
--version : check for floodmon updates
--help : display help menu and version number.
--debug : run the script in debug mode and display all operations in the terminal. However, it does not run in daemon mode and to stop it you must use the [CTRL-C] sequence. That option is interesting if your server is under attack and you want to know what is going on.
--daemon : run in daemon mode. That is the default option.
--reload : if you have to make some change to the configuration file, use this parameter so that the daemon will reload it.
--stop : stop the daemon. Any modification it has done will be deleted and your original configuration will be fully restored. That is the default behavior.
--stop-save : stop the daemon. All modifications done will be kept when quitting (TPC/IP stack, null-routed blocks). To restore your original configuration you will need to use the backup files located in /var/cache/floodmon.
--munin-node* : floodmon stats (alert levels) to use with munin-node. You can use the floodmon-munin file and copy it to your /etc/munin/plugins/ directory. Munin-node 'config' parameter is also handled by floodmon.
--sms-test* : allows to test SMS alerts.
--capture <num>* : create an on-demand capture of the last <num> SYN packets and save it to disk or email it to the admin.
--version : check whether a new version of floodmon is available.
* The last 5 options can be executed regardless of the status of floodmon (active or not).
V - User configuration file :
The configuration file 'floodmon.conf' must be edited and then copied into the '/etc/' directory. It contains some variables that must be configured as soon as floodmon is installed (network, email etc) and others that are already configured by default.
If you make changes to this file while floodmon is running, use the # floodmon --reload command so that the daemon will reload it.
VI - Variables to configure :
INTERFACE : your network interface (ex : 'eth0', 'venet0'...).
EMAIL_ALERT : send an email when the alert reached a specific level. By default, it is level 2.
EMAIL_ALERT_FREQ : emails frequency during an attack. You will not receive more than one email every EMAIL_ALERT_FREQ minutes when your server is under attack. Default value is 30mn.
ADMIN_EMAIL : as the name says, the administrator email where the alert will be sent.
DUMP_SYNPACKETS : floodmon can attach to its email alert a capture of the last SYN packets received. The file uses 'pcap' format and can be open/read with a network analyzer like Wireshark (Windows · OS X · Linux).
SMS_ALERT : send a SMS alert to the admin. You must be using a service (VOIP etc) that allows you to send SMS from a simple URL (HTTP or HTTPS). The whole URL + its parameters should be included.
SMS_ALERT_FREQ : SMS alert frequency (in hours).
FLUSH_FREQ : null-routed IPs flush frequency. You shouldn't keep IPs blocked forever, thus this option will flush the routing table and the firewall every FLUSH_FREQ minutes. By default, every 30mn.
NETMASK : network mask used to nullroute IPs (see NULLROUTE_SUBNET below). Value can only be 8 or 16 or 24. Default value : '16'.
LOG_LEVEL : log file verboseness. Default value : '2'.
NO_CONNTRACK : enable this variable if your server doesn't use the connection tracking table. Floodmon will not modify its values (size, buckets and timout). Default value : '0'.
CHECK_UPDATES : floodmon can automatically check for new releases and inform the admin by email. Default value : '1' (daily check).
VII - Variables configured by default :
Those variables are used to mitigate the attack and have been configured with the best values. They have been tested during both real and simulated attacks and have given some pretty good results. Don't change them unless you know what you are doing !
Important : if you are running floodmon from an openVZ or Virutozzo container, only MAX_SYN, LOOP_DELAY and NULLROUTE_SUBNET variables apply. Such VPS are using a single Linux kernel, therfore it is not possible to modify the TCP/IP stack parameters. Floodmon will run in 'light mode', that is, it will monitor TCP connections, send alerts (including packets capture) and block offending IPs.
MAX_SYN (N == max number of SYN) :
This is the maximum number of SYN_RECV allowed for the current level. When that number is reached or exceeded, floodmon switches to the next alert level. By default, 350 simultaneous SYN are allowed in the 1st level before the alert is triggered. Even if that value may seem a bit low, there shouldn't be any reason to change it, except of course if you receive too many false alerts.
For the second level, the maximum amount of SYN_RECV allowed is 500 and 1,000 for the 3rd level. For the 4th last level, 'MAX_SYN' value just doesn't matter but of course should be kept higher than the value of the previous (3rd) level.
SYN_COOKIES (0 = deactivated, 1 = activated) :
Syncookies (see syncookies.c, sysctl_net_ipv4.c and tcp_ipv4.c) have proved to be very effective during a SYN flood attack. Instead of keeping and filling up the backlog queue (tcp_max_syn_backlog) with the received SYN data, it is encrypted and sent along with the SYN/ACK packet and decrypted/verified upon the receipt of the client ACK.
Floodmon deactivates them and will only activate them when an attack will be detected. It should also be noted that even when enabled, syncookies are not used until the tcp_max_syn_backlog and the application listen queue are getting full. Finally, Linux is an OS that has also implemented some strong features to help to mitigate SYN flood attacks (minisock etc).
MAX_SYN_QUEUE (N == size of SYN backlog) :
The maximum number of queued connection requests which have still not received an acknowledgement from the connecting client (tcp_max_syn_backlog). When the queue is full, any new connection attempt will be dropped by the kernel (unless syncookies are activated). In Linux, its size is hard-coded to a value of 256 (cf request_sock.c) and adjusted at runtime according to the server available RAM (1024 if more than 256MB of RAM).
This value is quite low, hence floodmon adjust it from 10,000 to 200,000 depending on the current alert level.
SOMAXCONN ( N == backlog size) :
Limit of socket listen() backlog, known in userspace as SOMAXCONN.
When an application calls the listen() function, it gives 2 parameters : sockfd (file descriptor that refers to a socket) and backlog (queue length for completely established
sockets waiting to be accepted). If backlog has a higher value than SOMAXCONN, it will be lowered to the value of SOMAXCONN.
During a SYN flood attack, SOMAXCONN is as important as tcp_max_syn_backlog : if there are enough warm entries in syn queue, new connection attempts will be dropped (see tcp_ipv4.c).
In Linux, SOMAXCONN size is hard-coded to a very low value : 128 (see socket.h). Fortunately, its value can be dynamically adjust in '/proc/sys/net/core/somaxconn'. Floodmon set it from 10,000 to 60,000 depending on the current level alert.
Important : you must also at least increase your HTTP server (Apache, Nginx etc) listen backlog. See "VIII - Server setup" chapter below.
MAX_SYNACK_RETRY (N == maximum number of retransmissions - from 0 to max 255) :
The maximum number of times a SYN/ACK segment for a passive TCP connection will be retransmitted (tcp_synack_retries).
Upon receipt of a SYN packet, the server will reply with a SYN/ACK. If the client doesn't reply with a ACK packet, just like in the case of a SYN flood attack, the server will send it again several times. By default in Linux a SYN/ACK is retransmitted 5 times (see tcp.h) : first time 3s after the first one, then 6s after the first retransmission, then 12s after the 2nd one, 24s after the 3rd one and, finally, 48s after the 4th one. That last retransmission will only timeout after another 96s. That is a total of 189 seconds and also the reason why, during an attack, the netstat command will display so many SYN_RECV.
Floodmon will reduce the number of retransmissions to 3 (45s), and will keep lowering it at higher alert levels. In the last one, 4th level, it will not allow any SYN/ACK retransmission. Even if that seems a bit harsh, it is simply identical to the use of syncookies which also prevent any retransmission of a SYN/ACK packet.
Connection tracking table :
This table allows the firewall (netfilter) to keep track of one of more connections according to its state (NEW, ESTABLISHED, RELATED etc). Although very useful, it could overflow during a DDoS attack and any further incoming connection would be dropped by the kernel. In that case, kern.log and dmesg command would display : 'nf_conntrack: table full, dropping xx packet".
At start-up, floodmon will check which, if any, table the server is using (ip_conntrack which handles IPv4 or nf_conntrack which handles both IPv4 and IPv6). If your server doesn't use the connection tracking table, you are advised to set the NO_CONNTRACK variable in floodmon.conf.
Floodmon handles 3 variables related to that table :
CONNTRACK_TIMEOUT (N == time-out) :
Timeout value of the SYN and SYN/ACK segments in the connection tracking table. By default, values are 60 and 120s (see nf_conntrack_proto_tcp.c).
Floodmon handles those timeouts as well in order to avoid the table to quickly get full during an attack. It will simply lower them according to MAX_SYNACK_RETRY timeout value.
CONNTRACK_MAX (N == number of entries) :
Maximum number of allowed conntrack entries, also known as ip_conntrack_max or nf_conntrack_max that the kernel can handle simultaneously. By default its value is 65536 (cf nf_conntrack_core.c) and must not be less than 128.
CONNTRACK_HASHSIZE (N == hash table size) :
Also known as buckets, is the size of the hash table that handles the list of conntrack entries. On systems with more than 1Gb of RAM, its default size is set to CONNTRACK_MAX/4 for kernels using nf_conntrack and CONNTRACK_MAX/8 for those using ip_conntrack, that is, 16384 and 8192 respectively.
Note that increasing the size of CONNTRACK_HASHSIZE and CONNTRACK_MAX will also enlarge the size of RAM memory needed. At level 4, floodmon give them both a value of 131072 that will use approximately 38Mb of RAM. If you need to increase those values, use preferably a power of two : 65536 (2^16), 131072 (2^17) etc.
LOOP_DELAY (N == seconds/milliseconds) :
Number of seconds/milliseconds floodmon will pause between each check of your system. By default, connections are checked every 15s in the 1st level, 1.5s in the 2nd one, 1s in the 3rd one and 0.5s in the last one.
Note that floodmon retrieves all connection states directly inside '/proc/net' and '/proc/sys/net' without using a third-party program and therefore, even if LOOP_DELAY has a very low value, it uses very little CPU and RAM (approx 2MB in standby).
NULLROUTE_SUBNET (0 == deactived, N == max number of IPs) :
Floodmon doesn't use iptables to block IPs. It uses the routing table to null-route them, that is to redirect them to a blackhole. Even if the routing table is not intended to replace the firewall, Floodmon uses it because it is faster and uses much less system resources (almost none) than a software firewall during the attack and, most important, when a IP is null-routed, whether it is spoofed or not, the kernel will stop any further retransmission of SYN/ACK segments hence it will get rid of the half-open connections (SYN_RECV) quicker that way as they will be removed from the tcp_max_syn_backlog at the end of the current retransmission.
IPs are not blocked individually, but by entire subnets using /8, /16 or /24 network mask (see NETMASK above). The default value is 16, which means a whole class B (255.255.0.0 mask, CIDR /16). If there are more than NULLROUTE_SUBNET IPs from the same subnet, the whole class will be null-routed. For example, if you set NETMASK == 16 and NULLROUTE_SUBNET == 200, if there are more than 200 SYN from remote IP addresses 200.10.xx.xx, floodmon will block all IPs from 200.10.0.0 to 200.10.255.255
Null-routed IPs are flushed by default every 10mn (FLUSH_FREQ variable in 'floodmon.conf').
BEST_EFFORT (0 == deactived, 1 == actived):
This variable is used to modify miscellaneous TPC/IP options that could help during an attack. If you want more informations about them, use the command # man tcp .
VIII - Server setup :
When using floodmon, there are some important points you should take into consideration :
Apache :
As mentionned above, any application calling the listen() function must pass two parameters, one being the size of the backlog queue. Apache v2.x* uses a backlog named DEFAULT_LISTENBACKLOG and hard-coded with a value of 511 (see mpm_common.h). However, as we already know, Linux will lower the application backlog value if it is higher than the SOMAXCONN value. So, when running Apache, it will request a backlog value of 511 but it will be lowered to 128 only, the SOMAXCONN default value.
That difference between Apache and Linux default values makes some problems : if you are under a SYN flood attack on port 80, you will notice that Apache will quickly become irresponsive. In fact it is not Apache but the kernel that will reject any new connection attempt because the listen() and SYN backlogs are both linked together when it comes to handle incoming connections. Apache isn't responsible (it will never know about the attack !) because, and that is a SYN flood attack goal, no connection is established. SYN packets will remain in the kernel half-open connection backlog.
Fortunately, DEFAULT_LISTENBACKLOG can be changed by adding the ListenBacklog directive to apache2.conf (or httpd.conf). Since floodmon gives generous values to both SOMAXCONN and tcp_max_syn_backlog (10,000 minimum), you are advised to greatly increase Apache backlog as well, specially when using floodmon if you want to increase your chances to handle an attack on port 80. A minimal value of 5,000 seems to be a comfortable one, but it will depends on your server capacity, the number of visitors and of course, the magnitude of the attack you are facing as well.
You must edit your apache2.conf or httpd.conf and add the value you want to use. For example, with a value of 5,000 :
ListenBacklog 5000
Then modify SOMAXCONN value (only if floodmon is not already running !) :
echo 5000 > /proc/sys/net/core/somaxconn
You must restart Apache after the modification.
* Apache v1.x uses the same variable since version 1.3 but named DEFAULT_LISTEN_BACKLOG and having a value of 512 (see httpd.h). It can also be changed with the ListenBacklog directive as described above.
Nginx :
Nginx backlog can also be changed in the configuration file (inside the listen directive) :
backlog=5000
For more info, see Nginx online help.
Change this value, change SOMAXCONN (only if floodmon is not already running !) and restart Nginx.
Lighttpd :
Lighttpd is hardcoded to 1024 (128 * 8) and cannot be change in its configuration file. You must download the source code and edit network.c :
Change this value, recompile, then modify SOMAXCONN and restart Lighttpd.
If the attack targets another port (POP3 server, FTP, SMTP...) check the server doc to see if you can increase its backlog. Otherwise, you can always change its value in the source code and recompile (just look for a call to the listen() function).
iptables :
Since version 0.9.4, floodmon does not use iptables any longer.
There are many iptables rules on the Internet to help to 'fight' against a SYN flood, some of which you really should avoid, especially if you use floodmon. For example :
iptables -N syn-flood
iptables -A INPUT -p tcp --syn -j syn-flood
iptables -A syn-flood -m limit --limit 5/s -j RETURN
iptables -A syn-flood -j DROP
That rule rates limit any new TCP connection to 5 per seconds. There is no real difference between using that rule and shooting yourself in the foot.
Even if a website receives only few hundreds visitors pers day, such a rule will affect them, slowing down their attempts to get connected, forcing them to retransmit SYN packets several times and, during an attack, those retransmissions will even increase the flood. Also, the SYN rate limit will not only affect the HTTP server but the whole TCP protocol (POP3, SMTP, FTP etc). An attacker could send an as little as 10 SYN/s flood, that's a ridiculous 80bits/s bandwidth, to block any access to the server, including SSH access. In other words, a script-kiddie could 'flood' the firewall only with his iPhone...
If you are using such a rule, you must remove it. It is not compatible at all with floodmon whose purpose is to mitigate the attack to allow legitimate connections to succeed.
If floodmon has been useful to you, specially if you have been under attack, you can participate in its development :
send your feedback with the attack informations () :
- your server configuration (CPU, RAM, bandwidth etc)
- the attack strength (number of Kbits/s or Mbits/s)
- please include floodmon log file (/var/log/floodmon.log)
- your comments and/or any problem encountered
Important : if you are running floodmon from an openVZ or Virutozzo container, only MAX_SYN, LOOP_DELAY and NULLROUTE_SUBNET variables apply. Such VPS are using a single Linux kernel, therfore it is not possible to modify the TCP/IP stack parameters. Floodmon will run in 'light mode', that is, it will monitor TCP connections, send alerts (including packets capture) and block offending IPs.
