Now that I’m monitoring my logs using cetralised RSyslog, I regularly notice SSH attacks right when and as they happen. When it becomes obvious that someone’s trying to brute-force SSH, I don’t always wait to let fail2ban fix the issue – sometimes I ban the offending IP myself.
How To Ban Specific IP with fail2ban
Assuming a standard install, we’ll use the fail2ban-client command to notify sshd jail module to ban a specific IP.
Here’s how it works:
root@s1:/etc/fail2ban # fail2ban-client -vvv set sshd banip 202.70.66.228
30 7F0B121F6640 fail2ban.configreader INFO Loading configs for fail2ban under /etc/fail2ban
30 7F0B121F6640 fail2ban.configreader DEBUG Reading configs for fail2ban under /etc/fail2ban
31 7F0B121F6640 fail2ban.configreader DEBUG Reading config files: /etc/fail2ban/fail2ban.conf
31 7F0B121F6640 fail2ban.configparserinc INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
31 7F0B121F6640 fail2ban.configparserinc TRACE Reading file: /etc/fail2ban/fail2ban.conf
31 7F0B121F6640 fail2ban.configparserinc INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
31 7F0B121F6640 fail2ban.configparserinc TRACE Shared file: /etc/fail2ban/fail2ban.conf
32 7F0B121F6640 fail2ban INFO Using socket file /var/run/fail2ban/fail2ban.sock
32 7F0B121F6640 fail2ban INFO Using pid file /var/run/fail2ban/fail2ban.pid, [INFO] logging to SYSLOG
32 7F0B121F6640 fail2ban HEAVY CMD: ['set', 'sshd', 'banip', '202.70.66.228']
48 7F0B121F6640 fail2ban HEAVY OK : 1
48 7F0B121F6640 fail2ban.beautifier HEAVY Beautify 1 with ['set', 'sshd', 'banip', '202.70.66.228']
1
48 7F0B121F6640 fail2ban DEBUG Exit with code 0
Once you become comfortable, you can omit the -vvv option and skip all this verbose output:
root@s1:/etc/fail2ban # fail2ban-client set sshd banip 202.70.66.229
1
I have a number of servers, including a few on the home office network, that accept SSH connections. Even though they are serving on different (non-standard) SSH ports, there are regular attempts made to break it via brute-force – I can see how some random IP addresses start trying to log in using different standard user names. It’s therefore never too late to use additional software for protecting SSH service, something like fail2ban.
What is fail2ban?
fail2ban is a tool that monitors OS logs, identifies failed connection and authentication (login) attempts and then temporarily bans these IP addresses using IPtables.
The idea is that any IP address that failed to login multiple times within a period of time must be blocked from further attempts to log in on a firewall level. This minimises risks because connections are simply blocked rather than allowed to try another username/password combination.
INTERESTING: fail2ban can do a lot more than just protect your SSH service – it hasa growing library of contextual log files knowledge.
Install fail2ban in Ubuntu
Even on my Raspberry system I can just do this to install fail2ban:
$ sudo apt install fail2ban
IMPORTANT: double-check that you have iptables installed – think it would be installed as part of dependencies for fail2ban.
Once installed, this software needs to be activated – so you need to start it using systemctl or service command.
Configure fail2ban
Before we can start, it makes sense to customise fail2ban to make sure it’s going to work properly.
Basic settings I focus on are:
SSH port – by default fail2ban will keep blocking standard SSH port 22, which isn’t going to be all that helpful if your SSH service is listening on a different TCP port
Configure email – fail2ban will notify you of new bans/unbans
So just edit the /etc/fail2ban/jail.conf file as root. I made the following changes:
Email settings for fail2banSpecifying custom port 202 for my SSH service
How to Use fail2ban
Start the service:
$ sudosystemctl start fail2ban
and check its log file:
2020-01-09 22:32:55,710 fail2ban.server [6038]: INFO -------------------------------------------------- 2020-01-09 22:32:55,712 fail2ban.server [6038]: INFO Starting Fail2ban v0.10.2
2020-01-09 22:32:55,727 fail2ban.database [6038]: INFO Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3'
2020-01-09 22:32:55,731 fail2ban.jail [6038]: INFO Creating new jail 'sshd'
2020-01-09 22:32:55,779 fail2ban.jail [6038]: INFO Jail 'sshd' uses pyinotify {}
2020-01-09 22:32:55,798 fail2ban.jail [6038]: INFO Initiated 'pyinotify' backend
2020-01-09 22:32:55,801 fail2ban.filter [6038]: INFO maxLines: 1
2020-01-09 22:32:55,934 fail2ban.server [6038]: INFO Jail sshd is not a JournalFilter instance
2020-01-09 22:32:55,936 fail2ban.filter [6038]: INFO Added logfile: '/var/log/auth.log' (pos = 385669, hash = 9d2089e21756515d4394ead79bad08c298835101)
2020-01-09 22:32:55,939 fail2ban.filter [6038]: INFO encoding: UTF-8
2020-01-09 22:32:55,940 fail2ban.filter [6038]: INFO maxRetry: 3
2020-01-09 22:32:55,942 fail2ban.filter [6038]: INFO findtime: 600
2020-01-09 22:32:55,943 fail2ban.actions [6038]: INFO banTime: 1800
2020-01-09 22:32:55,974 fail2ban.jail [6038]: INFO Jail 'sshd' started
2020-01-10 02:46:49,790 fail2ban.filter [6038]: INFO [sshd] Found 218.93.239.44 - 2020-01-10 02:46:49
2020-01-10 02:46:49,825 fail2ban.filter [6038]: INFO [sshd] Found 218.93.239.44 - 2020-01-10 02:46:49
2020-01-10 02:46:51,811 fail2ban.filter [6038]: INFO [sshd] Found 218.93.239.44 - 2020-01-10 02:46:51
2020-01-10 02:46:52,382 fail2ban.actions [6038]: NOTICE [sshd] Ban 218.93.239.44
How To Inspect fail2ban Logs
As you can see from the output, the service created a “jail” for SSHd service and started looking at failed SSH login attempts. I started fail2ban at 22:32 last night, and at 2:46am got the first IP address blocked: it found 3 failed logins from 218.93.239.44 and banned it immediately.
You can also check iptables, they might have some IP addresses blocked already:
root@srv:/# iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
266 17432 f2b-sshd tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 202
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain f2b-sshd (1 references)
pkts bytes target prot opt in out source destination
0 0 REJECT all -- * * 218.93.239.44 0.0.0.0/0 reject-with icmp-port-unreachable
266 17432 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
That’s it for one day. Hope you’ve learned something new today!
This is not the most obvious functionality, hence I decided to share it as a separate post. It’s quite easy and perfectly acceptable to specify more than one SSH port for your sshd daemon – useful for debugging or added security (when bound to separate IP addresses).
Adding Extra SSH ports
Simply edit the /etc/ssh/sshd_config file and add more port numbers under the existing default port (it’s commented out because 22 is used by default):
If we want to, we can even try connecting to a non-standard ssh port like 221 or 222 as per our changes.
Don’t be alarmed about warning:
root@server:~# ssh greys@localhost -p 222 The authenticity of host '[localhost]:222 ([127.0.0.1]:222)' can't be established. ECDSA key fingerprint is SHA256:12efZx1MOEmlxQOWKhM5eaxDwJr4vUlLhcpElkGHTow. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '[localhost]:222' (ECDSA) to the list of known hosts. greys@localhost's password: Welcome to Ubuntu 19.04 (GNU/Linux 4.18.0-20-generic x86_64)
Here I was trying to ssh from my XPS laptop to MacBook Pro for some quick command, when SSH started giving me the too many authentication failures error. I decided to capture findings here as a blog post.
Too Many Authentication Failures
Here’s how the error looked from my Ubuntu 19.04 command line:
greys@xps:~ $ ssh greys@maverick
Received disconnect from 192.168.1.200 port 22:2: Too many authentication failures
Disconnected from 192.168.1.200 port 22
The weird thing is that this was happening without any passwords asked, so at first it seemed really strange: you get authentication failures but you actually haven’t tried authenticating at all.
Why Too Many Authentication Failures Occur
So yes, these errors happen when you attempt to log in using some credentials and you are denied access for a few times in a row due to incorrect credentials.
Something as fundamental as SSH client and server are rarely wrong in such basic things. So thinking about the error a bit more (and Googling around, of course) I realised that authentication attempts were made using SSH keys I have configured on my Ubuntu laptop. There’s quite a few and SSH client was offering them one after another to the MacBook’s SSH daemon in attempts to log me in.
So I never got asked for a password because my SSH client already offered a few SSH keys and remote SSH server counted each offering as an authentication attempt. So when this maxed out the SSH server limit, I got the error.
MaxAuthTries Setting
Related to the error above is this MaxAuthTries setting in /etc/ssh/sshd_config file.
This option is set to something fairly reasonable usually, in MacOS Mojave it’s set to 6 by default. But because I changed it to 3 in the past for better security, it limited my access when my SSH client was offering more than 3 SSH keys to log in.
Working around the Too Many Authentication Attemps Problem
There’s a number of approaches, all of them to do with SSH identities used for remote access. They are managed by SSH agent, a special software usually starting automatically with your laptop login that tracks all the usernames and SSH keys you have to try them when accessing things remotely.
Disable SSH agent temporarily
So the easiest fix is to disable SSH agent temporarily and try login again (for password logins it does the trick).
I’ll quickly show the steps but will need to write a separate proper post on using SSH agent usage soon.
Step 1: we check user variables for SSH_AUTH_SOCK
This variable will usually confirm if you have SSH Agent. If this variable exists and points to the valid file, that’s the Unix socket used by your SSH agent:
That’s it, now logins to Macbook laptop should work again:
greys@xps:~ $ ssh greys@maverick
Password:
Last login: Wed Jun 12 12:31:33 2019 from 192.168.1.60
greys@maverick:~ $
That’s it for today! Quite a few advanced topics in just one post, so I’ll be sure to revisit it and expand with further posts on the concepts of SSH ports, SSH agent, passwordless SSH and generating SSH keys.
