Now that I’m monitoring my logs using cetralised RSyslog, I regularly notice SSH attacks right when and as they happen. When it becomes obvious that someone’s trying to brute-force SSH, I don’t always wait to let fail2ban fix the issue – sometimes I ban the offending IP myself.
How To Ban Specific IP with fail2ban
Assuming a standard install, we’ll use the fail2ban-client command to notify sshd jail module to ban a specific IP.
Here’s how it works:
root@s1:/etc/fail2ban # fail2ban-client -vvv set sshd banip 202.70.66.228
30 7F0B121F6640 fail2ban.configreader INFO Loading configs for fail2ban under /etc/fail2ban
30 7F0B121F6640 fail2ban.configreader DEBUG Reading configs for fail2ban under /etc/fail2ban
31 7F0B121F6640 fail2ban.configreader DEBUG Reading config files: /etc/fail2ban/fail2ban.conf
31 7F0B121F6640 fail2ban.configparserinc INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
31 7F0B121F6640 fail2ban.configparserinc TRACE Reading file: /etc/fail2ban/fail2ban.conf
31 7F0B121F6640 fail2ban.configparserinc INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
31 7F0B121F6640 fail2ban.configparserinc TRACE Shared file: /etc/fail2ban/fail2ban.conf
32 7F0B121F6640 fail2ban INFO Using socket file /var/run/fail2ban/fail2ban.sock
32 7F0B121F6640 fail2ban INFO Using pid file /var/run/fail2ban/fail2ban.pid, [INFO] logging to SYSLOG
32 7F0B121F6640 fail2ban HEAVY CMD: ['set', 'sshd', 'banip', '202.70.66.228']
48 7F0B121F6640 fail2ban HEAVY OK : 1
48 7F0B121F6640 fail2ban.beautifier HEAVY Beautify 1 with ['set', 'sshd', 'banip', '202.70.66.228']
1
48 7F0B121F6640 fail2ban DEBUG Exit with code 0
Once you become comfortable, you can omit the -vvv option and skip all this verbose output:
root@s1:/etc/fail2ban # fail2ban-client set sshd banip 202.70.66.229
1
When reinstalling servers with new versions of operating system or simply reprovisioning VMs under the same hostname, you eventually get this Host Key Verification Failed scenario. Should be easy enough to fix, once you’re positive that’s a valid infrastructure change.
Host Key Verification
Host key verification happens when you attempt to access remote server with SSH. Before verifying if you have a user on the remote server and whether your password or SSH key match that remote user, SSH client must do basic sanity checks on the lower level.
Specifically, SSH client checks if you attempted connecting to the remote server before. And whether anything changed since last time (it shouldn’t have).
Server (host) keys must not change during a normal life cycle of a server – they are generated at server/VM build stage (when OpenSSH starts up the first time) and remain the same – it’s the server’s identity.
This means if your SSH client has one keyprint for a particular server, and then suddenly detects it’s a different one – it’s flagged as an issue: at best, you’re looking at the new, legit server replacement with the same hostname. At worst, someone’s trying to intercept your connection and/or pretend to be your server.
Host Key Verification Failed
Here’s how I get this error on my Macbook (s1.unixtutorial.org doesn’t really exist, it’s just a hostname I show here as example):
greys@maverick:~ $ ssh s1.unixtutorial.org Warning: the ECDSA host key for 's1.unixtutorial.org' differs from the key for the IP address '51.159.18.142' Offending key for IP in /Users/greys/.ssh/known_hosts:590 Matching host key in /Users/greys/.ssh/known_hosts:592 Are you sure you want to continue connecting (yes/no)?
At this stage your default answer should always be “no”, followed by inspection of the known_hosts file to confirm what happened and why identity appears to be different.
If you answer no, you’ll get the Host Key Verification Failed error:
greys@maverick:~ $ ssh s1.unixtutorial.org Warning: the ECDSA host key for 's1.unixtutorial.org' differs from the key for the IP address '51.159.18.142' Offending key for IP in /Users/greys/.ssh/known_hosts:590 Matching host key in /Users/greys/.ssh/known_hosts:592 Are you sure you want to continue connecting (yes/no)? no Host key verification failed.
How To Solve Host Key Verification Errors
The output above actually tells you what to do: inspect file known_hosts and look at the lines 590 and 592 specifically. One of them is likely to be obsolete, and if you remove it the issue will go away.
Specifically, if you (like me) just reinstalled the dedicated server or VM with a new OS but kept the original hostname, then the issue is expected (new server definitely generated a new host key), so the solution is indeed to remove old key from the known_hosts file and re-attempt the connection.
First, I edited the /Users/greys/.ssh/known_hosts file and removed the line 590, which looked something like this. We simply need to find the line with given number, or look for the hostname we just tried to ssh into (s1.unixtutorial.org in my case):
We can try reconnecting now, answer yes and connect to the server:
greys@maverick:~ $ ssh s1.unixtutorial.org
The authenticity of host 's1.unixtutorial.org (51.159.xx.yy)' can't be established.
ECDSA key fingerprint is SHA256:tviW39xN2M+4eZOUGi8UFvBZoHKaLaijBA581Nrhjac.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 's1.unixtutorial.org,51.159.xx.yy' (ECDSA) to the list of known hosts.
Activate the web console with: systemctl enable --now cockpit.socket
Last login: Fri Feb 7 21:18:35 2020 from unixtutorial.org
[greys@s1 ~]$
As you can see, the output now makes a lot more sense: our SSH client can’t establish authenticity of the remote server s1.unixtutorial.org – this is because we removed any mention of that server from our known_hosts file in previous step. Answering yes adds info about s1.unixtutorial.org, so any later SSH sessions will work just fine:
greys@maverick:~ $ ssh s1.unixtutorial.org
Activate the web console with: systemctl enable --now cockpit.socket
Last login: Sat Feb 8 18:31:39 2020 from 93.107.36.193
[greys@s1 ~]$
Copying Host Keys to New Server
I should note that in some cases your setup or organisation would require the same host keys to be kept even with server reinstall. In this case, you’ll need to use last know backup of old server to grab SSH host keys from, to re-deploy them onto the new server – I’ll show/explain this in one of the future posts.
I have a number of servers, including a few on the home office network, that accept SSH connections. Even though they are serving on different (non-standard) SSH ports, there are regular attempts made to break it via brute-force – I can see how some random IP addresses start trying to log in using different standard user names. It’s therefore never too late to use additional software for protecting SSH service, something like fail2ban.
What is fail2ban?
fail2ban is a tool that monitors OS logs, identifies failed connection and authentication (login) attempts and then temporarily bans these IP addresses using IPtables.
The idea is that any IP address that failed to login multiple times within a period of time must be blocked from further attempts to log in on a firewall level. This minimises risks because connections are simply blocked rather than allowed to try another username/password combination.
INTERESTING: fail2ban can do a lot more than just protect your SSH service – it hasa growing library of contextual log files knowledge.
Install fail2ban in Ubuntu
Even on my Raspberry system I can just do this to install fail2ban:
$ sudo apt install fail2ban
IMPORTANT: double-check that you have iptables installed – think it would be installed as part of dependencies for fail2ban.
Once installed, this software needs to be activated – so you need to start it using systemctl or service command.
Configure fail2ban
Before we can start, it makes sense to customise fail2ban to make sure it’s going to work properly.
Basic settings I focus on are:
SSH port – by default fail2ban will keep blocking standard SSH port 22, which isn’t going to be all that helpful if your SSH service is listening on a different TCP port
Configure email – fail2ban will notify you of new bans/unbans
So just edit the /etc/fail2ban/jail.conf file as root. I made the following changes:
Email settings for fail2banSpecifying custom port 202 for my SSH service
How to Use fail2ban
Start the service:
$ sudosystemctl start fail2ban
and check its log file:
2020-01-09 22:32:55,710 fail2ban.server [6038]: INFO -------------------------------------------------- 2020-01-09 22:32:55,712 fail2ban.server [6038]: INFO Starting Fail2ban v0.10.2
2020-01-09 22:32:55,727 fail2ban.database [6038]: INFO Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3'
2020-01-09 22:32:55,731 fail2ban.jail [6038]: INFO Creating new jail 'sshd'
2020-01-09 22:32:55,779 fail2ban.jail [6038]: INFO Jail 'sshd' uses pyinotify {}
2020-01-09 22:32:55,798 fail2ban.jail [6038]: INFO Initiated 'pyinotify' backend
2020-01-09 22:32:55,801 fail2ban.filter [6038]: INFO maxLines: 1
2020-01-09 22:32:55,934 fail2ban.server [6038]: INFO Jail sshd is not a JournalFilter instance
2020-01-09 22:32:55,936 fail2ban.filter [6038]: INFO Added logfile: '/var/log/auth.log' (pos = 385669, hash = 9d2089e21756515d4394ead79bad08c298835101)
2020-01-09 22:32:55,939 fail2ban.filter [6038]: INFO encoding: UTF-8
2020-01-09 22:32:55,940 fail2ban.filter [6038]: INFO maxRetry: 3
2020-01-09 22:32:55,942 fail2ban.filter [6038]: INFO findtime: 600
2020-01-09 22:32:55,943 fail2ban.actions [6038]: INFO banTime: 1800
2020-01-09 22:32:55,974 fail2ban.jail [6038]: INFO Jail 'sshd' started
2020-01-10 02:46:49,790 fail2ban.filter [6038]: INFO [sshd] Found 218.93.239.44 - 2020-01-10 02:46:49
2020-01-10 02:46:49,825 fail2ban.filter [6038]: INFO [sshd] Found 218.93.239.44 - 2020-01-10 02:46:49
2020-01-10 02:46:51,811 fail2ban.filter [6038]: INFO [sshd] Found 218.93.239.44 - 2020-01-10 02:46:51
2020-01-10 02:46:52,382 fail2ban.actions [6038]: NOTICE [sshd] Ban 218.93.239.44
How To Inspect fail2ban Logs
As you can see from the output, the service created a “jail” for SSHd service and started looking at failed SSH login attempts. I started fail2ban at 22:32 last night, and at 2:46am got the first IP address blocked: it found 3 failed logins from 218.93.239.44 and banned it immediately.
You can also check iptables, they might have some IP addresses blocked already:
root@srv:/# iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
266 17432 f2b-sshd tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 202
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain f2b-sshd (1 references)
pkts bytes target prot opt in out source destination
0 0 REJECT all -- * * 218.93.239.44 0.0.0.0/0 reject-with icmp-port-unreachable
266 17432 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
That’s it for one day. Hope you’ve learned something new today!
Super quick advice today, but one of them pearls of experience that now and then saves your day. Learn how to check and confirm your recent changes to SSH daemon config file (/etc/ssh/sshd_config) won’t break your remote SSH access.
Why Double-Checking Configs Is A Good Idea
I should probably start a special section of Unix Tutorial someday, just to talk about how and when things can go wrong. These things below would certainly belong to that section.
Why it’s a good idea to check that your new config file is error free:
avoid getting service outage (syntax error means service won’t restart)
prevent service malfunction (if you end up with only partial service functionality)
don’t get yourself locked out of service (or server, in case of broken SSH)
Ubuntu 19.10 was released on October 17th, which means it’s time to upgrade Ubuntu on my Dell XPS laptop. Please note this is NOT a long-term support (LTS) release, so 19.10 will only be supported until July 2020.
Ubuntu 19.10 – Eoan Ermine
Seems like a pretty solid upgrade:
WPA3 support for improved wireless security – need to check if I can upgrade my Ubiquiti equipment to support it
Linux Kernel upgraded to 5.3 – this is where I expect majority of improvements for my laptop
Dark Theme – in a very short space of just a few months I’m now using dark themes on my iPhone, desktop and laptop, plus in as many apps as support it properly
I did the usual sequence to get my laptop upgraded. Except the last step which needed the -d option to recognise that Ubuntu 19.10 is available. All of these commands were run through sudo.
apt update
apt upgrade
apt dist-upgrade
do-release-upgrade -d
Do you use Ubuntu? When are you planning to upgrade?
One of the greatest improvements introduced by the SSH protocol is key-based authentication – meaning your client and SSH server establish validity of your SSH keypair and let you gain remote SSH access without asking for your password.
SSH Authentication with Passwords
By default, SSH server will ask for your password when you’re trying to connect. Unless you specify a username, your SSH client will set it automatically to your username on local (client) system:
Connecting to remote server using SSH
In this example above, I’m running command line (iTerm2) session on my Macbook. greys is my local username, maverick is the hostname on my Macbook. I’m typing ssh command and specifying the server to connect to – with hostname becky.
As you can see, next thing that happens is that I get a password prompt.
How Key Based SSH Access Works
Key-based SSH authentication takes an extra step to setup but then saves you tons of time in the future:
you deploy your public SSH key to remote server (need to type SSH password for possibly the last time)
you start SSH agent to load your private SSH key and to use it for remote connections
you connect to the remote SSH server without typing any passwords – still enjoying the same great benefits like encryption and traffic compression that SSH brings
Deploy Your Public SSH Key to Remote Server
You guessed it right! There’s actually a command for that, it’s called ssh-copy-id. What it does is connect to remote SSH server using username and password that you supply and then edit the .ssh/authorized_keys there to include your public key.
When running ssh-copy-id, you need to specify 2 things at a minimum:
The SSH identity (name of a key you want to deploy)
The SSH server name (where you want to add your key to)
Here’s how it works:
greys@maverick:~ $ ssh-copy-id -i .ssh/id_ed25519 becky
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: ".ssh/id_ed25519.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'becky'"
and check to make sure that only the key(s) you wanted were added.
Next time you attempt to connect, your ssh client will offer remote server a list of SSH identities you have configured on your client. In most recent Unix-like distros you have an SSH agent starting with your graphics login – it’s called GNOME Keyring or or Keychain in MacOS (and there’s plenty of ways to start ssh-agent during startups of sessions like KDE).
The bottom line is that when I try to connect to becky now, my SSH key is offered and, if it’s available (loaded in ssh-agent), I get a passwordless SSH access to remote server:
This is not the most obvious functionality, hence I decided to share it as a separate post. It’s quite easy and perfectly acceptable to specify more than one SSH port for your sshd daemon – useful for debugging or added security (when bound to separate IP addresses).
Adding Extra SSH ports
Simply edit the /etc/ssh/sshd_config file and add more port numbers under the existing default port (it’s commented out because 22 is used by default):
If we want to, we can even try connecting to a non-standard ssh port like 221 or 222 as per our changes.
Don’t be alarmed about warning:
root@server:~# ssh greys@localhost -p 222 The authenticity of host '[localhost]:222 ([127.0.0.1]:222)' can't be established. ECDSA key fingerprint is SHA256:12efZx1MOEmlxQOWKhM5eaxDwJr4vUlLhcpElkGHTow. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '[localhost]:222' (ECDSA) to the list of known hosts. greys@localhost's password: Welcome to Ubuntu 19.04 (GNU/Linux 4.18.0-20-generic x86_64)
Here I was trying to ssh from my XPS laptop to MacBook Pro for some quick command, when SSH started giving me the too many authentication failures error. I decided to capture findings here as a blog post.
Too Many Authentication Failures
Here’s how the error looked from my Ubuntu 19.04 command line:
greys@xps:~ $ ssh greys@maverick
Received disconnect from 192.168.1.200 port 22:2: Too many authentication failures
Disconnected from 192.168.1.200 port 22
The weird thing is that this was happening without any passwords asked, so at first it seemed really strange: you get authentication failures but you actually haven’t tried authenticating at all.
Why Too Many Authentication Failures Occur
So yes, these errors happen when you attempt to log in using some credentials and you are denied access for a few times in a row due to incorrect credentials.
Something as fundamental as SSH client and server are rarely wrong in such basic things. So thinking about the error a bit more (and Googling around, of course) I realised that authentication attempts were made using SSH keys I have configured on my Ubuntu laptop. There’s quite a few and SSH client was offering them one after another to the MacBook’s SSH daemon in attempts to log me in.
So I never got asked for a password because my SSH client already offered a few SSH keys and remote SSH server counted each offering as an authentication attempt. So when this maxed out the SSH server limit, I got the error.
MaxAuthTries Setting
Related to the error above is this MaxAuthTries setting in /etc/ssh/sshd_config file.
This option is set to something fairly reasonable usually, in MacOS Mojave it’s set to 6 by default. But because I changed it to 3 in the past for better security, it limited my access when my SSH client was offering more than 3 SSH keys to log in.
Working around the Too Many Authentication Attemps Problem
There’s a number of approaches, all of them to do with SSH identities used for remote access. They are managed by SSH agent, a special software usually starting automatically with your laptop login that tracks all the usernames and SSH keys you have to try them when accessing things remotely.
Disable SSH agent temporarily
So the easiest fix is to disable SSH agent temporarily and try login again (for password logins it does the trick).
I’ll quickly show the steps but will need to write a separate proper post on using SSH agent usage soon.
Step 1: we check user variables for SSH_AUTH_SOCK
This variable will usually confirm if you have SSH Agent. If this variable exists and points to the valid file, that’s the Unix socket used by your SSH agent:
That’s it, now logins to Macbook laptop should work again:
greys@xps:~ $ ssh greys@maverick
Password:
Last login: Wed Jun 12 12:31:33 2019 from 192.168.1.60
greys@maverick:~ $
That’s it for today! Quite a few advanced topics in just one post, so I’ll be sure to revisit it and expand with further posts on the concepts of SSH ports, SSH agent, passwordless SSH and generating SSH keys.
Sometimes you need to tweak your SSH daemon on an important system and you just don’t know if particular settings will break connectivity to the server or not. In such cases it’s best to test new SSHd config using separate SSH daemon instance and separate SSH port – debug it there and only then apply new configs into your primary SSHd configuration.
Creating New SSHd Config
The easiest is to start by copying /etc/ssh/sshd_config file – you will need sudo/root privileges for that:
I then just remove everything I don’t need from it, leaving bare minimum. These are the parameters I kept (I ended up renaming my config to /home/greys/sshd_config.minimal after edits)
I only updated the SSH Port parameter – you can pick any other number instead of 2222.
Starting SSH daemon with custom config file
There’s a few rules for testing SSH configuration using separate file:
you need to have sudo/root privilege (mostly to avoid mess with host SSH keys)
it’s better to increase verbosity level to see what’s going on
it’s best to run SSHd in foreground (non-daemon) mode
With these principles in mind, here’s the command line to test the config shown above:
greys@s2:~ $ sudo /usr/sbin/sshd -f /home/greys/sshd_config.minimal -ddd -D
debug2: load_server_config: filename /home/greys/sshd_config.minimal
debug2: load_server_config: done config len = 194
debug2: parse_server_config: config /home/greys/sshd_config.minimal len 194
debug3: /home/greys/sshd_config.minimal:1 setting Port 2222
debug3: /home/greys/sshd_config.minimal:10 setting HostKey /home/greys/ssh_host_rsa_key
debug3: /home/greys/sshd_config.minimal:12 setting RSAAuthentication yes
/home/greys/sshd_config.minimal line 12: Deprecated option RSAAuthentication
debug3: /home/greys/sshd_config.minimal:13 setting PubkeyAuthentication yes
debug3: /home/greys/sshd_config.minimal:18 setting AuthorizedKeysFile /var/ssh/%u/authorized_keys
debug3: /home/greys/sshd_config.minimal:20 setting PasswordAuthentication no
debug3: /home/greys/sshd_config.minimal:22 setting UsePAM yes
debug1: sshd version OpenSSH_7.4, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: private host key #0: ssh-rsa SHA256:g7xhev6zJefXRFc0ClAG4rzpFI1Ts8H7PhQ/h3PTmLM
debug1: rexec_argv[0]=’/usr/sbin/sshd’
debug1: rexec_argv[1]=’-f’
debug1: rexec_argv[2]=’/home/greys/sshd_config.minimal’
debug1: rexec_argv[3]=’-ddd’
debug1: rexec_argv[4]=’-D’
debug3: oom_adjust_setup
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 2222 on 0.0.0.0.
Server listening on 0.0.0.0 port 2222.
debug2: fd 4 setting O_NONBLOCK
debug3: sock_set_v6only: set socket 4 IPV6_V6ONLY
debug1: Bind to port 2222 on ::.
Server listening on :: port 2222.
That’s it, the configuration is ready to be tested (assuming your firewall on server doesn’t block port 2222).
Testing SSH connectivity using Different SSH Port
Here’s my login session in a separate window, connecting from my MacBook Pro to the s2 server on SSH port 2222 (I have masked my static IP with aaa.bbb.ccc.ddd and my s2 server’s IP with eee.fff.ggg.hhh):
I needed to forward X11 output from one of my Linux servers recently to run virt-manager (manager for virtual machines in KVM), and because it’s been a while I had to download and install X11 server again.
As some of you know, Xorg server is no longer shipped/installed with macOS by default. So you have to download it from XQuartz page: https://www.xquartz.org/releases/index.html. Usually you do it, install it and that’s it – no additional steps are needed.
But things are slightly different for the macOS High Sierra, apparently.
The latest release hasn’t been updated since 2016 which I believe is before High Sierra – which explains why things don’t “just work” anymore. Fear not though – I tracked the issue down and it’s explained below.
UPDATE 03/2019: MacOS Mojave works just great, you may skip Step 3 in the procedure below.
Steps to get X11 Forwarding in macOS High Sierra
Download and install the latest release from xquartz.org website
Start XQuartz
IMPORTANT: verify xauth location SSH configuration file /etc/ssh/ssh_config might contain path to xauth tool, which may be incorrect depending on your OSX/MacOS version. Here’s how to check:
