How To: Filter RSyslog Messages by String
I think I’m pretty much done with the Centralised RSyslog server project – it’s finally in the stage I’m happy enough with. One of the final touches I needed on the server side was to filter more important messages into a separate file, so that I can tail that file instead of looking at all the RSyslog messages arriving from all the servers and appliances.
Why You Might Want to Filter Syslog Messages
Most of the reasons are around focusing on specific subset of the logs.
IMPORTANT: I want to highlight that there’s filtering where you drop (ignore) messages and filtering where you just forward certain logs to separate files. In my opinion, you should never discard logs – the risk of missing something important is just too high.
So collect everything, but filter it into separate files and only inspect the more useful elements. The rest is stored and quite possibly rotated very often – meaning you’re not really wasting that much space. But in a case of some security incident you’ll expand research surface and will look into all the logs and not just the usually important ones.
Specific reasons for filtering Syslog Messages
- You have multiple teams or team members with different responsibilities – someone needs to look at application logs, someone else at OS services logs
- You have specific processes that run regularly and need monitoring – for instance, auto-deployments or cron-based automations – they use the same common tools like HTTPS for git hooks or SSH for remote access or SUDO for privilege escalation – but will be lost among other standard traffic about HTTPS, SSH and SUDO
- You have parts of logging providing additional, non-essential information – like, when you ssh onto a server, you have pam_unix session management messages that you may not be interested in. It’s important to know when someone’s unsuccessfully trying to SSH, but not nearly as important to track when someone logs in or out.
How To Filter Syslog Messages by Expression
Relatively new approach is using expression-based filters in RSyslog – they’re using a common and readable enough format of if-then:
if $msg contains 'ssh' then /logs/security.log
How To Filter Syslog Messages by Application/Service
It’s also super useful to filter by program name – this is essentially a process name.
For instance, in this syslog line highlighted sudo is the program name and not the text of the logged message:
Mar 29 09:45:38 s7 sudo: greys : TTY=pts/0 ; PWD=/home/greys ; USER=root ; COMMAND=/bin/grep -E ^pi: /etc/shadow
The following RSyslog example helps me extract only sudo lines with COMMAND into /logs/security.log file. Any other sudo lines will not be captured in /logs/security.log, but will still be captured elsewhere.
if $programname == 'sudo' and $msg contains 'COMMAND' then /logs/security.log
