Unix Tutorial Projects: Centralised RSyslog

Think I have enough time writing about ongoing efforts, learnings and projects of mine here on Unix Tutorial, so it makes perfect sense to introduce another regular thing: Unix Tutorial Projects.

Usually these are small enough things to fit into a calendar week, and at the end of each project I'll be publishing a post like today's with expectations and achievements thus far.

Centralised RSyslog

What do I mean by that? It's going to be a server (initially – one of the Raspberry Pi systems in my home office) for collecting logs from the rest of my home office, automation, Unix Tutorial group and hosting/troubleshooting systems.

Main benefits of centralising logs are:

• easier and quicker access to logs – no need to log into each system
• greater flexibility reviewing logs – sometimes a quick grep across the same log on all the systems will reveal far more than looking at each individual log
• better understanding of fundamental issues – if something is too noisy with logs, this is usually a good thing: there are issues that need to be fixed, rather than alers to be muted or individual servers to be ignored
• easier analysis – you can forward all the logs into ELK (Elastic, Logstash, Kibana) setup to gain instant visualisation, etc
• robust incident response – sometimes issues are so sever that your servers go offline or disks fail or data gets lost. Having a copy of all the important logs stored elsewhere (on your centalised logs server) will be a great tool for incident response

Now, I'm also a DataDog user myself so I realise this mini-project is a reinvention of a whell in some ways, but it seems like both a fun way to spend a few geeky evenings and a great way to improve my RSyslog understanding which I need for other projects.

My Plan for Centalised RSyslog Setup

I'm doing the following so far, let me know if you think I'm missing a step or two:

• attach a USB stick to my Raspberry Pi for logs storage – always good to have logs on a separate filesystem, so if it gets 100% full you don't impact the root filesystem
• install rsyslog from standard Raspbian repos
• open external port on my Internet router and allow RSyslog traffic on it
• setup self-signed CA authority and generate keys/certs for RSyslog clients
• setup encryption for shipping logs safely (TLS) – wonder if this will be taking a serious toll on my Raspberry Pi's CPU
• configure RSyslog to create separate log file for each server
• update RSyslog on each of the monitored servers to ship logs to my home office log collector
• log each RSyslog client into separate hostname based file
• possibly take it even further and filter logs by application or message content
• setup real-time monitoring of syslog logs with grc colouriser
• write a bunch of Python scripts for assessing log volumes and predict storage requirements

I'll update this post when I'm done.

See Also

• Unix Tutorial Projects
• RSyslog website
• RSyslog: Separate Log File for Each Host
• Review Latest Logs with tail and awk
• Script: Save Your Session Log

Liked it? Take a second to support Gleb Reys on Patreon!