Unix Tutorial Projects: Centralised RSyslog
Think I have enough time writing about ongoing efforts, learnings and projects of mine here on Unix Tutorial, so it makes perfect sense to introduce another regular thing: Unix Tutorial Projects.
Usually these are small enough things to fit into a calendar week, and at the end of each project I’ll be publishing a post like today’s with expectations and achievements thus far.
Centralised RSyslog
What do I mean by that? It’s going to be a server (initially – one of the Raspberry Pi systems in my home office) for collecting logs from the rest of my home office, automation, Unix Tutorial group and hosting/troubleshooting systems.
Main benefits of centralising logs are:
- easier and quicker access to logs – no need to log into each system
- greater flexibility reviewing logs – sometimes a quick grep across the same log on all the systems will reveal far more than looking at each individual log
- better understanding of fundamental issues – if something is too noisy with logs, this is usually a good thing: there are issues that need to be fixed, rather than alers to be muted or individual servers to be ignored
- easier analysis – you can forward all the logs into ELK (Elastic, Logstash, Kibana) setup to gain instant visualisation, etc
- robust incident response – sometimes issues are so sever that your servers go offline or disks fail or data gets lost. Having a copy of all the important logs stored elsewhere (on your centalised logs server) will be a great tool for incident response
Now, I’m also a DataDog user myself so I realise this mini-project is a reinvention of a whell in some ways, but it seems like both a fun way to spend a few geeky evenings and a great way to improve my RSyslog understanding which I need for other projects.
My Plan for Centalised RSyslog Setup
I’m doing the following so far, let me know if you think I’m missing a step or two:
- attach a USB stick to my Raspberry Pi for logs storage – always good to have logs on a separate filesystem, so if it gets 100% full you don’t impact the root filesystem
- install rsyslog from standard Raspbian repos
- open external port on my Internet router and allow RSyslog traffic on it
- setup self-signed CA authority and generate keys/certs for RSyslog clients
- setup encryption for shipping logs safely (TLS) – wonder if this will be taking a serious toll on my Raspberry Pi’s CPU
- configure RSyslog to create separate log file for each server
- update RSyslog on each of the monitored servers to ship logs to my home office log collector
- log each RSyslog client into separate hostname based file
- possibly take it even further and filter logs by application or message content
- setup real-time monitoring of syslog logs with grc colouriser
- write a bunch of Python scripts for assessing log volumes and predict storage requirements
I’ll update this post when I’m done.