I’m writing an e-book on my approach to taking technical exams and obtaining relevant certifications.
There’s going to be two versions of the book:
Technical Exam Success – Checklist – mobile friendly, brief summary of improving your chances (you can get it by subscribing to my mailing list in sidebar)
Technical Exam Success – full edition of the book with examples and thorough description of techniques (this is a link to Leanpub placeholder, more information to follow)
Today I just wanted to share a mindmap based on the checklist – it’s a great way to structure your approach for any technical certification.
Click on the image below for the high resolution image:
Hi everyone, think by now most of us are settling into long-term working from home routine as part of COVID-19 social distancing. Hope you and your familiy are healthy and safe.
Ansible Reference – am building the ultimate go-to page for all things Ansible
How To: Filter RSyslog Messages by String
I think I’m pretty much done with the Centralised RSyslog server project – it’s finally in the stage I’m happy enough with. One of the final touches I needed on the server side was to filter more important messages into a separate file, so that I can tail that file instead of looking at all the RSyslog messages arriving from all the servers and appliances.
Why You Might Want to Filter Syslog Messages
Most of the reasons are around focusing on specific subset of the logs.
IMPORTANT: I want to highlight that there’s filtering where you drop (ignore) messages and filtering where you just forward certain logs to separate files. In my opinion, you should never discard logs – the risk of missing something important is just too high.
So collect everything, but filter it into separate files and only inspect the more useful elements. The rest is stored and quite possibly rotated very often – meaning you’re not really wasting that much space. But in a case of some security incident you’ll expand research surface and will look into all the logs and not just the usually important ones.
Specific reasons for filtering Syslog Messages
You have multiple teams or team members with different responsibilities – someone needs to look at application logs, someone else at OS services logs
You have specific processes that run regularly and need monitoring – for instance, auto-deployments or cron-based automations – they use the same common tools like HTTPS for git hooks or SSH for remote access or SUDO for privilege escalation – but will be lost among other standard traffic about HTTPS, SSH and SUDO
You have parts of logging providing additional, non-essential information – like, when you ssh onto a server, you have pam_unix session management messages that you may not be interested in. It’s important to know when someone’s unsuccessfully trying to SSH, but not nearly as important to track when someone logs in or out.
How To Filter Syslog Messages by Expression
Relatively new approach is using expression-based filters in RSyslog – they’re using a common and readable enough format of if-then:
if $msg contains 'ssh' then /logs/security.log
How To Filter Syslog Messages by Application/Service
It’s also super useful to filter by program name – this is essentially a process name.
For instance, in this syslog line highlighted sudo is the program name and not the text of the logged message:
The following RSyslog example helps me extract only sudo lines with COMMAND into /logs/security.log file. Any other sudo lines will not be captured in /logs/security.log, but will still be captured elsewhere.
if $programname == 'sudo' and $msg contains 'COMMAND' then /logs/security.log
I’ve finally decided to try running Unix Tutorial RU, copy of this blog in Russian, in Jekyll 4 CMS instead of WordPress.
Motivation for this Project
There’s a few reasons that make migration to Jekyll a very interesting project for me.
Before you ask: there’s nothing wrong with WordPress and I’m still a huge fan! But I’m embracing more and more of git-based workflows for creating and deploying things lately, so it’s a good opportunity to try blogging with git as well.
Improvements I Expect With Move to Jekyll
Here’s what I expect this project improves for me:
Much easier blog development and updating
All the development is done locally on my laptop, with any changes done and debugged locally before committing them into local git or pushing to GitHub.
Bonus: there’s no CPU/memory overhead for this development – so this is better and quicker than any VM, Docker (for WPengine dev) or even native httpd/mysql based WordPress setup.
Easier content editing
Specifically, creation of new posts and pages in Markdown using my editor of choice – Sublime Text 3. It’s possible to find plugins for Markdown editing in WordPress, but they are fairly limited, especially with the latest move towards block-based content editing.
Bonus: I can edit/write new posts without any connection to Internet (think WordPress desktop app either supports or plans to support it soon).
Better Code Examples and Shell Commands Snippets
I don’t know why, but process of posting syntax-highlighted snippets of code is more involved that necessary. Especially if one expects a preview right there in the WYSIWYG editor.
In Jekyll posting code is much simpler, even within Markdown text.
Bonus: because I’m editing the actual part of code that will be published, there’s no plugin or WYSIWYG or block editor dynamic formatting risking that some syntax will be corrupted.
Easier images management
I’m making a lot of screenshots and they use specific naming convention. If I upload them as is – files named like 2020-03-08-22-44-00-screenshot.png – it becomes rather tricky to later rename them for better SEO performance. With Jekyll renaming an image is as simple as renaming the image and updating Markdown page linking to it.
Bonus: any image uploads are as quick as moving files on my SSD drive! But yes, I obviously need to upload them into GitHub repo later.
Historical tracking of changes
This is becoming a very important point for me – it helps so much to keep track of all page updates as git commits! It’s possible to see latest update and previous versions of a page in WordPress as well, but there’s no easy way of seeing a timeline of all changes with description of changes. I’m using Jetpack status updates right now in WordPress environments and it helps, but you get only a week of history and even then you get a list of pages updated, but not the list of commit-style descriptions of what you changed.
Since I’m hosting Jekyll blogs on GitHub, I get to enjoy history, roll-back and automated deployments. Awesome!
Project Plan for Migrating WordPress Blog to Jekyll
These are the elements of migration as I see them:
Jekyll 4 Syntax Highlighting – I’m considering switching Unix Tutorial RU to Jekyll 4 as an experiment – need to get comfortable with features and extensions first
It seems RSyslog doesn’t suppot this kind of mixing config styles – so one of these config stanzas needs rewriting. In my case, I actually only needed imtcp for debug purposes – so I just commented it out and RSyslog restarted just fine.
Although Ansible provides support for managing firewall rules via module, I still find initial setup is best done with a tested batch of firewall rules instead of adding them one-by-one. Since I’m migrating CentOS 7 servers to CentOS 8 now, I decided to convert iptables into nftables.
Will probably post a Unix Tutorial Project about this, but today I’m just capturing notes.
What is nftables?
nftables is the next (current) generation of NetFilter based firewall solutions, replacing iptables and providing backward compatible tools with iptables syntax.
If all you used before is iptables, you can continue using familiar commands – but in CentOS 8 this means that on the firewall level there’s no longer iptables running, all the functionality is provided by NFT.
IMPORTANT: make sure you put this into some nft-rules.txt file outside of the /etc/sysconfig location – if things go wrong, you’ll just reboot server via hosting console and regain access.
Try/Check NFT Ruleset
Now comes the moment to disable iptables and try NFT tables in their place.
I did the following: flushed IPtables (removed any rules) and then applied NFT rules.
Flush iptables
# iptables -F
Apply NFT rules from nft-rules.txt file
# nft -f nft-rules.txt
We can now have a look at the list of active NFT rules:
# nft list ruleset
Configure nftables Rules to Apply upon Reboot
Assuming everything works as expected, we can now move the nfs-rules.txt file into default location that will be used by NFT upon reboot:
# mv nft-rules.txt /etc/sysconfig/nftables.conf
Make sure it belongs to root and has correct permissions (it’s not a script so needs no execution bits):
root@s1:~ # ls -lad /etc/sysconfig/nftables.conf -rw-------. 1 root root 5227 Mar 12 01:48 /etc/sysconfig/nftables.conf
I’m tidying up Centralised RSyslog setup on the newly reinstalled becky Raspberry Pi system. One of the tasks at hand was to configure TLS based encrypted log shipping from my dedicated servers to home RSyslog server, this post shows the command and technique I use.
How To Check SSL Connection
What we do is run openssl command with the s_client option and specify remote server we’re testing connection to. It can be an HTTPS connection (port 443) to a website (will do a post about it some other time), but in my case I’m connecting to home office server becky.ts.fm with port 6514 (TLS encrypted port for Syslog):
root@s2:/ # openssl s_client -connect becky.ts.fm:6514
CONNECTED(00000003)
depth=0 CN = becky.ts.fm, O = Tech Stack Solutions, L = Dublin, C = IE
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = becky.ts.fm, O = Tech Stack Solutions, L = Dublin, C = IE
verify error:num=21:unable to verify the first certificate
verify return:1
Certificate chain
0 s:/CN=becky.ts.fm/O=Tech Stack Solutions/L=Dublin/C=IE
i:/CN=syslog.ts.fm/O=Tech Stack Solutions/L=Dublin/C=IE
Server certificate
-----BEGIN CERTIFICATE-----
MIIEJDCCAoygAwIBAgIUJGqZcuyXa7ekrK+U8yfB2Cu54FYwDQYJKoZIhvcNAQEL
jMNHiZ0zdzolHWzkV6iKc20MxOv3ftQ1TsE7vg+/Z2fTSv2f0uirPZUPegSzwr75
ABRIDGED
9n7UHknn7/mV+lLcloOA8oyXB5zeVf+lxLufVRyhEIpLFVtRiu0Go6PW0gjwMoPM
QB/0E6WgtSDMf43f9qzSdtKNgHFw1MpxVQdULSabnI6n0gpfuUIvKDBmBazgh6lR
RtZqUqzO9pE=
-----END CERTIFICATE-----
subject=/CN=becky.ts.fm/O=Tech Stack Solutions/L=Dublin/C=IE
issuer=/CN=syslog.ts.fm/O=Tech Stack Solutions/L=Dublin/C=IE
Acceptable client certificate CA names
/CN=syslog.ts.fm/O=Tech Stack Solutions/L=Dublin/C=IE
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA256:0x09+0x08:0x04+0x08:ECDSA+SHA256:0x07+0x08:
RSA+SHA384:0x0A+0x08:0x05+0x08:ECDSA+SHA384:RSA+SHA512:
0x0B+0x08:0x06+0x08:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:
ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
SSL handshake has read 1704 bytes and written 427 bytes
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 224B0D3C5183426D7DDAD5A5FB361BC9C5175EC9CB0AA6A2F396DAAEE7178080
Session-ID-ctx:
Master-Key: 6BE67A8AD4E22029DE1B3D0DE1F4351FD0488AB1D8ABC7E25187
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1583790681
Timeout : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)
As you can see, ther is a problem. I’m checking the last line in the output, which should return code ok, but it tells me that my s2 server can’t verify the first (only) certificate presented by my RSyslog server becky.ts.fm.
From experience, I konw that’s because s2 somehow needs to acknowledge the certificate authority (CA) that issued certificate for becky.ts.fm.
How Successful TLS Connection Looks in OpenSSL
So if I specify this CA cert as a command line option, openssl will establish TLS connection and confirm code ok:
Now let’s specify the CA certificate I used for RSyslog, the connection and certificates verification should work just fine now:
root@s2:/ # openssl s_client -CAfile /etc/rsyslog.d/ca.crt -connect becky.ts.fm:6514
CONNECTED(00000003)
depth=1 CN = syslog.ts.fm, O = Tech Stack Solutions, L = Dublin, C = IE
verify return:1
depth=0 CN = becky.ts.fm, O = Tech Stack Solutions, L = Dublin, C = IE
verify return:1
Certificate chain
0 s:/CN=becky.ts.fm/O=Tech Stack Solutions/L=Dublin/C=IE
i:/CN=syslog.ts.fm/O=Tech Stack Solutions/L=Dublin/C=IE
Server certificate
-----BEGIN CERTIFICATE-----
MIIEJDCCAoygAwIBAgIUJGqZcuyXa7ekrK+U8yfB2Cu54FYwDQYJKoZIhvcNAQEL
BQAwVDEVMBMGA1UEAxMMc3lzbG9nLnRzLmZtMR0wGwYDVQQKExRUZWNoIFN0YWNr
ABRIDGED
jMNHiZ0zdzolHWzkV6iKc20MxOv3ftQ1TsE7vg+/Z2fTSv2f0uirPZUPegSzwr75
9n7UHknn7/mV+lLcloOA8oyXB5zeVf+lxLufVRyhEIpLFVtRiu0Go6PW0gjwMoPM
QB/0E6WgtSDMf43f9qzSdtKNgHFw1MpxVQdULSabnI6n0gpfuUIvKDBmBazgh6lR
RtZqUqzO9pE=
-----END CERTIFICATE-----
subject=/CN=becky.ts.fm/O=Tech Stack Solutions/L=Dublin/C=IE
issuer=/CN=syslog.ts.fm/O=Tech Stack Solutions/L=Dublin/C=IE
Acceptable client certificate CA names
/CN=syslog.ts.fm/O=Tech Stack Solutions/L=Dublin/C=IE
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA256:0x09+0x08:0x04+0x08:ECDSA+SHA256:0x07+0x08:RSA+SHA384:
0x0A+0x08:0x05+0x08:ECDSA+SHA384:RSA+SHA512:0x0B+0x08:
0x06+0x08:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:
ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
SSL handshake has read 1704 bytes and written 427 bytes
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: C6797515EEA312D7A9EC6685F895AE004798550FF70619E85F24AB5ACF80F0A9
Session-ID-ctx:
Master-Key: 4B84DF3CFE9697EEC634DC271B2A490D94B7A7AB1CA218F016B1ED141FA1479C
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1583790782
Timeout : 300 (sec)
Verify return code: 0 (ok)
^C
That’s it – this means secure connection establishes successfully, so I can enjoy secure log shipping from s2 to becky.ts.fm.
Now that I’m monitoring my logs using cetralised RSyslog, I regularly notice SSH attacks right when and as they happen. When it becomes obvious that someone’s trying to brute-force SSH, I don’t always wait to let fail2ban fix the issue – sometimes I ban the offending IP myself.
How To Ban Specific IP with fail2ban
Assuming a standard install, we’ll use the fail2ban-client command to notify sshd jail module to ban a specific IP.
Here’s how it works:
root@s1:/etc/fail2ban # fail2ban-client -vvv set sshd banip 202.70.66.228
30 7F0B121F6640 fail2ban.configreader INFO Loading configs for fail2ban under /etc/fail2ban
30 7F0B121F6640 fail2ban.configreader DEBUG Reading configs for fail2ban under /etc/fail2ban
31 7F0B121F6640 fail2ban.configreader DEBUG Reading config files: /etc/fail2ban/fail2ban.conf
31 7F0B121F6640 fail2ban.configparserinc INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
31 7F0B121F6640 fail2ban.configparserinc TRACE Reading file: /etc/fail2ban/fail2ban.conf
31 7F0B121F6640 fail2ban.configparserinc INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
31 7F0B121F6640 fail2ban.configparserinc TRACE Shared file: /etc/fail2ban/fail2ban.conf
32 7F0B121F6640 fail2ban INFO Using socket file /var/run/fail2ban/fail2ban.sock
32 7F0B121F6640 fail2ban INFO Using pid file /var/run/fail2ban/fail2ban.pid, [INFO] logging to SYSLOG
32 7F0B121F6640 fail2ban HEAVY CMD: ['set', 'sshd', 'banip', '202.70.66.228']
48 7F0B121F6640 fail2ban HEAVY OK : 1
48 7F0B121F6640 fail2ban.beautifier HEAVY Beautify 1 with ['set', 'sshd', 'banip', '202.70.66.228']
1
48 7F0B121F6640 fail2ban DEBUG Exit with code 0
Once you become comfortable, you can omit the -vvv option and skip all this verbose output:
root@s1:/etc/fail2ban # fail2ban-client set sshd banip 202.70.66.229
1