I’m not a software developer, but find myself writing and managing more and more code in the recent years – mostly infrastructure as code, but also Python scripts for my mini-projects.
I’ve been a user of Tower for the past 2 years and must say it’s a pleasure to use. I tried other GUI solutions for working with git, but on MacOS this Tower thing is so good there’s no competition.
Improvements in Tower 4.0
The biggest thing is the massively improved undo function – Cmd+Z now helps with undoing quite a bunch of accidental changes in your git environment.
For my (fairly basic) knowledge of git, here are the common mistakes that Cmd+Z will help with:
So, long story… After upgrading to macOS Catalina my years-old automount.sh script running via cron stopped working. It’s been a long enough journey of fixing the script itself (sudo permissions, PATH variable not having some important directories in it when run as a script), but after script was fixed I faced another problem: cron processes keep piling up.
Why is this a problem? Eventually, my Macbook would end up with having more than 10 thousand (!) cron related processes and would just run out of process space – no command can be typed, no app can be started. Only shutdown and power on would fix this.
I’ve been looking at this problem for quite some time, and now that I’m closer to solving it I’d like to share first findings.
What is this cron thing?
Don’t remember if I mentioned cron much on Unix Tutorial, so here’s a brief summary: cron is a system service that helps you schedule and regularly run commands. It has crontabs: files which list recurrence pattern and the command line to run.
Here’s an example of a crontab, each asterisk represents a parameter like “day of the week”, “hour”, “minute”, etc. Asterisk means “every value”, so this below would run my script every minute:
* * * * /Users/greys/scripts/try.sh
And here’s my automounter script, it runs every 15 minutes (so I’m specifying all the valid times with 15min interval – 0 minutes, 15 minutes, 30 minutes and 45 minutes):
Every user on your Unix-like system can have a crontab (and yes, there’s a way to prohibit cron use for certain users), and usually root or adm user has lots of OS specific tidy-up scripts in Linux and Solaris systems.
The thing with cron is it’s supposed to be this scheduler that runs your tasks regularly and then always stays in the shadows. It’s not meant to be piling processes up, as long as your scripts invoked from cron are working correctly.
Debugging cron in macOS
Turns out, /usr/sbin/cron has quite a few options for debugging in macOS:
-x debugflag[,...]
Enable writing of debugging information to standard output. One or more of the
following comma separated debugflag identifiers must be specified:
bit currently not used
ext make the other debug flags more verbose
load be verbose when loading crontab files
misc be verbose about miscellaneous one-off events
pars be verbose about parsing individual crontab lines
proc be verbose about the state of the process, including all of its offspring
sch be verbose when iterating through the scheduling algorithms
test trace through the execution, but do not perform any actions
What I ended up doing is:
Step 1: Kill all the existing crons
mcfly:~ greys$ sudo pkill cron
Step 2: Quickly start an interactive debug copy of cron as root
When I say “quickly” I’m referring to the fact that cron service is managed by launchd in macOS, meaning you kill it and it respawns pretty much instantly.
So I would get this error:
mcfly:~ root# /usr/sbin/cron -x ext,load,misc,pars,proc,sch -sh: kill: (23614) - No such process debug flags enabled: ext sch proc pars load misc log_it: (CRON 24156) DEATH (cron already running, pid: 24139) cron: cron already running, pid: 24139
And the approach I took is kill that last running process and restart cron in the same command line:
I’ll admit: this is probably way too much information, but when you’re debugging an issue there’s no such thing as too much – you’re getting all the clues you can get to try and understand the problem.
In my case, nothing was found: cron would start my cronjob, let it finish, report everything was done correctly and then still somehow leave an extra process behind:
[17464] TargetTime=1579264860, sec-to-wait=0
[17464] load_database()
[17464] spool dir mtime unch, no load needed.
[17464] tick(41,12,16,0,5)
user [greys:greys::…] cmd="/Users/greys/scripts/try.sh"
[17464] TargetTime=1579264920, sec-to-wait=60
[17464] do_command(/Users/greys/scripts/try.sh, (greys,greys,))
[17464] main process returning to work
[17464] TargetTime=1579264920, sec-to-wait=60
[17464] sleeping for 60 seconds
[17473] child_process('/Users/greys/scripts/try.sh')
[17473] child continues, closing pipes
[17473] child reading output from grandchild
[17474] grandchild process Vfork()'ed
log_it: (greys 17474) CMD (/Users/greys/scripts/try.sh)
[17473] got data (56:V) from grandchild
I’m still going to revisit this with a proper fix, but there’s at least an interim one identified for now: you must forward all the output from each cronjob to /dev/null.
In daily (Linux-based) practice, I don’t redirect cronjobs output because if there’s any output generated – it’s likely an error that I want to know about. cron runs a command, and if there’s any output, it sends an email to the user who scheduled the command. You see the email, inspect and fix the problem.
But in macOS Catalina, it seems this won’t work without further tuning. Perhaps there are some mailer related permissions missing or something like that, but fact is that any output generated by your cronjob will make cron process keep running (even though your cronjob script has completed successfully).
So the temporary fix for me was to turn my crontab from this:
That’s it for now! I’m super glad I finally solved this – took a few sessions of reviewing/updating my script because frankly I focused on the script and not on the OS itself.
I was researching SSH daemon configuration on my macOS Catalina system and realised that lsof is still the best tool for meaningfully confirming network ports that are LISTENed to.
What does it mean for port to LISTEN?
All the network services in Linux (and Windows, actually) operating systems start with the same basic pattern: some process is managing incoming network connections.
Nowadays most of network services are directly managing their own network connections – meaning service like SSH daemon or Apache web server are made available via main process (sshd or httpd in my examples) constantly running and waiting for incoming network connections on specific port. For SSH server, default port is 22. For web servers, default ports are 80 and 443.
When we say a port is LISTENing, it means there’s a process running on your system that’s monitoring this specific port. SSH is therefore listening on/for port 22, Apache (httpd) is listening for port 80 and possibly 443.
Meaningful TCP ports reporting vs Default
You may remember that netstat command shows network ports as well, but its implementations are sometimes limited to just confirming that a certain port is listened to, without helping us understand what process is doing that port listening:
That’s why I prefer the lsof tool – it’s reporting processes information, which means any output you get is bound to contain processes numbers (PIDs) and most likely process names (binary names like sshd or httpd).
Use lsof to show listening TCP ports
lsof command has specific options for reporting processes with network activity: -iTCP will report TCP specific information, and -sTCP:LISTEN qualifier will filter just the processes that are listening for incoming connections on TCP ports (rather than client processes that only initiate outgoing network connections).
Don’t Forget to run lsof with sudo!
Normally lsof is super useful even with standard user privileges, but since I’m investigating a system service (SSH server), I have to run lsof as root. Otherwise lsof will report a list of network services, but hide the ones running above your standard user privilege level.
On modern Linux servers, lsof without sudo won’t show me anything:
greys@s2:~ $ lsof -iTCP -sTCP:LISTEN greys@s2:~ $
My complete command to list all the services listening for incoming TCP connections in will therefore look like this (this is the macOS example):
IMPORTANT: Note that because SSH is a standard service, lsof reports its name (ssh) rather than port number (22) in the last column of output. TCP *:ssh means process is listening for TCP port for SSH service.
I was expecting sshd, actually. But turns out remote access via SSH is managed by launchd process in recent macOS versions. Once someone logs in though, you’ll see sshd process spun up to manage the connection.
That’s it for today, hope you learned something new!
Now that Jekyll 4 is the default, you simply install jekyll gem and it will get you the latest version of the 4.x branch.
But Jekyll gem install would fail with one of the dependencies like this:
greys@mcfly:~/proj/gleb.reys.net $ sudo gem install jekyll
Password:
Fetching: public_suffix-4.0.1.gem (100%)
Successfully installed public_suffix-4.0.1
Fetching: addressable-2.7.0.gem (100%)
Successfully installed addressable-2.7.0
Fetching: colorator-1.1.0.gem (100%)
Successfully installed colorator-1.1.0
Fetching: http_parser.rb-0.6.0.gem (100%)
Building native extensions. This could take a while…
Successfully installed http_parser.rb-0.6.0
Fetching: eventmachine-1.2.7.gem (100%)
Building native extensions. This could take a while…
Successfully installed eventmachine-1.2.7
Fetching: em-websocket-0.5.1.gem (100%)
Successfully installed em-websocket-0.5.1
Fetching: concurrent-ruby-1.1.5.gem (100%)
Successfully installed concurrent-ruby-1.1.5
Fetching: i18n-1.7.0.gem (100%)
HEADS UP! i18n 1.1 changed fallbacks to exclude default locale.
But that may break your application.
Please check your Rails app for 'config.i18n.fallbacks = true'. If you're using I18n (>= 1.1.0) and Rails (< 5.2.2), this should be
'config.i18n.fallbacks = [I18n.default_locale]'.
If not, fallbacks will be broken in your app by I18n 1.1.x.
For more info see:
https://github.com/svenfuchs/i18n/releases/tag/v1.1.0
Successfully installed i18n-1.7.0
Fetching: ffi-1.11.1.gem (100%)
Building native extensions. This could take a while…
Successfully installed ffi-1.11.1
Fetching: sassc-2.2.1.gem (100%)
Building native extensions. This could take a while…
Successfully installed sassc-2.2.1
Fetching: jekyll-sass-converter-2.0.1.gem (100%)
ERROR: Error installing jekyll:
jekyll-sass-converter requires Ruby version >= 2.4.0.
Which means that we need to upgrade Ruby, easily done with brew.
greys@mcfly:~/proj/gleb.reys.net $ sudo gem install jekyll
Password:
Jekyll 4.0 comes with some major changes, notably:
Our link tag now comes with the relative_url filter incorporated into it.
You should no longer prepend {{ site.baseurl }} to {% link foo.md %}
For further details: https://github.com/jekyll/jekyll/pull/6727
Our post_url tag now comes with the relative_url filter incorporated into it.
You shouldn't prepend {{ site.baseurl }} to {% post_url 2019-03-27-hello %}
For further details: https://github.com/jekyll/jekyll/pull/7589
Support for deprecated configuration options has been removed. We will no longer
output a warning and gracefully assign their values to the newer counterparts
internally.
Successfully installed jekyll-4.0.0
Parsing documentation for jekyll-4.0.0
Done installing documentation for jekyll after 0 seconds
1 gem installed
Standard command like hostname will still work in macOS, but perhaps it’s best to use the native way of updating system information on your Mac? scutil command is here to help.
I’m traveling on business today and tomorrow, so will mostly be working and blogging from my MacBook Pro – hence possibly a few macOS posts in the row. Today I’m taking a closer look at how kernel extensions are managed and where they can be found in macOS.
Listing all kernel extensions in macOS
kextfind is another cool utility from the kext* toolkit of standard macOS commands. Run without any parameters, it will show full path to every kernel extension on your system.
IMPORTANT: by default kextfind shows you all of the kernel extensions, not just the active ones.
If we just want to confirm what’s currently loaded and used by macOS kernel, then run kextfind with -loaded option. Naturally, it will be a shorter list:
macOS has a very flexible and striving kernel modules ecosystem. There are so many granularities for integrating with OS kernel that modules are called extensions. Many third party software packages install their own kernel extensions in macOS.
List Kernel Extensions in macOS
Here’s how many kernel extensions my MacBook with macOS Catalina runs:
It is so common for software to be installing kernel extensions in macOS, that some extensions stay running long after you stop using the software that brought them. Whether extensions stay or not highly depends on how you removed or upgraded the software (most installers are pretty good at tidying up).
If you exclude extensions starting with com.apple, you can see the third party ones:
This has been a useful exercise, cause I already see how some kexts are no longer needed – I’ll find out how to safely remove them and will create another post on Unix Tutorial later.
I’ve used a number of screenshot applications on my computers in the last few years, and it’s a never ending search for the perfect balance of flexibility and performance. Today I’m testing a new approach using native macOS grab functionality together with Hazel app.
Hazel by Noodlesoft
In case you haven’t heard about it, Hazel is an unbelievably powerful and flexible file management and automation tool for macOS:
Hazel by Noodlesoft
You select incoming folders to process (in my screenshot: Downloads and Desktop) and then define rules for automatically processing files. Hazel is running in background, monitoring folders for matching rules you configured and applying them as it finds new files – within seconds of a new file matching a Hazel rule it gets dispatched.
Here’s just a few of the workflows I have:
Rename PDF files to a specific name based on some keywords found inside the PDF document itself (great for automatically renaming utility bills)
Move ISO file downloads into specific folder on my NAS server
Move large files from Downloads folder into special archive area on NAS (to save space on laptop)
Rename all sorts of files to include datestamp – great for making these files easily found from Bash scripts later
Common Issues with Screenshot Software
I’ve found that most screenshot making software (for macOS) is not flexible enough in one way or another:
many apps are using proprietary database for storing images instead of standard folder
there’s no easy way to add screenshots from other software or folders – so unless you used the app to make a screenshot (or retake a screenshot of an existing image file) – it won’t make it
lots of apps allow you to export and import images but metadata is lost (like the date/time when the screenshot was taken)
there’s the problem of bloating up apps too, so something that started as a perfect screenshot tool can become an unusable (for me) collection of all sorts of image and video capturing functinality that’s just too slow or complex to use any longer
The Problem with Cloud Sync
I would also like to have screenshots from different laptops and PCs to be synced if at all possible – and it’s quite hard to find a screenshot app capable of doing this intelligently enough.
There’s a few great solutions that are geared towards team work and collaboration, but the problem with them is I have to create separate account for each computer or that all the images end up in a cloud with potentially auto-generated links.
Approach I’m trying Today
I decided to try a much simpler approach:
use built-in screenshot tool in macOS
use Hazel app to auto-collect screenshots from Desktop and place them into Pictures/screenshots folder
use Hazel app to auto-rename screenshots for better file sorting
Use Finder for accessing the screenshots folder with previews
Configure folder processing rule in Hazel
Here’s how my rule looks:
Hazel processing for screenshots
macOS Finder settings for the screenshots folder
And these are the settings I’ve selected for the screenshots folders:
I think it’s working out nicely: all my screenshots are sorted by the date they’re taken, there’s image size indication and thumbnails large enough for quickly finding the screenshot I want:
screenshots folder in macOS Finder
Syncing screenshots folder
Documents folder is iCloud synced, so I’m hoping to configure this setup on both desktop and laptop, this way both of them will always have the same screenshot collection. Potentially, I can even use my Ubuntu 19.10 XPS laptop to capture screenshots into Dropbox folder, which can be processed by Hazel on macOS to import images.
That’s it for now! Let me know how you’re dealing with screenshots and what tools you use!
I’ve just learned about a super cool command in macOS: mdind. It consults a database to find files using type or filename. mdfind is quite similar to locate command, I think, but more powerful?
Find file by filename with mdfind
macOS comes with Apache webserver pre-installed – you just need to configure and active it. It starts with httpd.conf file, that is tricky to find unless you know where to look.
Hurray! macOS 10.15 codenamed Catalina is finally here, available for download and install. In my case, as you can see from the screenshot, it’s ready to be installed any minute.
macOS Catalina Installe
I’ve watched some macOSCatalina reviews based on beta but didn’t want to risk installing it on my only work laptop until the final release.
macOS Catalina Improvements
New separate apps for managing music, podcasts and movies – wonder what the minimised music player is like
Screen Time is now managed from macOS – that’s a super powerful feature for controlling what apps everyone in your familiy (kids, especially) is allowed to run at chosen time intervals
Photos improvements – every version brings new things to Photos app, so I’m going to revisit it for sure
Notes improvements – I’ve just been thinking to give Apple Notes another try for daily to-do lists cause I need interoperability with Linux (will use iCloud.com based notes on Dell XPS laptop)
Voice improvements – a lot has improved even in macOS Mojave, but Catalina is meant to bring things to a whole new level. I really like what Apple is doing with voice recognition and interaction
Notable Changes in macOS Catalina
iTunes is gone – there are now separate apps for managing music and movies
Integration with Apple account – you can see Apple ID and related settings right in your System Preferences now (used to be hidden in iTunes, I think)
Use your iPad as second screen – this should be fun if you have a relatively recent model of iPad and iOS – think iPad’s screen is perfect for some text editing or text console for them SSH sessions
That’s it for now! I’m off to finish the Catalina install!
