I’ve come across XZ files more than once, most recently when downloading Kali Linux. It should be a fairly common knowledge now, but some operating systems still don’t support is – so I decided to research and to document it.
What is XZ file?
XZ is a modern lossless compression algorithm, it’s more efficient that gzip and bzip2. Apparently, many Linux distros are using XZ for compressing their software packages or ISO images. XZ is an open-source format maintained via Tukaani Project – XZ.
Support for XZ in tar
Modern Linux distros (certainly Ubuntu 19.x and CentOS 7.x when I checked today) have xz utils package installed, which allows tar command to automatically unpack XZ files.
If xz-utils didn’t come preinstalled, you can install it like this in Ubuntu/Debian:
# apt install xz-utils
or like this in CentOS/Fedora/RedHat:
# yum install xz
gzip/gunzip Support for XZ in macOS
Although tar in macOS doesn’t support XZ format natively:
greys@mcfly:~/Downloads $ tar xzvf kali-linux-2020.1-rpi3-nexmon-64.img.xz tar: Error opening archive: Unrecognized archive format
… you can still use gunzip to decompress the XZ file in macOS:
greys@mcfly:~/Downloads $ ls -al kali-linux-2020*
-rw-r--r--@ 1 greys staff 259647 30 Jan 22:57 kali-linux-2020-1.png
-rw-r--r--@ 1 greys staff 1048328860 30 Jan 23:06 kali-linux-2020.1-rpi3-nexmon-64.img.xz
greys@mcfly:~/Downloads $ gunzip kali-linux-2020.1-rpi3-nexmon-64.img.xz
greys@mcfly:~/Downloads $ ls -ald kali-linux-2020.1-rpi3-nexmon-64.img
-rw-r--r-- 1 greys staff 6999999488 30 Jan 23:06 kali-linux-2020.1-rpi3-nexmon-64.img
xz-utils in macOS
If you insist on managing XZ files using xz utils, you’ll need to install them with brew:
… and then a whole bunch of XZ commands becomes available:
2019 has been a tremendous year for my blog: almost a million visits to my posts and pages, hundreds of interesting topics researched and even more planned for the year ahead.
Here are just some of the notable changes on Unix Tutorial:
Glimpse, a fork of GIMP graphics editor, finally became available
Scary Stuff
It didn’t always seem like it, but 2019 turned out to be a very scary year in terms of exploits, hardware and software vulnerabilities and hacks of major software repos
Docker Hub was hacked and information about 190k users (including password hashes) got leaked in April
PEAR (PHP) repository got hacked
Even more hardware attacks got identified for both Intel and AMD processes
GitHub, Bitbucket and GitLab all got affected by ransom attacks encrypting repositories
In May Firefox had that incident with intermediary certificates which instantly blocked browser extensions in millions of browsers
We live in exciting times. It’s been fun to try new products and services in 2019 and all the things indicate that 2020 will be even more impressive in terms of innovations and rapid adoption of new standards and technologies.
Hey everyone, just wanted to wish you all a Happy New Year and to welcome you into 2020!
Unix Tutorial: 2019 in Numbers
2019 has been the greatest year in all of the UnixTutorial.org history: 970K views – that’s just one typical week shy of a whole 1 million post and page views!
I’m super thankful to everyone who stopped by and plan to make 2020 an even bigger success – for you and for me.
I made 258 posts last year, so it will be really great to do 300 or more in 2020.
Need Any Help?
Please get in touch or ask a question in chat widget, I would love to help you solve quick and not so quick Unix challenges.
I’m also looking for opportunities in training – so if you have a team of technical professionals looking for guidance with Unix/Linux administration – please get in touch to arrange a custom training program or just have me create or review a manual or technical guide.
We’re spending Christmas break away from Ireland this year, so most of the week has been pre-scheduled posts as I’m offline until we come back.
Want to take a moment and wish you a very happy Christmas! Great time to spend time with family and maybe catch up on RSS and ebook reading on some of the quiet wintery mornings.
Will be back soon, hope you’re all having a great time!
Live Chat on Unix Tutorial
Unix Tutorial Chat
Want to try an experiment for a few weeks – added a live chat box on Unix Tutorial, so if I’m online you can ask me a question.
Feel free to ask anything and I’ll do my best to help out!
Disclaimer:
this is just an experiment, I may take this down at any time
for now it’s a free support and therefore a best effort service
I’m in Dublin, Ireland – if you’re in a different time zone, I may be offline during your day time
for now I plan to only help with basic Unix/Linux questions, so mostly advice and not actual technical support
macOS has a very flexible and striving kernel modules ecosystem. There are so many granularities for integrating with OS kernel that modules are called extensions. Many third party software packages install their own kernel extensions in macOS.
List Kernel Extensions in macOS
Here’s how many kernel extensions my MacBook with macOS Catalina runs:
It is so common for software to be installing kernel extensions in macOS, that some extensions stay running long after you stop using the software that brought them. Whether extensions stay or not highly depends on how you removed or upgraded the software (most installers are pretty good at tidying up).
If you exclude extensions starting with com.apple, you can see the third party ones:
This has been a useful exercise, cause I already see how some kexts are no longer needed – I’ll find out how to safely remove them and will create another post on Unix Tutorial later.
I love reading man pages for even the most basic Unix commands like ls, because there’s always something interesting to learn. Today I discovered that it’s possible to sort ls output by file size.
Sort ls by file size
Simply add the capital S to command line options and you’ll see your output sorted by file size. Be sure to use long output form (-l option). I’m adding more command to the mix because I know that /usr/lib is a directory with quite a number of files:
greys@srv:/usr/bin $ ls -lS /usr/lib | more
total 492
-rwxr-xr-x 1 root 74688 Feb 1 2019 klibc-ae-2A4n9ZnfImcw9WIt7dF-3OvQ.so
-rw-r--r-- 1 root 56480 Jan 14 2018 libdiscover.so.2.0.1
-rw-r--r-- 1 root 22448 Aug 29 2016 libsupp.a
drwxr-xr-x 27 root 20480 Nov 24 01:00 python2.7
drwxr-xr-x 24 root 20480 Nov 26 22:12 x86_64-linux-gnu
-rw-r--r-- 1 root 18336 Feb 26 2019 libau.so.2.10
drwxr-xr-x 3 root 12288 Sep 14 18:43 git-core
drwxr-xr-x 30 root 12288 Sep 14 18:44 python3.7
drwxr-xr-x 15 root 12288 Nov 26 22:12 systemd
drwxr-xr-x 2 root 4096 Aug 30 10:03 apparmor
drwxr-xr-x 5 root 4096 Aug 30 10:02 apt
drwxr-xr-x 2 root 4096 Aug 30 10:05 bfd-plugins
Reverse Sorting with ls command
Don’t know about you, but I prefer reverse sorting in most commands – this means largest objects are shown at the end of the output. The reason I do it this way is to avoid using pagers like more or less. Another reason is I can scroll up to explore further output of the ls command without having to rerun anything.
So here’s the same command run with the -r option for reverse output (I’m only showing last few lines):
...
drwxr-xr-x 2 root 4096 Aug 30 10:04 discover
drwxr-xr-x 2 root 4096 Aug 30 10:04 dbus-1.0
drwxr-xr-x 2 root 4096 Nov 3 20:02 console-setup
drwxr-xr-x 2 root 4096 Aug 30 10:05 compat-ld
drwxr-xr-x 2 root 4096 Nov 10 06:04 chrony
drwxr-xr-x 2 root 4096 May 24 2019 binfmt.d
drwxr-xr-x 2 root 4096 Aug 30 10:05 bfd-plugins
drwxr-xr-x 5 root 4096 Aug 30 10:02 apt
drwxr-xr-x 2 root 4096 Aug 30 10:03 apparmor
drwxr-xr-x 15 root 12288 Nov 26 22:12 systemd
drwxr-xr-x 30 root 12288 Sep 14 18:44 python3.7
drwxr-xr-x 3 root 12288 Sep 14 18:43 git-core
-rw-r--r-- 1 root 18336 Feb 26 2019 libau.so.2.10
drwxr-xr-x 24 root 20480 Nov 26 22:12 x86_64-linux-gnu
drwxr-xr-x 27 root 20480 Nov 24 01:00 python2.7
-rw-r--r-- 1 root 22448 Aug 29 2016 libsupp.a
-rw-r--r-- 1 root 56480 Jan 14 2018 libdiscover.so.2.0.1
-rwxr-xr-x 1 root 74688 Feb 1 2019 klibc-ae-2A4n9ZnfImcw9WIt7dF-3OvQ.so
greys@srv:~ $
That’s it for today – hope you’ve learned something new today.
Just as I published last Unix Tutorial Digest on November 5th, RHEL 8.1 release got shipped – think this is a great incremental release bringing a number of key improvements to the Red Hat Enterprise Linux 8.
RHEL 8 Release Cadence
Red Hat announced that going forward Red Hat Enterprise Linux OS will be receiving regular updates every 6 months. Since RHEL 8 release was in May 2019, this current RHEL 8.1 update is right on time, 6 months after.
RHEL 8.1 Improvements I Want To Try
There’s a number of great improvements in this release:
Live Kernel Patching with kpatch
SELinux profiles for containers and tbolt for Thunderbolt devices – will be cool to try on my RHEL 8 PC
Perhaps try RHEL 7.6 in-place upgrade to RHEL 8.1
Review rhel-system-roles and specifically the new storage role added in RHEL 8.1
I also want to try Red Hat Universal Base Image for RHEL 8 – it’s been around since initial release in May, I just never got the chance to have a look.
Since reinstalling certain software packages on my Linux desktop, Linux laptop and Macbook a few times in the past months, I’m positively in love with the Homebrew project. Today I’ll share my understanding of its basic deployment modules: casks vs formulae.
What Homebrew does
Homebrew project is the magic behind brew command – it’s a software manager that assists with source-code and binary distributions of various software projects.
Specifically, brew is popular on macOS platform where most of software is traditionally installed using graphical user interface (GUI) or via AppleScript automation. Recent versions of macOS exposed a number of relevant interfaces via command line, but average user (and most of advanced users/developers) still had to resort to some enterprise level proprietary delivery and installation mechanism.
With the arrival of Homebrew, things got much easier. A typical software installation is now simpler and quite comparable to the number of steps required in other Unix and Unix-like operating systems:
This is mostly due to the fact that Homebrew supports both standard and third-party software using its own centralised approach to software packaging. Application owners define a formula for installing their software, get in touch with Homebrew team to add it to the project, and going forward things happen pretty much automatically.
In Linux world, many software packages are still distributed independently or have to make their way into a specific application/software store – so many small projects are hard to find.
brew formula
Each Open Source package integrated with Homebrew is configured using a brew formula. It’s a Ruby language based configuration file that explains how the software can be downloaded from GitHub or similar repository and then compiled.
class Htop < Formula
desc "Improved top (interactive process viewer)"
homepage "https://hisham.hm/htop/"
url "https://hisham.hm/htop/releases/2.2.0/htop-2.2.0.tar.gz"
sha256 "d9d6826f10ce3887950d709b53ee1d8c1849a70fa38e91d5896ad8cbc6ba3c57"
revision 1
bottle do
cellar :any
sha256 "c06ff60960f64f5c8395f53d7419cbcce2a22ee87f0cb0138352c8a88111d21c" => :catalina
sha256 "77aa302765353b4085dcad52356d3264183e06310dda8d5bac64642299ea2902" => :mojave
sha256 "0ebfb655b91566ba31f8effc94d642a43305ff95bdc9b30b46fadc132e2ced0c" => :high_sierra
sha256 "ed93b86f011de155c5d261b8c9cc9cb81fd0017667bf3ebe26ee090716bcd650" => :sierra
end
head do
url "https://github.com/hishamhm/htop.git"
depends_on "autoconf" => :build
depends_on "automake" => :build
depends_on "libtool" => :build
end
depends_on "pkg-config" => :build
depends_on "ncurses" # enables mouse scroll
def install
system "./autogen.sh" if build.head?
system "./configure", "--prefix=#{prefix}"
system "make", "install"
end
def caveats; <<~EOS
htop requires root privileges to correctly display all running processes, so you will need to run `sudo htop`.
You should be certain that you trust any software you grant root privileges.
EOS
end
test do
pipe_output("#{bin}/htop", "q", 0)
end
end
brew install
Installing software from brew formulae is easy:
$ brew install htop
brew cask
brew cask is an extension to standard brew based software management, it’s a type of formula that documents the process of installing a graphical application (and not entirely coincidentally closed-source software).
Similar to brew formula, cask defines where software can be downloaded and what dependenices it has, but specification is so flexible that you can even download binary packages from developer websites. If it’s a paid project, you’ll probably get a chance to download a trial copy (that you later can enable using a valid serial number for that software).
Here’s brew cask for my password manager of choice, 1Password:
