How To Resume wget Downloads

Resume wget download

I’m downloading CentOS 8 Stream as we speak, and it’s a large enough ISO file – standard 8GB DVD image. I stopped download because I wanted to restart it in a tmux session, and realised that it’s a good opportunity to explain how this works.

How To Start Wget Download

Assuming you go to the CentOS Stream Download page and pick an image to download, you’ll end up with a download URL like this:

http://ftp.heanet.ie/pub/centos/8-stream/isos/x86_64/CentOS-Stream-x86_64-dvd1.iso

Using wget, here’s how you can start downloading it:

greys@redhat:/dist/iso $ wget http://ftp.heanet.ie/pub/centos/8-stream/isos/x86_64/CentOS-Stream-x86_64-dvd1.iso
--2019-09-30 14:35:36--  http://ftp.heanet.ie/pub/centos/8-stream/isos/x86_64/CentOS-Stream-x86_64-dvd1.iso
Resolving ftp.heanet.ie (ftp.heanet.ie)… 87.44.34.235
Connecting to ftp.heanet.ie (ftp.heanet.ie)|87.44.34.235|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 8572108800 (8.0G) [application/octet-stream]
Saving to: ‘CentOS-Stream-x86_64-dvd1.iso’
CentOS-Stream-x86_64-dvd1.iso  1%[+      ] 100.83M  1.77MB/s    eta 73m 43s
^C

At any time you can press Ctlr+C to stop the process. A file with unfinished download will be left in the current directory:

greys@redhat:/dist/iso $ ls -ald *iso
-rw-rw-r--. 1 greys 118453636 Sep 30 14:37 CentOS-Stream-x86_64-dvd1.iso

And if you just re-run the same wget command, your download will actually restart:

greys@redhat:/dist/iso $ wget http://ftp.heanet.ie/pub/centos/8-stream/isos/x86_64/CentOS-Stream-x86_64-dvd1.iso
--2019-09-30 14:44:08--  http://ftp.heanet.ie/pub/centos/8-stream/isos/x86_64/CentOS-Stream-x86_64-dvd1.iso
Resolving ftp.heanet.ie (ftp.heanet.ie)… 87.44.34.235
Connecting to ftp.heanet.ie (ftp.heanet.ie)|87.44.34.235|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 8572108800 (8.0G) [application/octet-stream]
Saving to: ‘CentOS-Stream-x86_64-dvd1.iso.1CentOS-Stream-x86_64-dvd1.iso.1  0%[+      ]  24.88M   775KB/s    eta 2h 13m
^C

If you notice though, the download restarts with a different filename: CentOS-Stream-x86_64-dvd1.iso.1 in my case. So wget kept your previous download attempt and started a new one:

greys@redhat:/dist/iso $ ls -ald iso
-rw-rw-r--. 1 greys 118453636 Sep 30 14:37 CentOS-Stream-x86_64-dvd1.iso
-rw-rw-r--. 1 greys  26088163 Sep 30 14:44 CentOS-Stream-x86_64-dvd1.iso.1

How To Resume wget Download

Assuming we want to resume download, you need to use the wget -c option:

greys@redhat:/dist/iso $ wget -c http://ftp.heanet.ie/pub/centos/8-stream/isos/x86_64/CentOS-Stream-x86_64-dvd1.iso
--2019-09-30 14:45:27--  http://ftp.heanet.ie/pub/centos/8-stream/isos/x86_64/CentOS-Stream-x86_64-dvd1.iso
Resolving ftp.heanet.ie (ftp.heanet.ie)… 87.44.34.235
Connecting to ftp.heanet.ie (ftp.heanet.ie)|87.44.34.235|:80… connected.
HTTP request sent, awaiting response… 206 Partial Content
Length: 8572108800 (8.0G), 8453655164 (7.9G) remaining [application/octet-stream]
Saving to: ‘CentOS-Stream-x86_64-dvd1.iso’
CentOS-Stream-x86_64-dvd1.iso  1%[+      ] 120.68M  1.49MB/s    eta 91m 20s
^C

So it picked up the original 100MB or so downloaded file and resumed downloading from 100MB+ location.

If we stop this with Ctlr+C again, we’ll see the following:

greys@redhat:/dist/iso $ ls -ald iso
-rw-rw-r--. 1 greys 126668812 Sep 30 14:45 CentOS-Stream-x86_64-dvd1.iso
-rw-rw-r--. 1 greys  26088163 Sep 30 14:44 CentOS-Stream-x86_64-dvd1.iso.1
greys@redhat:/dist/iso $ du -sh iso
121M    CentOS-Stream-x86_64-dvd1.iso
25M    CentOS-Stream-x86_64-dvd1.iso.1

So yes, it’s clearly the first file that is slightly larger down – meaning if we continue downloading, it will start from about 126MB location:

greys@redhat:/dist/iso $ wget -c http://ftp.heanet.ie/pub/centos/8-stream/isos/x86_64/CentOS-Stream-x86_64-dvd1.iso
--2019-09-30 14:47:08--  http://ftp.heanet.ie/pub/centos/8-stream/isos/x86_64/CentOS-Stream-x86_64-dvd1.iso
Resolving ftp.heanet.ie (ftp.heanet.ie)… 87.44.34.235
Connecting to ftp.heanet.ie (ftp.heanet.ie)|87.44.34.235|:80… connected.
HTTP request sent, awaiting response… 206 Partial Content
Length: 8572108800 (8.0G), 8445439988 (7.9G) remaining [application/octet-stream]
Saving to: ‘CentOS-Stream-x86_64-dvd1.iso’

CentOS-Stream-x86_64-dvd1.iso  1%[+      ] 127.76M  1.63MB/s    eta 98m 41s

And that’s how you resume wget downloads and continue when previous wget download left off.

See Also




CentOS 8 and CentOS Stream Released

Great news, CentOS 8 is released now. Even better – there’s now a step in between Fedora and RHEL, called CentOS Stream.

Have you tried them yet? I’ll be upgrading to CentOS 8 this week and am also thinking of downloading and installing CentOS Stream in a KVM VM.

Let me know what you think!

See Also




Unix Tutorial for Russian Speakers

Unix Tutorial in Russian
www.UnixTutorial.RU

If you speak Russian, check out www.UnixTutorial.RU website – I only launched it recently so for now the focus is on the Unix Tutorial’s most popular pages and also the recent posts.

Если вы говорите по-руски, то у меня для вас сюрприз: несколько недель назад я запустил UnixTutorial.ru! Пока я перевожу самое популярное, но стараюсь и последние все заметки на основном сайте дублировать на русском языке.

Unix Tutorial RU

See Also




Attach Interface to Specific Firewall Zone in RHEL 8

RHEL 8

One of the first things I had to do on my recently built RHEL 8 PC was to move the primary network interface from public (default) zone to home zone – to make sure any firewall ports I open stay private enough.



How To List Which Zones and Interfaces are Active

Using the get-active-zones option of the firewall-cmd command, it’s possible to confirm where eno1 interface is at the moment. It’s already in the home zone cause I made the update earlier:

root@redhat:~ # firewall-cmd --get-active-zones
home
  interfaces: eno1
libvirt
  interfaces: virbr0

Attach Interface to a Firewall Zone

Here’s how one can move specified interface into a zone we want:

root@redhat:~ # firewall-cmd --zone=home --change-interface=eno1
success

Just to show how it works, I’m going to move eno1 into public zone and back to home one:

root@redhat:~ # firewall-cmd --zone=public --change-interface=eno1
success
root@redhat:~ # firewall-cmd --get-active-zones
libvirt
  interfaces: virbr0
public
  interfaces: eno1

Making Sure Firewall Changes Are Permanent

Don’t forget that after confirming a working firewall configuration, you need to re-run the same command with permanent option – this will update necessary files to make sure your firewall changes can survive a reboot:

root@redhat:~ # firewall-cmd --zone=home --change-interface=eno1 --permanent
The interface is under control of NetworkManager, setting zone to 'home'.
success

That’s it for today. Am really enjoying RHEL 8 configuration and still have this feeling I barely scratch the surface with all the new improvements this Red Hat Enterprise Linux brings.

See Also




Hello, World in podman

RHEL 8

Turns out it’s not that easy to install Docker CE in RHEL 8 just yet. Well, maybe there’s no immediate need since RHEL 8 comes with its own containerization stack based on podman?



Hello, World in podman

podman provides comprehensive compatibility with docker command, most non-Docker specific options are supported.

If you are familiar with docker command syntax, give it a try by just replacing docker with podman command. 

Let’s do the hello world exercise:

greys@redhat:~ $ podman run hello-world
Trying to pull registry.redhat.io/hello-world:latest…Failed
Trying to pull quay.io/hello-world:latest…Failed
Trying to pull docker.io/hello-world:latest…Getting image source signatures
Copying blob 1b930d010525: 977 B / 977 B [==================================] 0s
Copying config fce289e99eb9: 1.47 KiB / 1.47 KiB [==========================] 0s
Writing manifest to image destination
Storing signatures

Hello from Docker!

This message shows that your installation appears to be working correctly.
To try something more ambitious, you can run an Ubuntu container with:

$ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:

https://hub.docker.com/

For more examples and ideas, visit:

https://docs.docker.com/get-started/

As you can see, podman searches in Red Hat and Quay image repositories before moving on to Docker registry, but finally gets the hello-world image there.

Run Ubuntu image in podman

And if we want to follow Docker’s advice and try running the Ubuntu Docker image, we’ll replace

docker run -it ubuntu bash

with

podman run -it ubuntu bash

… It just works:

greys@redhat:~ $ podman run -it ubuntu bash
Trying to pull registry.redhat.io/ubuntu:latest…Failed
Trying to pull quay.io/ubuntu:latest…Failed
Trying to pull docker.io/ubuntu:latest…Getting image source signatures
Copying blob 5667fdb72017: 25.45 MiB / 25.45 MiB [==========================] 3s
Copying blob d83811f270d5: 34.53 KiB / 34.53 KiB [==========================] 3s
Copying blob ee671aafb583: 850 B / 850 B [==================================] 3s
Copying blob 7fc152dfb3a6: 163 B / 163 B [==================================] 3s
Copying config 2ca708c1c9cc: 3.33 KiB / 3.33 KiB [==========================] 0s
Writing manifest to image destination
Storing signatures
root@686f0d85b4ad:/# uname -a
Linux 686f0d85b4ad 4.18.0-80.el8.x86_64 #1 SMP Wed Mar 13 12:02:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
root@686f0d85b4ad:/# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"

I think it’s pretty cool. Will definitely read up and post more about podman and containerization in Red Hat in the following weeks.

See Also




Updating Index DB for locate command

Unix Tutorial

I sometimes forget there is a great locate command available in most distros, which is possibly the quickest way of finding a file or directory by their name. locate is super fast because it searches through index databases and not the filesystems directly. Here’s how you can update index DB for locate.

Why Do We Need Index Databases for Locate?

locate has one goal: find you files as quickly as possible. And while sometimes you need a real-time information, usually it’s enough if results of locate command are just recent, but not immediately fresh.

What this mean is that indexing of your filesystem can happen at any time, preferably daily, so that when you are looking for files with locate it has a fresh enough index of all the files in the available filesystems.

In other words, searching for files with locate command should NOT mean any of this:

  • indexing filesytems
  • updating indexes
  • searching on the filesystems directly

That’s why index databases for locate exist and that’s also why you do not have any delay with receiving results when you run locate.

Updating locate databases with updatedb

If your system doesn’t have a locate database, you would receive an error when using locate command:

greys@rhel8:~ $ locate sudoers
locate: can not stat () `/var/lib/mlocate/mlocate.db': No such file or directory

To fix this, simply regenerate the index database mlocate.db:

greys@rhel8:~ $ sudo updatedb

Should only take a few minutes, after that locate will start working:

greys@rhel8:~ $ locate sudoers
/etc/sudoers
/etc/sudoers.d
/usr/bin/cvtsudoers
/usr/libexec/sudo/sudoers.so
/usr/share/augeas/lenses/dist/sudoers.aug
/usr/share/doc/sudo/examples/sudoers
/usr/share/locale/ca/LC_MESSAGES/sudoers.mo
/usr/share/locale/cs/LC_MESSAGES/sudoers.mo
/usr/share/locale/da/LC_MESSAGES/sudoers.mo
/usr/share/locale/de/LC_MESSAGES/sudoers.mo

That’s it for today! Have fun!

See Also




Packt: Data Unlocked

Just realised there’s been something I wanted to mention in my Unix Tutorial digest, but it appears a time limited deal so it won’t wait until the next edition. Please note I’m not paid for sharing these news and won’t be earning any affiliate comission from the links below.

Packt: Data Unlocked

Seems the month of September is pretty busy at Packt Publishing: every week there are lots of great books and educational videos made available for just $10 each.

If you’re considering a career in Data Science or just looking for a really interesting direction to learn something new – have a look at their data science section and perhaps grab a book or two!

or just looking for a really interesting direction to learn something new – have a look at their data science section and perhaps grab a book or two!

Really cool: there’s also a free book of the day at Packt, I think – just created my account to download one. This isn’t data science specific – so every day there is a different book.

That’s it for today! Enjoy your weekend!

See Also




How To Show LUKS Passphrase Slots

Unix Tutorial

As you might know, it’s possible to add multiple passphrases to the same LUKS encrypted filesystem. This means each one of these will allow you to decrypt and to mount your filesystem.

Show LUKS Key Slots

This is how you can check how many passphrases or file-based keys are currently in use for managing access to your encrypted filesystem:

root@rhel8:~ # cryptsetup luksDump /dev/rhel/06
 LUKS header information for /dev/rhel/06
 Version:           1
 Cipher name:       aes
 Cipher mode:       xts-plain64
 Hash spec:         sha256
 Payload offset:    4096
 MK bits:           512
 MK digest:         85 84 e1 49 0f c3 7f df 9b fd 62 a0 50 c9 2a 14 a8 86 a8 0e
 MK salt:           58 e4 20 ed ac 01 56 cf cb b7 77 1d c6 11 20 6f
                    4c f2 fd 3a c5 c3 d4 f0 64 41 84 4c f9 58 a2 2f
 MK iterations:     87614
 UUID:              9144c0fe-7758-47ac-886b-330ae8bfa096
 Key Slot 0: ENABLED
     Iterations:             1418912
     Salt:                   65 fd 6c 78 84 7d 1b ae d1 42 13 0b f3 4c f7 41
                             9b 5f d5 e3 6e c5 a2 ce 05 28 02 f1 9b 56 07 b9
     Key material offset:    8
     AF stripes:             4000
 Key Slot 1: DISABLED
 Key Slot 2: DISABLED
 Key Slot 3: DISABLED
 Key Slot 4: DISABLED
 Key Slot 5: DISABLED
 Key Slot 6: DISABLED
 Key Slot 7: DISABLED

As you can see, only Key Slot 0 is ENABLED and reporting some setup details. This means I’m only using one passphrase for my encrypted filesytem.

If and when I add more passphrases or encryption keys, they will be occupying next available slot, starting with key slot 1.

See Also




Upgrading to CentOS 7.7

CentOS

CentOS 7.7 has just been released a few days ago and with any luck CentOS 8 will be released next week. I decided to quickly upgrade one of my dedicated servers from CentOS 7.6 to CentOS 7.7.

Confirm CentOS Version

Just a couple of steps to check what CentOS release we’re running:

root@s2:~ # cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)

There should also be a package reflecting CentOS release already installed on your system. This package will be upgraded along with the rest of the OS when we’re stepping up to CentOS 7.7:

root@s2:~ # rpm -qa | grep centos-release
 centos-release-7-6.1810.2.el7.centos.x86_64

Check Available CentOS Upgrades

yum command has the check-update option for verifying if any packages are available for updating:

root@s2:~ # yum check-update

This will return a rather long list. To be super-sure we’ll actually get the CentOS 7.7 upgrade, look for the same centos-release package:

root@s2:~ # yum check-update | grep centos-release
 centos-release.x86_64                    7-7.1908.0.el7.centos           base

Upgrade CentOS 7.6 to CentOS 7.7

We need the yum update command here. After you run it it will resolve dependencies and report something like this, prompting for your confirmation:

Transaction Summary
===============================================================
Install    3 Packages (+25 Dependent packages)
Upgrade  406 Packages
Total download size: 577 M
Is this ok [y/d/N]:

Sounds about right! We’ll press Y and let the server download and apply all the updates…

On my hosting it took about 1min to download all the packages!

Transaction Summary
===============================================================
Install    3 Packages (+25 Dependent packages)
Upgrade  406 Packages
Total download size: 577 M
Is this ok [y/d/N]:
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
 Updating   : libgcc-4.8.5-39.el7.x86_64                                                                                                       1/841
 Installing : urw-base35-fonts-common-20170801-10.el7.noarch                                                                                   2/841
 Updating   : 1:grub2-common-2.02-0.80.el7.centos.noarch                                                                                       3/841
 Updating   : centos-release-7-7.1908.0.el7.centos.x86_64                                                                                      4/841
 Updating   : langtable-0.0.31-4.el7.noarch                                                                                                    5/841
 Updating   : libreport-filesystem-2.1.11-43.el7.centos.x86_64                                                                                 6/841
...
yum-plugin-fastestmirror.noarch 0:1.1.31-52.el7                           yum-utils.noarch 0:1.1.31-52.el7
 Replaced:
   urw-fonts.noarch 0:2.4-16.el7
 Complete!

It took less than 10 min to apply all the package updates, so the only things left are to capture current kernel version before and after the reboot:

root@s2:/ # uname -a
Linux s2 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Now let’s reboot:

root@s2:/ # shutdown -r now

… and confirm that we’re running newer Linux Kernel and sporting the CentOS 7.7 release now:

greys@s2:~ $ uname -a
Linux s2 3.10.0-1062.1.1.el7.x86_64 #1 SMP Fri Sep 13 22:55:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Hmmm. This does look a bit conservative! It’s 3.10 branch of Linux kernel whereby desktop releases are sporting Linux Kernel 5.x already.

Anyway, that’s server s2 upgraded to CentOS 7.7 successfully!

We’re certainly running the CentOS 7.7 release now:

greys@s2:~ $ cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)

See Also




Install Brave Browser with Homebrew

Brave Browser

Part of my dotfiles strategy is to automate most of initial setup in all of my Linux and Unix environments. Although most attention goes towards dedicated and virtual servers, I do explore options for automating my primary macOS workstation – and that’s where Homebrew just keeps on helping.

Install Brave browser in macOS

Yes, the easiest is to download package from the official Brave website. There’s an auto-update functionality there too, to be fair. But regular updates can’t be easily automated. That’s why I think brew install of Brave is a better solution – plus you can update quite a few software packages all at once at regular intervals if you manage them via brew.

IMPORTANT: you will need to remove the previous Brave browser installation, otherwise brew will give you an error:

Remove previous Brave installation before installing with Homebrew

Once that is done, simply run:

$ brew cask install brave-browser

Here is how it will look, only takes a few seconds:

Install Brave browser via brew cask

That’s it! We’ve got the latest version of Brave installed:

Give Brave browser a try – it’s fast and privacy friendly, which is a great combination. Think Brave Ads are coming to Europe and Brave Sync is something that’s constantly being improved between mobile and desktop devices – I keep coming back to this browser to watch its progress.

See Also