This week’s Unix Tutorial Project is super geeky and fun: I’m setting up text-based email archive system using Mutt (NeoMutt, actually), OfflineIMAP and hopefully NotMuch. Will publish a project summary on the weekend.
Why use OfflineIMAP
OfflineIMAP tool is an open-source tool for downloading your email messages and storing them locally in a Maildir format (meaning each email message is stored in a separate file, each folder/GMail tag is a separate directory).
As the name suggests, this tool’s primary objective is to let you read your emails offline. Contrary to the other part of the name, offlineimap is NOT an IMAP server implementation.
I’d like to explore OfflineIMAP/Neomutt setup as a backup/archive solution for my cloud email accounts. I used to be with Fastmail but switched to gSuite email last year. I think it’s very important to keep local copies of any information you have in any cloud – no matter how big/reliable the service provider is, there are many scenarios where your data could be completely lost, and responsibility for keeping local backups is always with you.
Both gMail and Fastmail solutions are perfect for web browser use but any local email software is invariably bulkier and slower compared to web interface. I’m not giving up on finding the acceptably performance and reliable solution though.
This is one of the most recent attempts to download all emails and to have them easily searchable on my local PCs and laptops.
OfflineIMAP Configuration Steps
I’m only learning this tool, so this is probably the most basic usage:
- Confirm your mail server details (IMAP)
- Confirm your mailbox credentials (for Google, gSuite and even Fastmail you need to generate an app password – it’s separate and different from your primary email password)
- Create .offlineimaprc file in your home directory as shown below
- If necessary, create credentials file (for now – with cleartext app password for email access) – mine is /home/greys/.creds/techstack.pass
- Run offlineimap (first time and every time you want your email refreshed)
My .offlineimaprc file
Here’s what I have in my .offlineimaprc file for this experiment:
[general]
ui = ttyui
accounts = techstack
[Account techstack]
localrepository = techstack-local
remoterepository = techstack-remote
[Repository techstack-local]
type = Maildir
localfolders = ~/Mail/techstack/
[Repository techstack-remote]
type = Gmail
remoteuser = [email protected]
remotepassfile = ~/.creds/techstack.pass
maxconnections = 5
ssl = yes
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
folderfilder = lambda foldername: foldername not in ['Archive']
expunge = no
You can have multiple accounts in this one config file, they’ll be listed in the accounts section (accounts = techstack, unixtutorial would mean 2 accounts: techstack one and one for my Unix Tutorial email).
localfolders parameter specifies that I want OfflineIMAP to create a Mail directory in my homedir (so ) and then techstack subdirectory there – meaning you can have account subidrectories there like /home/greys/Mail/techstack and /home/greys/Mail/personal, etc.
You define two repositories, local and remote one. The task of OfflineIMAP is to sync the two.
IMPORTANT: The really important parameter is maxconnections one. Default is 3 and I’ve changed it to 5 for quicker email sync. Setting it to a higher value resulted in failures – probably because Google servers rate limit my connection.
CRITICAL: expunge parameter is set to yes by default, so you must set it to no if your plan is to keep emails on the mail server after you sync them. By default they will be removed from the server as soon as they are downloaded, meaning Gmail app won’t see any messages. Once deleted, it will be rather tricky to restore all the emails – so it’s important to get this setting right from the very start. Since my primary usage is still web and Gmail app based, I certainly want all my emails to stay in Google cloud even after I download them using OfflineIMAP – that’s why I configured it as expunge = no.
As you can see, this config references the /home/greys/.creds/techstack.pass file. This file has an clear-text application password I generated for my email address in gSuite admin panel. My understanding is that this can be improved, so I’ll do a follow-up post later.
How To Use OfflineIMAP
Simply run the offlineimap command and you should see something like this:
greys@xps:~ $ offlineimap
OfflineIMAP 7.2.2
Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
imaplib2 v2.57 (system), Python v2.7.16, OpenSSL 1.1.1b 26 Feb 2019
Account sync techstack:
*** Processing account techstack
Establishing connection to imap.gmail.com:993 (techstack-remote)
Folder 2016 [acc: techstack]:
Syncing 2016: Gmail -> Maildir
Folder 2016/01-January [acc: techstack]:
Syncing 2016/01-January: Gmail -> Maildir
Folder 2016/02-February [acc: techstack]:
Syncing 2016/02-February: Gmail -> Maildir
Folder 2016/01-January [acc: techstack]:
As you can see, it processes account techstack, connects to gmail and starts processing remote folders (gmail tags) like 2016, 2016/01-January, 2016-02-February etc – these are the tags I have in my gSuite account.
Initial download would take a while. My 150K messages took almost 3 days to download.
That’s all for today, hope you give OfflineIMAP a try!
See Also
