Dovecot Local Email Server

Local Server: Dynamic DNS

I assume you know how to set up a Linux box. I used my main tower, which I had recently upgraded from Windows 10 to Linux Mint and fresh disks^[3]. First, we need to put our server online, ultimately so that we will be able to access our server from anywhere to work with our emails.

Residential ISPs typically give a router, usually not at a static IP (nor is such necessarily desirable). However, we do need a consistent address to point our email clients at. The solution is dynamic DNS. I use freemyip.com, though there are many equivalent services, and your router may even have one built in! The gist is as follows: first, your computer authenticates to ⟨identifier⟩.freemyip.com. Subsequently (after DNS propagation), traffic to ⟨identifier⟩.freemyip.com will know to go to the IP address you authenticated from. Your IP address change? No problem! Simply reauthenticate and the address is updated.

This is actually very simple. Simply pick some ⟨identifier⟩ and register it on freemyip.com. You will get back an associated ⟨token⟩. This is basically your password. Now, to make the DNS binding, you simply navigate to the URL https://freemyip.com/update?token=⟨token⟩&domain=⟨identifier⟩.freemyip.com^[4]. You should get back a message "OK"^[5]. And that's it!

We want this to run periodically so that if the ISP changes our router's IP address, our identifier updates to point at the new address. I simply put it in a cron job, which executes every 20 minutes, by running:

(crontab -l;echo "*/20 * * * * curl \"https://freemyip.com/update?token=⟨token⟩&domain=⟨identifier⟩.freemyip.com\" > /dev/null 2>&1")|crontab -

(You can alternately run crontab -e to edit manually. Here's a guide to the syntax.)

Local Server: NAT Forwarding

A residential router also almost always implements a NAT. The important point is that the external IP address we've just set up is actually our router's IP. There's no way for incoming traffic (in particular, our email clients' traffic) to know it's supposed to go to our computer. We must "open a port", basically meaning incoming traffic on that port gets routed to our server.

You want to open a port for HTTP (for SSL verification in the next sections) and for IMAPS (for accessing our email via clients). The instructions will vary by router, but basically:

1. Log into the router from a computer inside the network (point your browser at somewhere like "192.168.1.1"^[6] and use the credentials your ISP hopefully gave you).
2. Find where it says something like "Address Reservation" or "Static Routing"^[7], and add a static reservation of your server's local IP to its MAC address. This ensures our server always gets a specific local IP.
3. Find where it says something like "NAT Forwarding" or "Port Forwarding"^[8]. Add entries for TCP ports 80 and 1993^[9] that go to the local IP. This says that the specific local IP (i.e. our server) should handle incoming traffic on these ports.

Local Server: Setup Firewall

We should have a firewall since we're putting the server on the internet. Ubuntu (and the Linux Mint I'm actually using) come with one, ufw, already. In addition to opening the ports in the router, we must open the ports in the server's firewall:

#Needed for Let's Encrypt, if nothing else
sudo ufw allow from any to any port 80 proto tcp

#For IMAPS
sudo ufw allow from any to any port 1993 proto tcp

#Enable firewall and show status
sudo ufw enable && sudo ufw status

Note that you can delete rules from the firewall by writing the rule again but with delete allow instead of allow, and also disable disables the entire firewall in case you ever want to roll back for some reason^[10][11].

Local Server: Setup SSL

Now we need to set up the ability to connect securely over SSL. The key technology is Let's Encrypt.

Install Let's Encrypt and make certificates for our "freemyip.com" subdomain:

sudo apt install certbot

#Run the following, answering the simple questions as prompted.
sudo certbot   certonly   --standalone   -d ⟨identifier⟩.freemyip.com

#This will create files like:
#	"/etc/letsencrypt/live/⟨identifier⟩.freemyip.com/fullchain.pem"
#	"/etc/letsencrypt/live/⟨identifier⟩.freemyip.com/privkey.pem"

(Note that any pre-existing stuff listening on port 80 may cause this to fail, and you will need to disable it. The main thing that you might have running is an HTTP server; you should instead run an HTTPS server certified by this certificate!)

The setup will also set up an automated job to renew the certificate automatically (see systemctl list-timers | grep certbot). You can test cert renewal in advance with:

sudo certbot renew --dry-run

Note that you should use --dry-run when testing; Let's Encrypt places severe rate-limits on the real thing for some reason; getting locked out can be annoying.

Local Server: Testing Connection

You can use e.g. Nmap to test port 1993 (you might have to install it):

#Should return "1993/tcp open  snmp-tcp-port" at this point
nmap -p 1993 ⟨identifier⟩.freemyip.com | grep 1993

You should be able to run this from inside the router network. The term for that is 'NAT hairpin', because the traffic goes to the router, but then instead of going out to the internet, because the router is the destination, it immediately "turns around" and goes through the router to our server.

You should now be able to test SSL! Run e.g. the following (most usefully from another device):

#(Exit with CTRL+C, add `-brief` to taste)
openssl s_client -connect ⟨identifier⟩.freemyip.com:1993

If something goes wrong . . . uh, good luck! Most likely it'd be a problem with the preceding steps. Ensuring they produce the expected results should be the first thing.

Local Server: Dovecot Config File

The following installs the Dovecot modules we will need:

#There's also `dovecot-pop3d`, but this shouldn't be necessary
sudo apt install dovecot-core dovecot-imapd

We now must create a configuration file. Dovecot ships with a large modularized bunch of config files, which I found excessive and complicated. Instead, the following suffices.

First, create a backup of the original config.

sudo cp /etc/dovecot/dovecot.conf /etc/dovecot/dovecot.conf.original #backup

Then, edit "/etc/dovecot/dovecot.conf":

code /etc/dovecot #VSCode; then make file "dovecot.conf"
#sudo nano /etc/dovecot/dovecot.conf #Nano
#(etc.)

Write the following into "/etc/dovecot/dovecot.conf"^[12]:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
#https://doc.dovecot.org/2.3/settings/core/

#dovecot_config_version  = 2.3.21
#dovecot_storage_version = 2.3.21

protocols = imap #(note includes IMAPS)

service imap-login {
	inet_listener imap {
		port = 0 #disable IMAP (to force IMAPS)
	}
	inet_listener imaps {
		port = 1993 #Note the port from above
		ssl = yes
	}
}

mail_location = maildir:⟨emaildir⟩/%u-Maildir

passdb {
	driver = passwd-file
	args = /etc/dovecot/passwd
}
userdb {
	driver = static
	args = uid=⟨sysuser⟩ gid=⟨sysuser⟩ home=⟨emaildir⟩/%u
}

#Namespaces
#	https://doc.dovecot.org/2.3/configuration_manual/namespace/#core_setting-namespace
namespace inbox {
	#location = #set from default above
	#separator = . #default
	inbox = yes
	#prefix = "" #default
	#type = private #default

	#Note Thunderbird wants to call it "Archives"
	mailbox "Archives" {
		auto = subscribe
		special_use = \Archive
	}
	mailbox "Drafts" {
		auto = subscribe
		special_use = \Drafts
	}
	mailbox "Sent" {
		auto = subscribe
		special_use = \Sent
	}
	mailbox "Junk" {
		auto = subscribe
		special_use = \Junk
	}
	mailbox "Trash" {
		auto = subscribe
		special_use = \Trash
	}
}

ssl = yes
ssl_cert = </etc/letsencrypt/live/⟨identifier⟩.freemyip.com/fullchain.pem
ssl_key  = </etc/letsencrypt/live/⟨identifier⟩.freemyip.com/privkey.pem

The code is simple, but unfortunately still not self-evident.

# indicates comments; in addition to some light comments, I've included some disabled directives that don't need to be set but that I thought may be helpful or informative (look at the original config we made a backup of for way more of that!).

We use protocols = imap because our email clients will connect via IMAPS (which Dovecot also includes under imap). The service imap-login block tells it to listen on port 1993 and use SSL (which is configured with ssl = yes at the bottom; notice we are using the cert generated from Let's Encrypt, above).

The namespace block defines what each Dovecot user account has. You can adjust this to taste, but something like this seems to be common. The default separator = . directive should almost certainly be left alone; while a slash may seem more intuitive, it will muck up a lot of stuff, causing lots of trouble for negligible gain.

But what even is a Dovecot user (userdb and passdb)? Dovecot is very vague about this, but basically a Dovecot user is its own thing, completely separate from other types of users—and, in particular, they are separate from the system user(s). That said, confusingly, the commonest way everyone sets Dovecot up is to make the Dovecot users be 1:1 with system users, so that you can log into your Dovecot user email with your system user credentials^[13]!

We DO NOT do this, and in my view it is a bad practice, especially for a personal server. Although it superficially makes some sense, it plays very badly with multiple email addresses: clients group the addresses together into folders instead of identities, and one would need to have a system login to the email server for each email identity, which I definitely didn't want (and don't know why anyone else would, either).

Indeed, what we want instead is one Dovecot user for each email address. The userdb and passdb records above set this up. (Note that because we're no longer relying on mirroring the existing system user(s), we will have to define the Dovecot users ourselves; we'll do this in the next section.)

Each user's mail goes in the mail_location; you can configure this however you like. The %u bit will fill in the Dovecot username, which I chose to be the email address. (Making the username be the email address is slightly unusual, but Dovecot specifically supports this, and I find it makes the accounts very clear; note that whatever you choose should be filesystem-safe.) For my particular case, I have it like so:

mail_location = maildir:/home/agatha/Documents/email/%u-Maildir

Most guides also set up a separate system user ("vmail") for Dovecot to run as. This makes sense for a server for many people, but since I'm just me, I didn't see much point to this additional complexity. Wherefore, I specify in userdb to simply run as my own system user, "agatha", instead.

Local Server: Dovecot Users

Since we don't do the weird conflation of system and Dovecot users mentioned above, we must actually define the Dovecot users. This is simply creating the file "/etc/dovecot/passwd" (see docs), which the conf refers to above. Inside, we put a record for each email address. These are the credentials the email client will log in with.

An example "/etc/dovecot/passwd" file looks like this:

foo@domain.com:{SHA256-CRYPT}$5$/jtYAHeszMHTMDzT$PddmKGj/KYy9MSFdmXiejwgrsjAxMuc5nS8Mb7.nWJ1::::::
bar@domain.com:{SHA256-CRYPT}$5$CgxpF.TFjAP1G7Sp$DCooUroAvG1UgaSX8GqQuoBrt6KStsnVXQB2ardep90::::::
qux@gmail.com:{SHA256-CRYPT}$5$P16Dmc1aQ64xFCrT$5RglSc72Ag8JpZ4iKOQUM37ENk0Ieah1rl2SG4dHs20::::::

The file is a passwd file (syntax guide), and for our simple use-case, we can generate it by hand.

Each record has the Dovecot username, which as above I've chosen to be the email address itself.

The passwords are encoded with a cryptographically secure hashed format^[14], preventing compromise even if the file itself is breached.

To generate these records, Dovecot provides a tool:

#Generate password record
doveadm pw -s sha256-crypt

Run this, entering in a good password (in the example file above, the passwords are all "abc" for instructional purposes^[15], but in actuality I use passwords that are themselves generated with a cryptographically secure RNG, and you should too)! The output of the command above will be a record you can copy into the "/etc/dovecot/passwd" file.

Many other password schemes besides SHA-256 are possible. Check out the list. The simplest scheme is plaintext, no hashing at all (e.g. {PLAIN}abc)—but then your email is compromised if anyone gets hold of this file, so please don't actually do that in production!

Local Server: Starting Dovecot!

Finally, we can start our email server!

#Enable and (re)start Dovecot email server service!
sudo systemctl enable dovecot
sudo systemctl restart dovecot || systemctl status dovecot.service

##Stop and disable it, for reference
#sudo systemctl stop dovecot
#sudo systemctl disable dovecot

Here are some sanity checks:

#Print the mailboxes/folders for Dovecot user "foo@domain.com":
sudo doveadm mailbox list -u foo@domain.com

#Print messages for Dovecot user "foo@domain.com" (may print a lot!):
sudo doveadm fetch   -u foo@domain.com   "text"   MAILBOX INBOX

Local Server: Setup getmail

Our server may be running, but we don't actually have a way to get email. For this, we periodically poll our public email inboxes from the remote hosting providers, and move (not copy) any emails they find onto our local server.

The standard tool for this is getmail (see also docs FAQ and tutorial). It wants its config and cache files to be in a directory named ".getmail/", so we make that too as part of the install (it can be anywhere, but I put it with the rest of the email stuff in ⟨emaildir⟩):

sudo apt install getmail6

mkdir -p ⟨emaildir⟩/.getmail

We then have to make "getmailrc" files in this directory. These configure how getmail operates. For example, for a sensible mailbox like "foo@domain.com", we might have a file "⟨emaildir⟩/.getmail/getmailrc-foo@domain.com" with contents:

[options]
delete = true

[retriever]
type = SimplePOP3SSLRetriever
server = mail.domain.com
username = foo@domain.com
password = ⟨password⟩

[destination]
type = Maildir
path = ⟨emaildir⟩/foo@domain.com-Maildir/

This tells getmail to log in to "mail.domain.com" over SSL with the given credentials, which you will know from your hosting provider. Any emails in the inbox are copied to "⟨emaildir⟩/foo@domain.com-Maildir/" using the POP3 protocol and deleted^[16] from the remote server.

Note: GMail is unfortunately not a sensible inbox. Recently, they disabled this sort of login^[17], bogusly labeling^[18] it "less secure". To get around this, you must (1) enable 2FA on your Google account (go to myaccount.google.com/signinoptions/twosv), and then (2) create an "app password" for GMail (go to myaccount.google.com/apppasswords). This will generate a password of sixteen letters, grouped into four quartets. You put this into a getmailrc file like so:

[options]
delete = true

[retriever]
type = SimplePOP3SSLRetriever
server = pop.gmail.com
username = qux@gmail.com
password = ⟨four-quartets-password⟩
port = 995

[destination]
type = Maildir
path = ⟨emaildir⟩/qux@gmail.com-Maildir/

You also must configure GMail. Under the GMail's "Settings" -> "Forwarding and POP/IMAP" tab, section "POP download", enable POP for all mail and select delete GMail's copy.

After creating a getmailrc file for each email, we can run getmail to grab our email!

At first, we can run it manually, for testing, like so:

#Fetch mail from the configured remote inbox and move to local directory.
#	Note: this WILL move your emails!  Only run it when ready.
getmail   --getmaildir ⟨emaildir⟩/.getmail   --rcfile getmailrc-foo@domain.com

We do not actually want to have to run this manually in production though! Instead, I set up cron jobs to do it by running the following (again, or run crontab -e to edit manually according to the syntax):

(crontab -l;echo "*/1 * * * * getmail --getmaildir ⟨emaildir⟩/.getmail --rcfile getmailrc-foo@domain.com")|crontab -
(crontab -l;echo "*/10 * * * * getmail --getmaildir ⟨emaildir⟩/.getmail --rcfile getmailrc-bar@domain.com")|crontab -
(crontab -l;echo "*/5 * * * * getmail --getmaildir ⟨emaildir⟩/.getmail --rcfile getmailrc-qux@gmail.com")|crontab -
crontab -l

So, fresh email will be pulled for "foo@domain.com" every minute, for "bar@domain.com" every 10 minutes, and for "qux@gmail.com" every 5 minutes.

Local Server: Optional RSync Backup

For backups, I use SyncThing (based on rsync)^[19].

Simply ensure the mail_location from the config is a (sub)directory managed by SyncThing, and let your other computer(s) know about it. Note that this is only a protection against loss. These backed up files should not be used by other software on these other computers—and especially not by other Dovecot instances!

Note that if you ever get into a situation where you are restoring from a SyncThing backup, the directory permissions matter. SyncThing does not by default synchronize all permissions, due to cost, and Dovecot will be irritated with you until you fix it. You want everything in the Maildir to be owned by your system user (remember, we chose to run the service as ourselves, above), with all folders having permission "drwx------" and all files having permission "-rw-------".

Client Devices: Set up Email Client

We now need to connect to our server with an email client so we can interact with our email. Thunderbird seemed like a reasonable choice, so that's what I'm using for now (read: the following directions are intended to be generic, but are written while under Thunderbird's influence).

For every device you want to check email on, install the email client. Then, for every email, create an email account or identity and proceed as follows.

Incoming mail should be configured with protocol IMAPS (or IMAP with SSL/TLS) connecting to our server at its public DNS name "⟨identifier⟩.freenode.com" on port 1993. Authentication is normal password and the username is the Dovecot user's username for that email address (e.g. "foo@domain.com"). The password is the password for the Dovecot user you made above and put in the "/etc/dovecot/passwd" file.

Outgoing mail should be configured to the professionally hosted mailboxes' SMTP servers. For example "mail.domain.com" port 465. Use SSL/TLS with normal password, and username the username on the professionally hosted server (which may also be e.g. "foo@domain.com"). For GMail, instead use authentication method OAuth2 (the server is "smtp.gmail.com").

One incredibly dumb issue I had was I told Thunderbird to remember SMTP credentials, and the credentials turned out to be incorrect. It will never ask you for a password again (and won't show it in the account's settings, which I consider a soft UI bug). If you get in this situation, go to "Settings" -> "Privacy & Security", section "Passwords", and edit "Saved Passwords". Delete the offending entries, and it will prompt you for them again.

Also note you probably want to avoid using local folders in your client. They are, after all, local, and you won't be able to access their contents from other devices. Just make folder(s) under your email accounts.

Final Test!

To test that you can receive email, I recommend sendtestemail.com.

To test that you can both send and receive email, I recommend ismyemailworking.com.

To check that your professionally hosted domain's SMTP server is doing something reasonable, try MxToolbox.

Appendix: Import "mbox" Format

You may wish to import existing emails from "mbox" format (e.g. from an old usage of Thunderbird) onto the local server. The mbox format is crufty and bad, but fortunately Dovecot provides a tool to help. The basic command is (see also):

doveadm import   -u ⟨dst-dovecot-user⟩   mbox:⟨path-to-mbox-file⟩ ⟨mailbox-folder-name⟩

The following script prints out the commands to run, manually, to accomplish the import (from a directory "⟨emaildir⟩/tbird_old" containing a mbox file^[20]). Be sure to fill in the ⟨⋯⟩ variables!

TMP_MBOX=$(mktemp -d)
echo $TMP_MBOX

find ⟨emaildir⟩/tbird_old -type f ! -name '*.msf' ! -name '*.dat' ! -name '*.html' | while read f; do
	folder="tbird_new/${f#⟨emaildir⟩/tbird_old/}" #replace prefix
	folder_safe="${folder//\//_}"  #convert '/' → '_', not supported otherwise
	printf "cp %q %q/data.mbox   &&   sudo doveadm import -u ⟨dst-dovecot-user⟩ mbox:%q/ %q ALL\n" "$f" "$TMP_MBOX" "$TMP_MBOX" "$folder_safe"
done

#[run each printed command manually here!]

rm -r $TMP_MBOX

If you have a lot of mbox files and you're feeling confident, here's the same script but actually running the commands automatically:

TMP_MBOX=$(mktemp -d)
echo $TMP_MBOX

find ⟨emaildir⟩/tbird_old -type f ! -name '*.msf' ! -name '*.dat' ! -name '*.html' | while read f; do
	folder="tbird_new/${f#⟨emaildir⟩/tbird_old/}" #replace prefix
	folder_safe="${folder//\//_}"  #convert '/' → '_', not supported otherwise
	echo "Importing \"${f}\" -> \"${folder_safe}.data_mbox\""
	cp "${f}" $TMP_MBOX/data.mbox
	sudo doveadm import -u ⟨dst-dovecot-user⟩ mbox:$TMP_MBOX/ "$folder_safe" ALL
done

rm -r $TMP_MBOX

For at least one mailbox, I got an error:

Error: Mailbox data.mbox: Sync failed for mbox: seq=4 uid=16 uid_broken=0 originally needed 0 bytes, now needs 18446744073709551615 bytes

This basically means the mbox file was corrupt. Try looking inside it and seeing if there's anything worth saving.

Thunderbird also seems to do sent messages weirdly. You can export the "Sent" mailbox into an mbox file and then import it like so:

TMP_MBOX=$(mktemp -d)
echo $TMP_MBOX

formail -s < ⟨emaildir⟩/tbird_old/⟨⋯⟩/Sent > $TMP_MBOX/data.mbox

sudo doveadm import -u ⟨dst-dovecot-user⟩ mbox:$TMP_MBOX/ ⟨mailbox-folder-name⟩ ALL

rm -r $TMP_MBOX

Appendix: Import "Maildir" Format

The Maildir format is used internally by our server and is generally technically superior. We can import a Maildir pretty much by just copy-pasting it into our actual Maildir as a subdirectory. The "right" way to do it though is again offered by Dovecot:

doveadm import -U ⟨src-dovecot-user⟩ -u ⟨dst-dovecot-user⟩ 'maildir:⟨⋯⟩/Maildir' Imported ALL

Notes