Our local cable TV and broadband provider, Spectrum, in their infinite wisdom, appears to have blocked the entire IP range owned by Digital Ocean (and possibly other similar hosting providers) from sending mail to their email users. I discovered this a month or two ago when mail from my scout troop’s emails just stopped going to any of our families that use @rochester.rr.com addresses. Of course this is just speculation because Spectrum has not acknowledged any of my many, many emails requesting unblocking, and trying to get help from customer service is a painful experience of being bounced between help desks of techs that are trained to handle front-end issues and who have no idea what to do with back-end questions. I suspect the block is more widespread than just Digital Ocean, too, as another email account I use, this one hosted by a much smaller hosting company, also seems no longer to be able to communicate with my Spectrum account.
Since I can’t seem to get anywhere with Spectrum, I started thinking about alternative solutions. The mail server in question runs on a Digital Ocean droplet running VestaCP, which uses Exim 4 as it’s MTA. A little research into Exim configurations showed that I could set up a “smart host,” basically a process for relaying all outgoing mail matching certain criteria to an external mail server. “Perfect,” I thought, “I’ll just set up a new Spectrum email and relay all of the mail bound for Spectrum users through that!” For the most part, that worked—the only issue was authentication. Exim already had two authentication schemes set up to authenticate email clients when they’d try to send email and these conflicted with the configuration I needed for Exim to authenticate to Spectrum’s SMTP server.
Not to be outdone, I quickly had a new idea. Exim supports a construct called a pipe, based on the Unix construct of the same name, where the contents of an email message are passed on to an external program for further processing. If I could just find some utility that could take the message contents from Exim and pass them on to an authenticated SMTP session with Spectrum, I’d be all set. After a couple of hours of comparing Linux command line tools for sending mail I came up with nothing that would do exactly what I wanted. Maybe I’d have to write something myself.
As I was researching SMTP client libraries for PHP, hoping not to have to write my own, I came across a recommendation to reconfigure PHP to use msmtp, instead of sendmail, with the mail() function. “msmtp—-what’s that?” Turns out that it’s an SMTP client that implements the sendmail command-line interface. Unlike Exim however, which also implements the sendmail interface, msmtp won’t try to route mail itself; it will only send it through one of it’s preconfigured SMTP servers. In other words, exactly what I needed.
I should note that I happen to be a Spectrum customer, so I was able to create a new email account with Spectrum to do this. If you don’t have that luxury, you can probably use a different provider, such as Gmail (there are numerous examples to using Gmail with msmtp available online). Note, though, that doing this could cause problems for you with SPF, DKIM, and DMARC validations, so plan accordingly. When using a Spectrum account, since the connection is authenticated, Spectrum seems to accept the message without doing any further verifications.
Installing msmtp on my CentOS server was easy since there’s already a package available. The package is in the epel repo, however, so if you don’t have epel configured already, you’ll need to do that first. You’ll also need to be root to do this.
yum install -y msmtp
Thinking about how best to set it up, I decided to create a new service account to handle mail being passed to msmtp. While I could have used a global configuration that would not have required this, it would require me to make a file containing the password for the mail relay readable to everyone on the server. In reality, I’m the only user on the server, but still, that’s not a very good security decision. With the separate user account, I can better protect the msmtp configuration.
I created the user account with the following command. The -r sets it up as a service account with no password, but the -m and -d set up a home directory for the account, which is not normally done for service accounts. I’m using this home directory to store the configuration and scripts needed to make this thing work.
useradd -rmd /usr/lib/msmtp msmtp
msmtp wants to write to a log file, so I’ve also set up a directory for that. I’m putting it into /var/log so that it would be easy to find when I need it and also to be close to the logs for other processes, such as Exim, that I may need to consult concurrently when doing email debugging.
mkdir /var/log/msmtp chmod 750 /var/log/msmtp chown msmtp.msmtp /var/log/msmtp
Next, I switch to the newly created msmtp account to set up my configuration.
su - msmtp
Here I create the configuration file, When it starts up, msmtp looks for a file named .msmtprc in the user’s home directory.
vi .msmtprc
The contents of the file should look like the following sample. Note that it is possible to set up multiple accounts in the file; just start each one with a new “account” line and a unique name. The indentation in the file is my own—I found it easier to follow what was what when I used indentation, but it is not required.
defaults
tls on
tls_starttls on
tls_trust_file /etc/ssl/certs/ca-bundle.crt
logfile /var/log/msmtp/msmtp.log
account spectrum
host mail.twc.com
port 587
auth on
user EMAIL_ADDRESS
password PASSWORD
from EMAIL_ADDRESS
OK…so what’s going on here? The lines following the “defaults” section apply to all accounts configured in the file. Here I’m saying that I want to use TLS encryption on all connections and I provide the path to the CentOS default certificate store that msmtp will use to validate the certificate that servers present when a connection is made. I also define the path to the log file that msmtp should use to record it’s activities.
The next section defines an account. I’ve given the account the name “spectrum,” which will be used when calling msmtp a bit later. The rest is fairly straightforward: SMTP server host and port, a directive to use an authenticated connection along with the credentials to use, and the default “from” address for the so-called SMTP envelope, the initial communication between the client and server before the message is actually sent. Obviously, you should replace EMAIL_ADDRESS and PASSWORD with your actual credentials.
For security reasons, msmtp checks that the .msmtp file is only readable and writeable by the owner and won’t let you use the program if it is not.
chmod 600 .msmtprc
msmtp is now set up and ready to use. We can test it by creating a sample email message in a file and trying to send it.
vi email.txt
The contents of this file should look something like this:
From: me@mydomain.com To: you@yourdomain.com Subject: This is a test Testing mail from msmtp.
Note that the headers in this file, such as the “From” and “To” email addresses, only reflect what is displayed in the recipient’s email client when they receive the message. The actual recipients are specified in the SMTP Envelope, which is generated from parameters passed to msmtp on the command line or specified in the .msmtprc file.
To test msmtp, run the following, replacing EMAIL_ADDRESS with the email address of your desired recipient.
cat email.txt | msmtp -a spectrum me@rochester.rr.com
msmtp won’t show any output if it’s successful, but you can check the log file to make sure your message went through.
tail /var/log/msmtp/msmtp.log
If it worked, you should see something like this:
Nov 29 10:34:03 host=mail.twc.com tls=on auth=on user=YOUR_EMAIL from=YOUR_EMAIL recipients=RECIPIENT_EMAILS mailsize=7959 smtpstatus=250 smtpmsg='250 2.0.0 MESSAGE_ID mail accepted for delivery' exitcode=EX_OK
Since msmtp emulates sendmail, I had figured that I’d be able to simply include the call to smtp in my Exim transport, but I found that Exim includes an extra newline at the beginning of the message that causes the email headers to get pushed into the message body. To get around this, I use a short shell script that uses the read command to strip off the extra line.
vi route-spectrum
Add the following to this file:
#!/bin/bash CONFIG_FILE=/usr/lib/msmtp/.msmtprc ACCOUNT=spectrum #strip leading newline from messaage passed from exim read # pass remainder of stdin (via cat) to msmtp to send to remote MTA # additional arguments from Exim are passed in $@ cat | msmtp -C $CONFIG_FILE -a $ACCOUNT $@
I’m not sure that Exim creates a full shell environment for the script when it runs, so I added the full path to the .msmtprc file I wanted to use on the command line, via the -C option. Be sure to adjust the $ACCOUNT variable to reflect the name you gave to the server configuration in the .msmtprc file.
Since it’s a script, it needs to be executable.
chmod 755 route-spectrum
Now I’m done working in the msmtp account, so let’s go back to the root account.
exit
Now I need to build the Exim configuration. My server uses a single file for the Exim config, so this tutorial reflects that. YMMV.
vi /etc/exim/exim.conf
In the section that starts with begin routers I add the following. This tells Exim to give special attention to messages bound for the rochester.rr.com domain (adjust this as necessary for your use, something like “rr.com : *.rr.com” might be more inclusive).
spectrum_mail: driver = accept domains = rochester.rr.com transport = spectrum_smtp no_more no_verify
This basically tells Exim that when it encounters a message bound for someone on rochester.rr.com to process that message using the spectrum_smtp transport, which I’ll add next.
In the section that starts begin transports, I add the block below. This is the workhorse of the process, passing the email message on the script we created above so that it can be passed on through msmtp.
spectrum_smtp: driver = pipe command = "/usr/lib/msmtp/route-spectrum $pipe_addresses" user = msmtp batch_max = 10
The user directive tells Exim to run the command as the msmtp user we set up earlier. This ensures that the script has access to the server configuration in the .msmtprc file that we took care to protect because it contained our password. The batch_max directive tells Exim that it can process up to 10 recipients of the same message with one call to the script; otherwise it would process each one with a separate connection to the Spectrum server. I’m not sure what the perfect number to use here is, but 10 seemed decent for my needs. The list of email addresses that are processed are placed into the $pipe_addresses variable and are passed as an argument to the script.
Since I changed the configuration, I need to restart Exim.
service exim restart
Now I can go into my email client and send a message to a Spectrum user. Checking the Exim main log (/var/log/exim/main.log) and msmtp log (/var/log/msmtp/msmtp.log) will confirm that it was delivered correctly.
The last thing that I want to do is set up log rotation for the msmtp log file. This will start a new log file each week, compressing the old one to save space on the server. To do this, create a new file in the logrotate.d directory:
vi /etc/logrotate.d/msmtp
And add the following to it:
/var/log/msmtp/msmtp.log {
missingok
}
That’s about it. This process isn’t the most sustainable, but it’s a decent workaround until Spectrum realizes that legitimate small businesses and non-profits do run mail servers on bulk hosting and they shouldn’t treat us all as spammers.
There is flaw in this configuration, but it should affect anyone too often: if, for some reason, msmtp is not able to connect to the remote mail server, Exim will view the failure as a hard fail and the message will be dropped. While I can work around this, I need to do a little more research on the exit status codes returned by msmtp, which I haven’t yet done.