NSLU2-Linux
view · edit · print · history

Set Up an Email Server with
Postfix+Cyrus-imapd+Sieve+SpamAssassin

''Built on the work of pTweety et al at SetUpAnEmailServer. Please update here or send questions to Lee Kimber <lee atnospam kimberconsulting dot com>. ''

Prepare the system

Before you start installing the email-specific packages, update the system, then install these feeds and utilities:

ipkg update
ipkg install unslung-feeds
ipkg update

Install packages Install perl, libdb, coreutils, cyrus-sasl.

Although this howto does not go into how to use cyrus-sasl, having it present allows you to change the mail delivery options to the server once you have the server up and running correctly.

After installing cyrus-sasl run:

/opt/sbin/saslpasswd2 mail

which lets you set the password.

That'll create the sasl2 file that you should then set the permissions on:

chmod 644 /opt/etc/sasl2

There's some ambiguity about how the permisions of file /opt/etc/sasl2 should look:

# ls -l /opt/etc/sasl2
-rw-r--r-- 1 root root ...

or

# ls -l /opt/etc/sasl2
-rw------- 1 root root ...

I have the latter, but I'm not using SASL yet.

Install readline, fetchmail, cyrus-imapd, postfix, (bogofilter or perl-spamassassin) using ipkg.

Configuration

To retain your sanity while testing and troubleshooting, you are better off configuring, starting up and testing the email server software in the following order:

  1. Postfix (followed by a sending test)
  2. Cyrus-imapd
  3. Bogofilter or SpamAssassin (perl-spamassassin)

Building and testing the email server this way allows you to test and debug each essential piece before building on top of it. The reason we install postfix after installing cyrus-imapd is that the Unslung build of postfix allegedly configures itself around cyrus-imapd. However, we have to touch /opt/etc/sadl2 before installing postfix as described above due to a glitch in the way this file is created.

Configuring Postfix

The two main files for configuring postfix to send (and receive) are:

/opt/etc/postfix/main.cf
and
/opt/etc/postfix/master.cf

The relevant changes (un-comments and edits) I made to /opt/etc/postfix/main.cf are:

myhostname = cen-nas1-1.leedomain.com (was #myhostname = virtual.domain.tld)
mydomain = leedomain.com
myorigin = $mydomain
inet_interfaces = all
proxy_interfaces = <external NAT interface IP address>
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain
mynetworks = 192.168.1.0/24, 192.168.2.0/24, 127.0.0.0/8 (internal subnets on my LAN)


Comment: [JamesL] Don't put underscores in your hostname - postfix doesn't like this.



Comment: [Jeroen] I also needed to make my ISP's SMTP server known for outgoing mail to work:

relayhost = <my-isp's-smtp-server>

--Jeroen

In /opt/etc/postfix/master.cf:

Beneath the line:
smtp inet n - n - - smtpd

I added:
#smtp inet n - n - - smtpd -o content_filter=spamassassin
#smtp inet n - n - - smtpd -o content_filter=bogofilter

At the end of master.cf I added: spamassassin unix - n n - - pipe
user=spamd argv=/opt/bin/spamc -f -e /opt/sbin/sendmail -oi -f ${sender} ${recipient}

bogofilter unix - n n - - pipe
flags=R user=bogo argv=/opt/sbin/postfix-bogofilter.sh -f ${sender} -- ${recipient}

Change your aliases file to ensure you receive error messages:

# vi /opt/etc/aliases
#root: you
root: tux

Then run:

# /opt/bin/newaliases

Add maildrop to /etc/group if it isn't already there (on mine it wasn't):

# vi /etc/group
maildrop:x:69:maildrop

Reload the Postfix configuration by issuing:

/opt/sbin/postfix -c /opt/etc/postfix reload

Test Postfix's ability to send

Start the postfix service by issuing:

/opt/etc/init.d/S69postfix

You should see the following messages:

starting service postfix
postfix/postfix-script: starting the Postfix mail system

You can also see if postfix is running by issuing:

ps aux | grep postfix

If it is running you should see something like:

3479 root 2588 S /opt/libexec/postfix/master

You'll see the system try to stop the presumably running service (and fail if the service is not running) and start the service. Hopefully, it will succeed in starting the service.

If the service is running, we can test its ability to send email to a known-working remote address by telnetting to the email server from a LAN machine using port 25:

telnet 192.168.1.77 25\\
220 `SEANAS1.leedomain.com ESMTP Postfix\\
ehlo leedomain.com\\
250-`SEANAS1.leedomain.com\\
250-PIPELINING\\
250-SIZE 10240000\\
250-VRFY\\
250-ETRN\\
250 8BITMIME\\
mail from: lee@leedomain.com\\
250 Ok\\
rcpt to: tux@leedomain.com\\
250 Ok\\
data\\
354 End data with <CR><LF>.<CR><LF>\\
Message-ID: test1.00000@leedomain.com\\
From: lee <lee@leedomain.com>\\
To: lee <tux@leedomain.com>\\
Subject: Test 2\\
\\
Body of test 2\\
.\\
250 Ok: queued as 6DA50EAAC\\
quit\\
221 Bye\\
\\
Connection to host lost.

Provided you receive the mail at the remote email account, your system is working. If it isn't, check the /var/log/messages file for hints about the cause of the problem. You'd be wise to monitor the message log in real-time by tailing /var/log/messages. Do that by opening an ssh session to the email server and then issuing:

tail -f /var/log/messages

Also, search the web for notes on how to troubleshoot postfix. There's lots out there.

Configure client read access to email

Assuming that sending was successful, we can work on the system's ability to receive email.

In /etc/services, uncomment - if necessary - the lines:

#pop3 110/tcp pop-3 # POP version 3
#pop3 110/udp pop-3
...
imap2 143/tcp imap # Interim Mail Access Proto v2
imap2 143/udp imap
...
imap3 220/tcp # Interactive Mail Access
imap3 220/udp # Protocol v3

and - if necessary - add these lines at the end of the file:

# start of cyrus-imapd services
imsp 406/tcp
acap 674/tcp
sieve 2000/tcp
lmtp 2003/tcp
fud 4201/udp
# end of cyrus-imapd services

Configure Cyrus-imapd

Create your email users with the web front-end of your nslu2, e.g. tux in this example. You do not have to create a user called mail/

Note: for those using OpenSlug (ie no web-frontend), use:

  # adduser [username]

Start cyrus-imapd by issuing:

/opt/etc/init.d/S59cyrus-imapd

You can test that Cyrus-imapd is offering pop and imap services by telnetting to the relevant ports. eg:

telnet 192.168.1.77 110
or
telnet 192.168.1.77 143

In each case, the Cyrus-imapd server should respond with a message along the lines of:

+OK SEANAS1 Cyrus POP3 v2.2.10 server ready <1751987801.1140277042@SEANAS1>
or
* OK SEANAS1 Cyrus IMAP4 v2.2.10 server ready

Create user mailboxes

You need the Cyrus-imapd server up and running before the mailbox creation tool cyradm will work.

Run the following command Ignoring the termcap error message and You'll be asked for your password which you entered earlier

# /opt/bin/cyradm --user mail localhost
> cm user.tux
> cm user.tux.Drafts
> cm user.tux.Outbox
> cm user.tux.Sent
> cm user.tux.Trash
> cm user.tux.Ham
> cm user.tux.Spam
> lm
> quit

If tux is a name containing a dot ie fred.bloggs then use the following command syntax cm user/fred^bloggs/Drafts

We can now test whether the Cyrus-imapd server is working properly by using telnet to create and send a message to the user "tux". Before starting, you might want to open an ssh connection the email server and tail (monitor) the messages log by issuing (once you've ssh'd in):

tail -f /var/log/messages

Then do an email-sending test:

telnet 192.168.1.78 25
220 seanas1.leedomain.com ESMTP Postfix
ehlo leedomain.com
250-seanas1.leedomain.com
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-AUTH PLAIN
250 8BITMIME
mail from: lee@leedomain.com
250 Ok
rcpt to: tux@leedomain.com
250 Ok
data
354 End data with <CR><LF>.<CR><LF>
Message-ID: 000001.00000@leedomain.com
To: tux@leedomain.com
From: lee@leedomain.com
Subject: Test 1 to Cyrus-imapd user tux

Body of test 1
.
250 Ok: queued as C7E6E3F02
quit
221 Bye

Connection to host lost.

The email should have been sent successfully, as shown above. If you tailed the messages log, you should see postifx processing the email and handing it off to Cyrus-imapd. At this point you may see a non-fatal error from Cyrus-imapd's sieve program, which has no filtering scripts to run against the incoming email.

Set up a remote pop or imap client to test retrieval of email from the server. Hopefully, you should find the test email is there. If not, track down the problem using the messages log.

Having tested this far and checked that the basic email-realted services are working correctly, we can add spam-filtering, knowing that if anything goes wrong, the error is in the spam filtering set up, not the email set up.

Setting up spam filtering

These instructions set up SpamAssassin. In my experience, the default Bogofilter package places Bogofilter scripts in /opt/var/lib/ but Bogofilter's scripts (postfix-bogfilter.sh for example)set the default location to /opt/var/spool/. I fixed this by working through the various Bogofilter scripts replacing paths that included "spool" with paths to "lib" instead. This worked but the scripts for training Bogofilter are unclear to me so I continue to use SpamAssassin.

Create user and group for spamd:

$ vi /etc/passwd
-> spamd:x:40:40:spamd:/opt/var/spool/spamd:
         $ vi /etc/group
spamd:x:40:spamd

Create file processing locations for spamd:

$ mkdir -p /opt/var/spool/spamd/.spamassassin
-> $ chown -R spamd:spamd /opt/var/spool/spamd/.spamassassin

For Bogofilter, do the following:

Create user and group

$ vi /etc/passwd
-> bogo:x:41:41:bogo:/opt/var/lib/bogofilter:
$ vi /etc/group
-> filter:x:41:filter

Create file processing locations

$ mkdir -p /opt/var/lib/bogofilter
-> $ chown -R bogo. /opt/var/lib/bogofilter

add SpamAssassin content filter definition to end of /opt/etc/postfix/master.cf if you missed it in one of the comments above. (lines starting with a blank space in the master.cf file are automatically assumed to be a continuation of the previous line)

spamassassin unix - n n - - pipe

 user=spamd argv=/opt/bin/spamc -f
 -e /opt/sbin/sendmail -oi -f ${sender} ${recipient}

Make sure /opt/var/spool/postfix/maildrop has read/write permissions for user spamd when running /opt/bin/spamc (There's probably a better way of doing this)

chmod 777 /opt/var/spool/postfix/maildrop

I found that after subsequently upgrading the system's packages as part of routine maintenance, I would see the following errors in /var/log/messages:

<20>Dec 13 13:00:48 postfix/postdrop[26875]: warning: mail_queue_enter: create file maildrop/942012.26875: Permission denied

It was the chmod command above that fixed these errors.

Note: having installed both Bogofilter and SpamAssassin, you can set which spam filter you want to run by commenting one or other out in the /opt/etc/postfix/master.cf file. For example:

In /opt/etc/postfix/master.cf, comment out the line:
smtp inet n - n - - smtpd

You can then switch in either SpamAssassin or Bogofilter by writing in and uncommenting one of either:
#smtp inet n - n - - smtpd -o content_filter=spamassassin
or
#smtp inet n - n - - smtpd -o content_filter=bogofilter

Theoretically you are supposed to reload the Postfix config after doing this but I found re-starting and sometimes rebooting gave me significantly better testing results because - I presume - it forced Postfix to flush queues. I am not sure though. Certainly it is possible to manually flush postfix's queues but I haven't had time to experiment with that yet.

Restart postfix by issuing:

# /opt/sbin/postfix -c /opt/etc/postfix stop
postfix/postfix-script: stopping the Postfix mail system

# /opt/sbin/postfix -c /opt/etc/postfix start
postfix/postfix-script: starting the Postfix mail system


Comment: [Mark] Before you can test the spam filtering as described below, you first need to start the spamd server

# /opt/etc/init.d/S62spamd

Also, on my system at least, the /opt/bin/spamd script references the perl interpreter assuming you have unslung onto disk1. I'm running on disk2 so the line that reads

#!/share/hdd/data/opt/bin/perl5.8.6 -T -w

becomes

#!/share/flash/data/opt/bin/perl5.8.6 -T -w

--Mark

If, when you start spamd, you see error messages along the lines of: "/opt/bin/spamd: file not found" then check the first line of /opt/bin/spamd. If you have a later version of perl installed (as you will have these days) then the perl directory name will need changing to the perl directory name you actually have. Eg

/share/flash/data/opt/bin/perl5.8.6 -T -w should likely be /share/flash/data/opt/bin/perl5.8.8 -T -w

Then try restarting spamd again

If the S62spamd script is missing then this appears to do the job

#/bin/sh echo "Starting spamd" /opt/bin/spamd -d -c -m 1 -u spamd --max-conn-per-child=100

Give everyone permission to run it just to be sure.

chmod 777 /opt/etc/init.d/S62spamd

Test the spam filtering by telnet-sending your email server user another email. Then read the email in your email software and check the email's headers for new headers added by the spam-filter. If using spamAssassin, you should see email headers similar to the following:

X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on SEANAS1
X-Spam-Level:
X-Spam-Status: No, score=-1.9 required=2.1
tests=ALL_TRUSTED,AWL,NO_REAL_NAME autolearn=ham version=3.0.4

You can - and should - also tail the /var/log/messages file to watch the spam filter as it processes and reports on the email. The only error you should see at this point is an error relating to sieve, which we will deal with in a moment.

Once spam-tagging is working, you can modify Spamassassin's handling of spam by editing the /opt/etc/spamassassin/local.cf file. Mine looks like this:

# This is the right place to customize your installation of SpamAssassin.
#
# See 'perldoc Mail::SpamAssassin::Conf' for details of what can be
# tweaked.
#
###########################################################################
#
rewrite_header Subject ***SPAM***
report_safe 1
# trusted_networks 212.17.35.
# lock_method flock
# Added by Lee
required_hits 2.1
body WHITELIST_HOUSING /Subject: Housing/
tflags WHITELIST_HOUSING [ nice ]

During testing, I found that spamd would fail to process messages during a SIGCHILD thread handover (caused when child threads reach their maximum default connection limit of five. Thos emails would be handed off to Cyrus-imapd without being spam-checked. I am testing how to get around this by playing with spamd's connection limits. I edited the spamd start-up script at /opt/etc/init.d/S62spamd. In the daemon start section I took the line:

/opt/bin/spamd -d -c -m 1 -u spamd

and added a --max-conn-per-child argument as shown below:

/opt/bin/spamd -d -c -m 1 -u spamd --max-conn-per-child=100

This worked much better the first time I used it but I am still testing it.

Comment: [JamesL] Note the -m 1 setting above. This is the max number of child processes to be spawned. The default of 5 can cause the Slug to run out of memory. I'm running with -m 3 OK, but am considering reducing it.

Lee adds: I am currently testing how well Ham and Spam folder training works. I separate my email into these folders and have a weekly cron job that runs the following script to train spamassassin:

  1. !/bin/sh

/opt/bin/sa-learn --showdots --spam /opt/var/spool/imap/user/tux/Spam /opt/bin/sa-learn --showdots --ham /opt/var/spool/imap/user/tux/Ham /opt/bin/sa-learn --showdots --spam /opt/var/spool/imap/user/lee/Spam /opt/bin/sa-learn --showdots --ham /opt/var/spool/imap/user/lee/Ham

exit 0;

Configure sieve to provide server-side mailbox sorting

Provide a script to control how cyrus-imapd's built in sieve program handles spam-tagged messages on the server. Using sieve in this way is useful for imap clients, who want filtering carried out on the server to save bandwidth. I'm not certain my sieve script works and am currently testing it.

Create a sieve file on the email server. The following file is overkill - it checks headers from different spam checkers. The location that you should save this file is the directory given by the parameter sievedir in the /opt/etc/imapd.conf file.

@@# vi user.tux.siv
require "fileinto";
require "imapflags";

if anyof ( header :contains "X-Spam-Flag" "YES",
-> header :contains "X-Spam-Level" "***",
-> header :contains "X-Bogosity" "Spam, ",
-> header :contains "X-Spam-Status" "Yes",
-> header :contains "Subject" "***SPAM***" )
-> { setflag "\\Seen"; fileinto "INBOX.Spam"; }
else { fileinto "INBOX"; }@@

Now, log in to sieve, upload the file and then activate it.

# sieveshell -u tux <hostname>
> put user.tux.sieve.siv
> list
> activate user.tux.sieve.siv
> list
> quit

Test the sieve script by sending an email to the server using the telnet technique shown earlier. If you tail the messages log while doing this, you should see no more sieve errors.

Comment: [nsc] I had a problem that even after doing the sieveshell it still failed with the IOERROR that the defaultbc script was missing and the scripts didn't work. I had to create a symlink called defaulttbc in the user's sieve directory (which for me is /opt/var/lib/imap/sieve/first_initial_of_username/user_name) and then it worked.

There's not much on how to use sieve (a server-based mail filter program that comes with cyrus-imap). I found the best source was here: http://www.bsdforums.org/forums/showthread.php?t=8238

After some testing, I found the above multi-test sieve file manipulated every email as though the email were spam, so I simplified it to only manipulate emails that spamassassin had identified as spam. This has been remarkable successful. Here's how my current sieve file looks. Load and activate it as described above:

require "fileinto"; require "imapflags";

if anyof ( header :contains "Subject" "***SPAM***" )

      { setflag "\\Seen"; fileinto "INBOX.Spam"; }

else { fileinto "INBOX"; }

Back to sieve, if the sieve test worked, then the only thing left to do is to decide how to feed incoming emails into the email server. There are three ways of doing this:

  1. Set the email server up directly connected to the Internet. I'm not doing this because I haven't satisfied myself that the server's security is adequate.
  2. Use your ISP's email server via a SASL connection
  3. Use your ISP's email server and uses fetchmail to poll your email account on the ISP's server

We'll go the fetchmail route because it is secure, well-documented and allows us to establish that the rest of the email server is working correctly in the real world before we use methods 1 or 2.

Edit the fetchmail configuration file called fetchmailrc in the email server's /opt/etc/ directory:

# vi /opt/etc/fetchmailrc
set postmaster lee
set no bouncemail
set syslog

poll "www.server.com" protocol imap user 'tuxuser' there
->with password "noneofyourbusiness" is "tux" here

I found that using fetchmail in daemon mode locked up my server. To maintain control while I debugged it, I fired fetchmail with a cronjob that looked like this:

2 * * * * root /opt/bin/fetchmail -f /opt/etc/fetchmailrc &> /dev/null

Note: I found fetchmail and my ISP's pop server often experienced protocol failures that stopped downloads. In as far as I have investigated this, it seems to be a spat over the list command.

Next steps To reduce the load on the email server I am working to dump (not bounce) known spam before it gets to SpamAssassin. I am using Postfix's header_checks and fetchmail's antispam option to have fetchmail drop offending mail before it is allowed into the Postfix system.

view · edit · print · history · Last edited by Bullfrog.
Based on work by Lee Kimber, fcarolo, bullfrog, warti, nsc, JamesL, Jeroen, Bullfrog, and MarkH.
Originally by Lee Kimber.
Page last modified on November 23, 2008, at 09:18 PM