NSLU2-Linux
view · edit · print · history

HowTo.ScheduledEMailAttachmentDownloader History

Hide minor edits - Show changes to markup

January 17, 2009, at 06:52 PM by vivekv --
Changed line 67 from:

Hope this article helps you setup a good email download capability

to:

Hope this article helps you setup a good email download capability. Ideas suggestions ? just drop me a note - I have an account on gmail called venugopalan.vivek

January 17, 2009, at 06:48 PM by vivekv --
Added line 28:
Changed line 36 from:

<code>

to:

[@

Changed lines 52-53 from:

</code>

to:

@]

Added line 58:

[@

Added lines 63-64:

@]

January 17, 2009, at 06:46 PM by vivekv --
Added line 13:

[@

Changed line 27 from:
to:

@]

January 17, 2009, at 06:06 PM by vivekv --
Changed lines 8-9 from:

First you have to install fetchmail and procmail on your slug. You will find the instructions for installation and configuration of fetchmail at SetupFetchMailWithGMail. If your ISP mail server is more easier to deal with than google you should look at [http://www.nslu2-linux.org/wiki/HowTo/SetUpAnEmailServer|Setting up an email server] and follow the instructions for Fetchmail setup.

to:

First you have to install fetchmail and procmail on your slug. You will find the instructions for installation and configuration of fetchmail at SetupFetchMailWithGMail. If your ISP mail server is more easier to deal with than google you should look at SetUpAnEmailServer and follow the instructions for Fetchmail setup.

January 17, 2009, at 06:05 PM by vivekv --
Changed lines 8-9 from:

First you have to install fetchmail and procmail on your slug. You will find the instructions for installation and configuration of fetchmail at [[http://www.nslu2-linux.org/wiki/HowTo/SetupFetchMailWithGMail][SetupFetchMailWithGMail]. If your ISP mail server is more easier to deal with than google you should look at [http://www.nslu2-linux.org/wiki/HowTo/SetUpAnEmailServer|Setting up an email server] and follow the instructions for Fetchmail setup.

to:

First you have to install fetchmail and procmail on your slug. You will find the instructions for installation and configuration of fetchmail at SetupFetchMailWithGMail. If your ISP mail server is more easier to deal with than google you should look at [http://www.nslu2-linux.org/wiki/HowTo/SetUpAnEmailServer|Setting up an email server] and follow the instructions for Fetchmail setup.

January 17, 2009, at 05:58 PM by vivekv --
Changed line 34 from:
to:

<code>

Changed lines 50-51 from:
to:

</code>

January 17, 2009, at 05:58 PM by vivekv --
Changed lines 1-2 from:

I receive a lot of emails with attachments which are essentially encrypted bank and other financial statements on a sepcial account that I had setup. All these statements have to be catalogued and archived. This was something I used to do every weekend and finally I decided to get it done right - ie have my little SLUG take care of this administrative detail of downloading and cataloging it every day and preparing a report that I can look at later.

to:

I receive a lot of emails with attachments which are essentially encrypted bank and other financial statements on a special account that I had setup. All these statements have to be cataloged and archived. This was something I used to do every weekend and finally I decided to get it done right - ie have my \SLUG take care of this administrative detail of downloading and cataloging it every day and preparing a report that I can look at later.

Changed line 10 from:
  • Configure Fetchmail and Procmail*
to:

Configure Fetchmail and Procmail

Changed line 29 from:
  • Processing attachments*
to:

Processing attachments

Changed lines 52-53 from:
  • Tying it all together*
to:

Tying it all together

Changed line 62 from:

Hope this article helps you setup a good email download capbility

to:

Hope this article helps you setup a good email download capability

January 17, 2009, at 05:56 PM by vivekv --
Changed lines 8-9 from:

First you have to install fetchmail and procmail on your slug. You will find the instructions for installation and configuration of fetchmail at http://www.nslu2-linux.org/wiki/HowTo/SetupFetchMailWithGMail][SetupFetchMailWithGMail]. If your ISP mail server is more easier to deal with than google you should look at [[http://www.nslu2-linux.org/wiki/HowTo/SetUpAnEmailServer][Setting up an email server and follow the instructions for Fetchmail setup.

to:

First you have to install fetchmail and procmail on your slug. You will find the instructions for installation and configuration of fetchmail at [[http://www.nslu2-linux.org/wiki/HowTo/SetupFetchMailWithGMail][SetupFetchMailWithGMail]. If your ISP mail server is more easier to deal with than google you should look at [http://www.nslu2-linux.org/wiki/HowTo/SetUpAnEmailServer|Setting up an email server] and follow the instructions for Fetchmail setup.

January 17, 2009, at 05:56 PM by vivekv --
Changed line 7 from:
  • Setup Fetchmail and Procmail*
to:

Setup Fetchmail and Procmail

January 17, 2009, at 05:56 PM by vivekv --
Added lines 1-62:

I receive a lot of emails with attachments which are essentially encrypted bank and other financial statements on a sepcial account that I had setup. All these statements have to be catalogued and archived. This was something I used to do every weekend and finally I decided to get it done right - ie have my little SLUG take care of this administrative detail of downloading and cataloging it every day and preparing a report that I can look at later.

The solution involved orchestration of fetchmail + procmail + scripting of my own. The fetchmail downloads the mail from the POP3? account and forwards it to procmail. Procmail then applies its rules to forward the mail to my custom program which then downloads the attachment and saves it to the right directory. It also generates a summary file that contains information about the mail received and the attachments. This summary can be reviewed by me once a week to check if everything is in order.

If you are still curious and read along so far, then here is how you set things up.

  • Setup Fetchmail and Procmail*

First you have to install fetchmail and procmail on your slug. You will find the instructions for installation and configuration of fetchmail at http://www.nslu2-linux.org/wiki/HowTo/SetupFetchMailWithGMail][SetupFetchMailWithGMail]. If your ISP mail server is more easier to deal with than google you should look at [[http://www.nslu2-linux.org/wiki/HowTo/SetUpAnEmailServer][Setting up an email server and follow the instructions for Fetchmail setup.

  • Configure Fetchmail and Procmail*

This section has to be a rant since the amount of pains that procmail put me through to configure and manage is too hard to explain. Finally I had to accept defeat and made procmail forward all emails to a custom script that I wrote which would do the bulk of the "real" processing. So my procmail rule looks something like this

  1. .procmailrc
  2. routes incoming mail to appropriate mailboxes

PATH=/usr/bin:/usr/local/bin:/opt/bin MAILDIR=$HOME/Mail PMDIR=$MAILDIR LOGFILE=$PMDIR/procmail.log LOGABSTRACT=NONE SHELL=/bin/sh ATTACHDIR=/share/hdd/data/HDD_1_1_1/Mails VERBOSE=NONE

:0

        |$HOME/filtermail.sh

I suspect that this kind of usage pretty much makes procmail useless and can probably be removed from the chain but that is not something I did and decided to live with procmail configured.

  • Processing attachments*

Ideally I wanted each attachment to be catalogued, stored in a separate directory and a report prepared on a monthly basis that tracked the statements that were received for the month. So instead of trying to create a full blown report, I decided to just generate a flat file that I could process further to get the kind of report that I needed. Essentially for each email, I unpacked it and then write out a log about the email.

The shell script that does all the processing is given below.

  1. !/bin/sh

ATTACHDIR=/share/hdd/data/HDD_1_1_1/Mails TMPFILE=`/bin/mktemp /tmp/filtermail.XXXXXX` DATEDIR=`/bin/date +m-%d` export SENDER=`tee $TMPFILE | formail -zxFrom: -zxReply-To:|cut -f2- -d'<' | cut -f1 -d'>'` SUBJECT=`/bin/cat $TMPFILE | formail -zxSubject:`

/bin/mkdir -p $ATTACHDIR/$SENDER/$DATEDIR >/dev/null munpack -t -C $ATTACHDIR/$SENDER/$DATEDIR $TMPFILE | while read decodedfile;do

        /bin/echo $SENDER~$DATEDIR~$SUBJECT~$ATTACHDIR/$SENDER/$DATEDIR/$decodedfile >>$ATTACHDIR/mails

done /bin/cat $TMPFILE /bin/rm $TMPFILE

  • Tying it all together*

Finally, we need all this to happen magically on a daily basis. So I tried to configure fetchmail on a new rc scipt called rc.fetchmail which is supposed to start fetchmail directly. The contents of the script are

  1. !/bin/sh

fetchmail

Thats all there is to it. This script should be called rc.fetchmail and should be saved in /etc/rc.d/ directory.

Hope this article helps you setup a good email download capbility

view · edit · print · history · Last edited by vivekv.
Originally by vivekv.
Page last modified on January 17, 2009, at 06:52 PM