PerlJacket User's Guide

rev. date: 05-Jun-2005

rev. by: Fred Morris

PerlJacket rev: 2

copyright: (c) 2002-2005 Fred Morris, Seattle WA. 98117 m3047@inwa.net

PerlJacket is provided for your use free of charge, at at your own risk: only you can determine its suitability for the purposes which you have in mind. You are granted a perpetual, royalty-free license to use, distribute and modify PerlJacket provided that you agree to hold the author, Fred Morris, harmless and to defend and indemnify him against any and all claims arising from your use.

What is PerlJacket?

PerlJacket is, appropriately enough, a perl script which can perform certain filtering and tagging operations on e-mail messages. It is meant to run on a mail server, typically in the context of a user (mail recipient's) account on that server.

As an "out of the box" solution it offers filter control by sending mail to oneself, and the ability to strip html parts of messages entirely (when the plaintext of the message is available) as well as rewrite multipart/alternative or multipart/parallel messages so that the parts are saved as attachments instead of being loaded by the mail reader; as a customizable utility it offers a single-pass, MIME-aware framework for processing incoming e-mail messages.

Design Goals

There were a few important design goals which motivated the development of PerlJacket:

• Something which was MIME-aware;
• that could be used in conjunction with procmail;
• which would strip/remove from active display all non- text/plain e-mail body parts;
• and which could be "tuned" by using the technique of sending e-mail to oneself;
• which ran in a single pass.

The reasoning for each of these goals is discussed below.

MIME-aware

MIME, which stands for Multipurpose Internet Mail Extension, is the protocol within the internet e-mail protocol which describes how e-mails are made up of multiple body parts, of different kinds, including plaintext, html text, attachments, etc.

procmail knows nothing about MIME; all it knows is that a message has a header and a body. It is impossible to filter for example specifically on an attachment, or to specifically do something with a particular part of an e-mail message, such as the HTML version, or a particular attachment type.

usable with procmail

Just because! It could be used with other things and in other ways, but it was envisioned as an adjunct for use with procmail. Off-the-shelf procmail filtering scripts often include what are essentially programs embedded into the script, many of them written in perl. I chose to write this as a separate perl script (or program) rather than embedding it in a .procmailrc file because I felt that maximized the flexibility of both.

stripping of non- plaintext parts

I generally don't need the HTML version of a message, the plaintext conveys meaning or the message is meaningless (I'm a snob and I believe in the power of the written word itself; deal with it.) And I certainly don't need bugged e-mail, or less-than-savory active payloads. But I do nonetheless get a lot of e-mail with attachments which I do want.

updatable via e-mail

As just alluded to, there is some stuff I want and some that I don't. I change my mind often. Seems silly that the filter which controls the filtering can't be updated in the same mode: I shouldn't have to telnet in or use a web interface to change my e-mail filter, I should be able to do it using e-mail itself.

single pass

Some sophisticated MIME processing frameworks and tools exist. However they require buffering of the message so that it can be operated on in a "random access" manner, and this takes space if not processing power.

I was willing to sacrifice the ability to look ahead and write behind in return for something which would run in a single pass, and never load more than one MIME part of the messsage into memory at a time.

Synopsis of Use

Prerequisites

You must have:

• an internet account and access to a "shell account" on the mail server
• a mail server with procmail and perl installed
• some minimal knowledge of procmail, or else a willingness to learn.

Setup

1. Copy perljacket.plx to your home (login) directory. The file is available here in an HTML-ized version which you can save as text and rename perljacket.plx.

2. Change the permissions so that perljacket.plx is protected and executable. Execute the command
   
   chmod a=rx perljacket.plx
   
   
3. Copy the template filter file perljacket.filter to your home directory. The file is available here in an HTML-zied version which you can save as text and rename perljacket.filter.
   
4. Change the permissions so that perljacket.filter is writeable by you, but (at most) can be read by anybody else. Execute the command
   
   chmod u=rwx,go=r perljacket.filter
   
   On some systems you can leave off the go=r, and make it so nobody can read your filter file, but on some systems it needs to be world readable so that it will be accessible to the mail delivery thread.
   
5. Edit perljacket.filter and create the initial filter settings. Technically and theoretically speaking you can leave this file blank when you turn filtering on at the end of this setup procedure and everything will keep working just the same, but you're probably not interested in that if you're going to the trouble of installing PerlJacket, are you? Execute the command
   
   pico perljacket.filter
   
   to edit the file from the shell account, or if you prefer you can use FTP to transfer the file as text to your workstation and edit there and copy it back to the mail host. Your system administrator may set up your account with a default copy of this file, providing you with appropriate initial filter settings.
   
6. Copy the .procmailrc file to a temporary location. The file is available here in an HTML-ized version which you can save as text and rename procmailrc.new.
   
   You don't want to "turn it loose" until you have a properly configured file named .procmailrc! You may already have a file named .procmailrc, in which case suddenly switching to a different one might cause unexpected problems; plus, on some systems such as the Cobalt Qube, the simple act of placing a .procmailrc in a user's directory enables it.. there is no final setup of .forward. Call it something like procmailrc.new.
   
   To see if you already have a .procmailrc use the command
   
   ls -al .proc*
   
   
7. Edit procmailrc.new and create the initial handling and postprocessing options. The sample .procmailrc offers categories for: discarding messages, bouncing messages based on who they're from, and bounding messages based on who they're (not) to; filtering of MIME parts within an individual message is done internally in PerlJacket as is the e-mail updating of the filter.
   
   Do not use the filter as shipped! You should edit it to suit your needs, and especially to get rid of any references to my account which you should replace with your own. My Procmail Resource Guide can probably get you started. You can also play with my Procmail Recipe Generator.
   
8. Copy the temporary file to .procmailrc. Use the command
   
   mv procmailrc.new .procmailrc
   
   Alternatively, your system administrator may provide you with a .procmailrc tailored to your corporate culture and policies, and documentation on how it interacts with PerlJacket's filtering.
   
9. Change the permissions so that .procmailrc is writeable by you, but (at most) can be read by anybody else. Execute the command
   
   chmod u=rx,go=r .procmailrc
   
   On some systems you can leave off the go=r, and make it so nobody can read your filter file, but on some systems it needs to be world readable so that it will be accessible to the mail delivery thread.
   
10. "Turn it on" by creating a .forward file. On some systems you won't need to do this, and it "went live" during the previous step. On others you will need to create one of these. Instructions can be found in the Procmail Resource Guide and the Procmail Recipe Generator.

Use

PerlJacket will filter your messages based on the filter you define:

• inserting headers based on successfully matching filter expressions; and/or
• performing an aggressive stripping/rewriting of MIME-formatted emails based on successfully matching filter expressions.

After that, you will probably want to have procmail perform some action(s) based on the headers inserted, such as sending an autoreply, discarding the message, saving it to a special folder, etc.

At this point, the mail arrives in your "dropbox" or inbox on the server, where it will be downloaded or viewable by your e-mail client.

You may want to define filters in your e-mail client as well, which do certain things depending on the headers inserted.

Updating the filter

The filter can be updated by sending yourself an e-mail in two stages. It is important when you send these e-mail messages that you:

• disable HTML/fancy/"enhanced" e-mail, and send good-old-fashioned plaintext;
• turn off line wrapping; and
• turn off Quoted-Printable encoding.

To update the filter:

1. Send yourself an e-mail with the subject
   
   REQUEST UPDATE COOKIE


2. PerlJacket will respond with a message with the subject line
   
    UPDATE COOKIE=(some large number) TTL=(some large number)
   
   This message will contain the current contents of your filter, plus brief instructions on making edits.
   
   
3. Reply to this message with a subject of
   
    Re: UPDATE COOKIE=(some large number) TTL=(some large number)
   
   
4. Make the desired changes, creating new records with the keywords ADD and DELETE at the beginning of the appropriate lines; all other lines will be ignored.
   
   NOTE: Be sure to make your reply plaintext, with no line wrapping and Quoted-Printable encoding turned off.
   
5. Send your reply.
   
6. You will receive a confirmation message describing the changes which were successfully applied.

Filtering and flagging in the mail client

The headers inserted by PerlJacket can be utilized in e-mail clients such as Eudora and Outlook to flag incoming mail messages or send them to particular folders.

Checking server-based mail repositories or folders

Common post-processing by procmail includes discarding mail outright (such as that identified as spam); but more conservative approaches include stripping spam to just the headers and saving those to a separate mail folder on the server, where they won't be downloaded by a mail client such as Eudora or Outlook. Providing this folder is located in an appropriate directory (typically the subdirectory mail under the user's login directory), pine will be able to open the folder and you can review the headers and delete delete them one at a time; or, you can just delete this folder every so often.

Log maintenance

By default, the supplied .procmailrc recipe writes a logfile named procmail.log in the user's login directory. This file will keep growing and growing and growing, and will need to be deleted every so often. Alternatively, once you're sure everything is working file, you can disable logging.

Other Documentation

• The POD file, for documentation on PerlJacket itself.
• My Procmail Guide, for more information on getting started with procmail.

Known Issues/Persnicketyness

I use this every day, and I've set it up for at least one client as of this writing. But we're all computer-savvy, and at least somewhat familiar with mail filtering, so we have a general idea of what to do and what not to do.

Particularly, I call your attention to the following:

• things may be case-sensitive for no particular reason
• extra spaces may not be ignored
• errors are unlikely to cause mayhem, but they may be silently ignored
• the actions of PerlJacket in rewriting messages, plus postprocessing in procmail, may cause some messages to be reported as mangled by certain e-mail clients under certain conditions (however, spammers send mangled mail deliberately to get around filtering countermeasures, so what's new?)

Debugging

PerlJacket will write headers describing the actions it is taking, and even for some error conditions it detects.

By default, procmail will write a log. Procmail has different logging modes, and verbose in particular will give you lots of information about exactly what is happening (and result in a logfile which grows very rapidly!).

What do those recipes in the supplied .procmailrc do?

The supplied .procmailrc disposition recipes are pretty close to verbatim as to what the recipes in my own .procmailrc do. I really intended them to be samples, but based on feedback it seems as though people want to use them "as shipped", and as a consequence of that they want more documentation as to what they do.

First off, you can't simply use it as shipped

There are several places in the file where you must substitute your own account and domain. This section is not a tutorial on using procmail; there are plenty of those around, if you need one, go find one.

Calling perljacket

The following fragment invokes perljacket to preprocess all messages which are less than 3Mbytes in size:

# Pass everything through perljacket
:0 H B h b f w
* < 3000000
| ./perljacket.plx -t1000

:0 E h b f w
| formail -A"X-PerlJacket: Warning: oversized message or error!"

Place any other preprocessing before this.

After PerlJacket processes the message, the various mungings described elsewhere will have been performed, and gratuitous headers will have been inserted. The sample recipes deal with them as outlined in the following section. After that, you may want to call something like SpamAssassin... or you may want to call it first. That's up to you.

Basically my paradigm is:

• Use PerlJacket to perform various mungings which depend on the structure of the e-mail, and insert gratuitous headers based on the filters defined in perljacket.filter.
• Use recipes in the .procmailrc and based on the gratuitous headers to determine the disposition of the message.

That said, here's how the gratuitous headers I usually insert interact with the sample recipes

As documented, I typically have rules in my perljacket.filter file which insert the following headers:

# In addition to the X-PerlJacket header, my perljacket.filter defines
# the following headers:
#
# X-Null         Nuke em!
# X-Bcc          Stuff that was BCCed
# X-Alias        Stuff that was sent to an alias
# X-Friend       Friendlies
# X-Mail-List    Stuff from a recognized mailing list
# X-Special      Stuff that is special
# X-Nozzle       Stuff from misconfigured autoresponders
# X-Banned       Obnoxious domains
# X-Banned-Pass  Recognized subject line passwords for banned domains
# X-Alias-Pass   Recognized subject line passwords for aliased mail

The rest of this section describes how those headers interact with each other and with the remaining 4 recipes. These recipes deal with messages in the following order of precedence:

1. X-Null:
2. X-Banned:
3. X-Alias:
4. X-Bcc:

The X-Null: Recipe

This recipe immediately discards any mail which was tagged with this header. For instance if I have a filter such as

<EXPR>bozo\.com</EXPR> <SCOPE>From:,Received:</SCOPE> <HEADER>X-Null:</HEADER>

Then any mail with bozo.com in the From: or Received: headers is immediately discarded.

The X-Banned: Recipe

Places identified by filters which I don't expect to get e-mail from, but which I do get spam from.

1. If they're tagged as a friend, special or a known mailing list, they're ok and not processed as banned.
2. If they've supplied a special keyword on the subject line (X-Banned-Pass: filter) they're ok and not processed as banned
3. A note is returned to the sender.
4. The mail is stored on the server, and not downloaded.

The X-Alias: Recipe

Mail which is detected by filters as being to some account other than my primary account.

1. If they're tagged as a friend, special or a known mailing list, they're ok and not processed as banned.
2. If they've supplied a special keyword on the subject line (X-Banned-Pass: filter) they're ok and not processed as banned.
3. If the mail was Bcc'ed, it is discarded.
4. A note is returned to the sender.
5. The full text is stored on the server.
6. Just the headers and a couple lines of the body are downloaded.

The X-Bcc: Recipe

Mail which is detected by filters as not being addressed to me at all... so-called "blind carbon copies".

1. If they're tagged as a friend, special or a known mailing list, they're ok and not processed as banned.
2. If they've supplied a special keyword on the subject line (X-Alias-Pass: filter) they're ok and not processed as banned.
3. A note is returned to the sender.
4. I hate Pegasus (and a few other spammer mailers and things...)
5. The full text is stored on the server.
6. The body is replaced with the phrase "junk-mail".

X-Friend: and X-Special:

My friends and other important things get tagged like this. They are largely immune from the disposition processing above. Having more than one category allows me to highlight them different in my mail client.

X-Banned-Pass: and X-Alias-Pass:

Special keyphrases allow people who make the effort to bypass the usual disposition processing for mail sent to a "junk" account or from banned domains.

X-Nozzle:

Some people don't use X-Loop: headers on their autoreplies. Worse still, they insist on stripping your headers as well. As a consequence you end up with an "infinite loop" of mail autoreplies.

This includes vacation programs, thank yous saying how much they appreciate your contacting them, etc., etc. Over the years I've had several conversations with people with such brain-damaged autoresponders, and their responses invariably seem to involve self-justification based on one or both of the following:

• This is the way that <insert beloved software vendor here> ships it. (I'm supposed to conclude, apparently, that that makes it the right thing to do).
• There's nothing about X-Loop: in the RFCs. (True, but there's precious little about vacation programs, mailing lists, or autoresponders in general.)

So, I use a filter to tag them with an X-Nozzle: header which disables my autoreplies... or I just give them an X-Null: header and send them straight to /dev/null.. it just depends.

To summarize...

The following table attempts to summarize the interaction of the various headers and disposition recipes:

	/dev/null	banned domains	mail addressed to aliases	blind carbon copies
X-Null:	Straight to /dev/null!
X-Banned:		Tagged by a filter as being from a banned user/domain.
X-Alias:			Tagged by a filter as being addressed to some address other than my primary account (an alias).
X-Bcc:				Tagged by a filter as being a blind carbon copy.
X-Friend:		Tagged by a filter as being someone I recognize as a friend; exempted from disposition processing.
X-Mailing-List:		Tagged by a filter as being a posting to a mailing list to which I belong. My mail client uses these headers to place the messages in special folders. Exempted from disposition processing.
X-Special:		Tagged by a filter as being special for some reason. My mail client uses these headers to highlight the messages. Exempted from disposition processing.
X-Nozzle:		Disables autoreplies.
X-Banned-Pass:		Tagged by a filter as containing a special keyword or passphrase. Exempted from disposition processing.
X-Alias-Pass:			Tagged by a filter as containing a special keyword or passphrase. Exempted from disposition processing.

Customizing

procmail As a Calling Environment

procmail is a convenient calling environment because it knows how to do certain things with e-mail already, and lets you interact with the operating system in certain ways (such as calling PerlJacket).

Take advantage of the capabilities of procmail to perform actions based on the results of PerlJacket's filtering!

Alternatively, you don't have to use procmail as a calling environment: PerlJacket expects to see a "proper" e-mail message on stdin, and writes the filtered message to stdout.

Cliches

Once you know anything at all about the structure of e-mails and the purposes of various mail headers, it is apparent that semantically speaking a message for instance "To" somebody might be addressed to them on the To: header, or the Cc: header, etc. It's tedious to have to repeat each of these in every scope statement, so certain cliches are defined for your use, for instance =To= for the above case.

You can edit these cliches in perljacket.plx, and define your own as well. This is probably one of the simplest customizations you can make, and also very useful.

Part Handlers

Don't like what PerlJacket does to your e-mails? Want something special done with a particular kind of attachment? Change it!

perljacket.plx is a big file (!), but don't be intimidated as long as you know at least some perl: most of the action occurs in three routines near the end of the file:

• at_Header is called immediately after the e-mail's headers have been loaded and before reading the body;
• at_Multipart is called immediately after processing the multipart headers for a multipart MIME message (or immediately after at_Header for a top-level MIME multipart);
• at_Entity is called immediately after processing the part headers for a terminal or atomic MIME part.

Start there.