PROCMAIL QUICK START
An introduction to email filtering with a focus on procmail

What is Procmail and What You Need to Use this Quick Start

Procmail is free/libre open-source software that is both a mail processor and a mail delivery agent (MDA). It can be used by either a system administrator or a user to automatically process and deliver incoming mail messages. It can also be used to re-process and re-deliver messages that are already in a mailbox.

This Procmail tutorial is aimed at regular users, not system administrators. In order to use these instructions, you need:

1. A basic understanding of Internet mail flow, which you can learn about at:
   • bath.ac.uk: Anatomy of an Email Message
   • Google Help > Gmail Help > Privacy & Security > Learn to read full email headers
   • How UW Email Works -- this article, including its nice diagrams, is relevant to most modern Internet email systems (BTW, if you have ever taken a class at the University of Washington, you can get a complimentary or premium MyUW.net account)
   • Reverse Spam Filtering: Winning Without Fighting, especially in the Email Deflexion Flowchart
   • MUAs, MTAs, and MDAs
   • How Unix Email Works
   • E-Mail Server Administration 101 by Dee Ann LeBlanc and Robert LeBlanc, which describes mail flowing through MUA->MSA->MTA->...->MTA->(MFA)->MDA->MUA
   • Procmail FAQ: Appendix B -- Figuring Out the Mail Flow
   • SPF Mail Flows diagram, which is also available on a T-shirt
   • Internet mail infrastructure by D. J. Bernstein
   • The Internet Protocol Journal: Volume 8, Number 4: Challenges in Anti-Spam Efforts by Dave Crocker
   • Archive.org: Cyrusoft.com: Sieve: Documents: The Sieve Language and a General Model for Delivery and Interoperable Filtering in Internet Mail by Matthew Wall (PDF or HTML)
   • The message Re: Usability. Spamassassin in the SpamAssassin list -- this includes a quiz that anyone who edits SpamAssassin or Procmail configuration files should be able to pass

In addition to understanding Internet mail flow, your mail messages must be delivered to a system that:

2. has procmail installed
3. you have shell access to
4. either
   1. uses procmail as the default local delivery agent (LDA), or
   2. uses a message transfer agent (MTA) that lets you use a .forward file to set procmail as your LDA, or
   3. allows you to set procmail as your LDA some other way. For example, with qmail, use dot-qmail files; with MMDF, use .maildelivery; with getmail, use .getmailrc; and with fetchmail, use .fetchmailrc.

If your system satisfies 4a or 4c, make sure that you skip Steps 8 & 9 below, that is, do not set up a .forward file.

	If you are looking for a provider that has procmail installed, see my list of Free or Reasonably-Priced IMAP Service Providers — most of the providers who give full Unix shell access also let their users write and manage their own procmail recipes.

Terms Used in This Article

Note about the ordering of these terms:  The terms listed earlier are used in the definitions of later terms.

PM  or  pm

In this article I use PM as an abbreviation for Procmail. For example, I use pmlog for the Procmail log file, PMRE for Procmail regular expression (defined below), and PMDIR for the variable that points to the directory that holds your Procmail-related files. Note that there are others who use this abbreviation, for example, Paul Chvostek's Procmail Log Watch awk script is called pmlw and Jari Aalto's Procmail Documentation and Procmail Library are located at pm-doc.sf.net and pm-lib.sf.net.

mail client  or  mail user agent  or  MUA

A program — such as Mulberry, pine, alpine, mutt, Opera Mail (M2), Thunderbird, SeaMonkey Suite, Outlook Express, Outlook, Mail.app, or Eudora — that a user uses to process mail messages. "Processing" includes reading, extracting or launching attachments, moving, printing, deleting, and composing & initiating the sending of messages. For more information, see the definition of MUA at FOLDOC and Email client at . For discussion about offline-, online-, and disconnected-mode IMAP clients, see this 2002-June-01 comp.mail.imap message. For a comparison of Pine, Mutt, Mulberry, SeaMonkey Suite and Thunderbird, see Comparison of Pine and Its Main Competitors on the All About Pine page.

news client  or  NNTP client

A program — such as pine, alpine, nn, slrn, Opera Mail (M2), SeaMonkey Suite, or Microsoft Outlook Express — that a user uses to process Usenet and other NNTP messages. "Processing" includes reading, extracting or launching attachments, moving, printing, canceling, and composing & initiating the sending of messages. Note that many NNTP clients can save to, read from, and process the mailboxes that Procmail delivers messages to. For more information, see the definition of news reader at FOLDOC.

server

A computer or a computer program that responds to requests from — or serves — other computers or computer programs. Whether the word “server” means “server software” or “server computer” is usually clear from the context and sometimes people explicitly say which is meant. For example, the following questions are equivalent and both are in standard parlance.

• “What IMAP server software are you running?”
• “What IMAP server are you running?”
  
  

For more information, see the definition of server and client-server at FOLDOC.

MTA  or  message transfer agent  or  mail transport agent  or  mailer

The underlying program that a mail server uses to send and receive mail messages. There are many MTAs in use today, for example CommuniGate Pro, Courier, Exchange, exim, MMDF, postfix, Post.office, qmail, sendmail, smail, and Zmailer. Note that on some systems the sending MTA is different from the receiving MTA. For more information, see the definition of MTA at FOLDOC, Cameron Laird's personal notes on message transfer agents, Unix Mail Transport Systems reviewed by JdeBP, and the definition of Message Transfer Agent at .

MSA  or  mail submission agent

To “send” a message, a mail user agent traditionally submitted the message to a message transfer agent. Nowadays most systems have a layer that sits between the MUA and MTA called the mail submission agent or MSA, and the MUA submits the message to the MSA rather than to the MTA. For details, see RFC2476, Message Submission and Gregory Neil Shapiro's Message Submission Agent (MSA). An MSA is used to authenticate the sender and, optionally, to rewrite headers, catch outgoing spam, viruses, & other vermin, and to provide an alternate outgoing port number (because, e.g., port 25 is often blocked or hijacked by Internet Access Providers).

LDA  or  local delivery agent  or  MDA  or  mail delivery agent  or  DA  or  delivery agent  or  local mailer

The program your server uses to deliver mail messages to the mailboxes of users on the system. Sometimes delivery is built into the message transfer agent (for example, sendmail's mail.local and Postfix's local) and sometimes another program, such as tmail, dmail, Dovecot deliver, maildrop, or procmail, is used as the delivery agent.

mailbox  or  mail folder  or  folder

A collection of messages stored on either a local or a remote computer. A mailbox might be a single flat file (this is the structure used by spool (mbox) and c-client MBX mailboxes); it might be a directory with each message a file in the directory (this is the structure of MH, maildir, and Netnews mailboxes); or it might use some other data structure, for example the structure used by c-client mix mailboxes. For more information, see "Appendix A: Folder Formats" in the Procmail FAQ and Before You Begin: Note About MH, maildir, and Traditional Unix Spool (mbox) Mailbox Formats below.

default inbox  or  system mailbox

Whatever you or your system administrator has specified as your primary incoming mailbox. Messages that fall through all your procmail recipes are delivered to this mailbox. To find out your system mailbox, type  procmail -v  at your Unix prompt. To tell procmail to override your default inbox, set the DEFAULT environment variable in your procmailrc. Warning: Overriding $DEFAULT may have undesired consequences with some MTAs.

MAILDIR variable  and  maildir format  (which are two completely different beasts!)

Unfortunately, the word “maildir” is used in (at least) two different ways in the Procmail world. MAILDIR (in all caps) is a Procmail variable that changes Procmail's current working directory and is used to point to the directory where your personal (non-system-spool) mailboxes are stored. For more information about Procmail's MAILDIR variable, see the blinking IMPORTANT note in Step 4 below. Maildir (not in all caps) is the name of a mailbox format. For more information about the maildir mailbox format, see Using maildir format, the maildir man page, and the section below about delivering to a mailbox that is in maildir format.

 

list  or  subscription  or  solicited bulk email  or  SBE or  solicited bulk mail  or  SBM

A mailing list that is used either for group discussion or for newsletters & announcements. For more information, see the definition of mailing list at FOLDOC and Webopedia; also see the definition of UBE (Unsolicited Bulk Email), which is an antonym of these terms.

 

recipe

A set of conditions and an action to be performed if the conditions are satisfied. Note that some programs that filter messages use a different term for this concept. For example, Pine uses the term filtering rule, Visual Office use the term rule, SeaMonkey Suite and Thunderbird use the term message filter, SmartSieve uses the term Mail Filter Rule, and Mulberry uses the terms script, trigger, & rule to encompass this concept.

 

string

A sequence of characters. Letters, digits, punctuation, symbols, tabs, and spaces are all considered characters. A substring is a string that is contained in another string. For more information, see the definition of string at FOLDOC.

 

regular expression  or  RE  or  RegEx  or  RegExp

A string in which some characters are interpreted as “wild cards” or metacharacters. Here are some metacharacters and metaphrases that can be used in a Procmail regular expression (PMRE).

Metacharacter or Metaphrase	Meaning in a Procmail Regular Expression
.	any character other than newline (i.e., other than Line Feed or LF or ASCII character 10)
(string)	treat string as a single item
*	zero or more of preceding item
?	zero or one of preceding item
+	one or more of preceding item
^	newline; usually used to match beginning of line; Note: As discussed in Re: procmail help! $MATCH grabs newline too, the meaning of the ^ metacharacter is different in Procmail than in most other RE tools.
$	newline; usually used to match end of line; Note: As discussed in Re: procmail help! $MATCH grabs newline too, the meaning of the $ metacharacter is different in Procmail than in most other RE tools.
[characterList]	any single character in characterList
[^characterList]	any single character NOT in characterList
\< \>	each of \< and \> is a metaphrase for a "non-word character"; details are in the definition of word & Matching a Word below, and in Jari Aalto's Procmail and egrep differences
|	OR; ORing in Procmail is discussed below
()	() is the null item and is always a match; it is used to make a recipe condition more readable, to escape a leading backslash (\), or to escape the special meaning of other metacharacters or metaphrases. For examples, see this discussion of ()< and this discussion of ()\/. Note that it is more common to escape a metacharacter with a backslash, which is discussed in the next item.
\metacharacter	Escape the special meaning of metacharacter and treat it as the literal character that it is;  for example  \.  means the dot character rather than “backslash followed by any character” and  \*  means the asterisk character rather than “zero or more backslashes.”  Note: The less-than character (<) needs to be escaped differently. Also see David W. Tamkin's 2004-April-24 message about leading backslash problem (was persistent lock files).  Tip: Another way to escape a metacharacter is to put it in a [characterList], e.g., you can use either \. or [.] to mean a literal dot (.) in a PMRE.  Example: In Setting Keywords or Labels below, there is a recipe condition that contains a literal $ character.
\/	extraction operator (discussed below)
\newline	If the last two characters on a line are a backslash (\) followed by newline, this is a “continuation backslash” and it tells Procmail to continue the current line with the text on the next line and ignore these characters: backslash (\), newline, white space (spaces, tabs, and/or newlines).

In Procmail, regular expressions are used in the conditions of a recipe to, e.g., look for a pattern in the Subject header of a message. For more information, see the definition of regular expression at FOLDOC, regular-expressions.info, Regular Expressions HowTo at OmniWeb, Columbia University's page on regular expressions for procmail, the Regular Expression section of era eriksson's Procmail Quick Reference, Scott Wiersdorf's Regular Expressions - Procmail Mail Filtering Syntax, manual page for regexp, the Extended regular expressions section of the procmailrc manual page, and .

	Procmail “normal egrep” regular expressions do not support all the metacharacters that are used in GNU egrep. For example, PMREs are not PCREs (perl compatible regular expressions) and PMREs do not support the limiting repetition operator {min,max}. Make sure you read the regular expression documentation that is specific to Procmail. The regular expression .* means any sequence of any characters, including the empty sequence. It is not needed at the beginning or end of a Procmail regular expression (except when you are using the extraction operator and you want to extract all characters either from the beginning of the line or through to the end of the line). A Procmail regular expression sometimes matches greedily, which means that it will match the longest string possible, and sometimes matches stingily, which means that it will match the shortest string possible. For details, see Understanding procmail's minimal matching (stingy vs. greedy) in Jari Aalto's PM Tips.

 

 

word

A substring that contains only word characters - letters, digits, and underscores - and is bounded on the left by the beginning of a line or non-word character and on the right by the end of a line or non-word character. For example, in the line

           Subject: testing procmail

the substrings Subject, testing, and procmail are each words and the substrings Subject:, mail, and test are not words. The diagram below shows the relationship between words and substrings, namely: Every word is a substring but not every substring is a word.

           +----------------+
           |  +---------+   |
           |  |  words  |   |
           |  +---------+   |
           |     substrings |
           +----------------+

Since you need to look at the boundaries of a string to determine whether it is a word, the term word only makes sense when you are talking about a substring in the context of a larger string. For more information, see the discussion about matching the word “test” in Understanding and Refining Your Procmail Recipes below.

 

pico

A friendly text editor that is the default pine composer. Throughout this article you can replace pico with a different text editor if you like. To learn more about Pico, see the pico man page and What is Pico at the University of Washington Pine Information Center.

 

~  or  $HOME  or  ${HOME} 

Your home directory. You can get home by typing any of the following:

cd
cd ~
cd $HOME
cd ${HOME}

Note: Do not use ~ in a procmailrc file; use $HOME or ${HOME} instead.
Remember:  There's no place like ~ 

 

See Also

• IMAP Service Providers: Terminology, which discusses terminology that is especially relevant when comparing mail service providers including account, user, personal mailstore, alias, subaddress, catchall address, and domain hosting.
• Power Pine: Terminology, which discusses the IMAP terms RECENT, SEEN (aka READ), and UNSEEN (aka UNREAD).
• Stopping Spam Creating a Stronger, Safer Internet: Glossary (according to this, John R. Levine wrote this Glossary).

Strategies

With procmail you can organize your environment variables, recipes, mailboxes, and directories however you like. Only the .procmailrc file, which I discuss below in Step by Step Through Setting Up and Testing Procmail, is required to have this name and to reside in your home directory. This freedom is part of what gives procmail its power but it can lead to chaos if you do not have a strategy for organizing and naming your procmail-related environment variables, files, and directories.

Simplicity

The first strategy that I recommend and try to follow in this article is simplicity. For example, I recommend that you use the Procmail defaults whenever possible and do not set unnecessary environment variables in your procmailrc file(s). For more about this, including problems caused by not using the defaults, see the Warnings () notes in Step 4 below.

Modularize and Plug & Play

A key to good programming is modularization. This, along with simplicity, will help to make your Procmail configuration portable and will make it easy to plug in, unplug, or reorder recipes. Here are examples of recipe modules you might want to use:

• rc.testing
• rc.subscriptions (aka rc.sbe or rc.bulk or rc.blue)
• rc.log
• rc.forward
• rc.rmdupes
• rc.munge-windows-1252
• rc.munge.newsgroups, which I discuss on my main Pine page in this section
• rc.autoreply
• rc.vacation
• rc.green — used, along with a greenlist (AKA “whitelist” or “accept list” or “allow list” or “goodguy list” or “trusted senders”), to do Reverse Spam Filtering
• rc.greenlister — uses an incoming message to trigger an update to the greenlist; for details see Reverse Spam Filtering: Winning Without Fighting
• rc.quarantine (aka rc.virus or rc.violet) — see Snagging Viruses below
• rc.spamassassin — see Using SpamAssassin below
• Sean B. Straw's listname_id.rc, which I discuss in Generic SBE Sorting below
• Ruud H.G. van Tol's procmail .inc (include) files, which are here & here, and the .rc and other files in his procmail directory
• Dallman Ross's vsnag.genvars.rc, which is part of Dallman's Vsnag package and about which Dallman said “the genvars contents are meant for, and I'm fine with their use for, satisfying your general private procmail needs. Ditto public, if that means a server running procmail for the good of its users, including commercially, so long as I don't lose credit for what I've offered.”
• Bart Schaefer's rfc822rc.txt, which can be used to parse RFC822 addresses; an example of using this is in this 2004-Nov-26 message by Ruud H.G. van Tol

In the Step-By-Step section below, I walk you through creating the first two recipe modules and then, after testing, I show you how to unplug rc.testing. I discuss rc.quarantine and rc.spamassassin in the Advanced Recipes section.

Use Consistent Naming and Formatting Styles

Using consistent naming and formatting styles will help you keep your directories, files, and mailboxes organized. Here are the styles that I use:

*	Notes About Mailbox Names
	Here are more details about my strategy for choosing mailbox names. These notes are in addition to the comments in the starred item (*) above. I use dash (-) rather than underscore (_) or a space ( ) as a visual separator in my mailbox names. I don't like underscores because they are a pain to type (you need to hold down the SHIFT key) and they are sometimes hard to see (they can fade into the baseline of the text and looks like a space). I don't like spaces in mailbox names because in many tools, especially command-line tools, spaces need to be escaped. Another reason to avoid the underscore (_) character in a mailbox name is discussed in the 2007-August thread with subject "-" and "_" in folder names in the dbmail mailing list. The string “INBOX” has a special meaning to IMAP servers and if you give a mailbox this name, your IMAP client might not be able to see this mailbox on the IMAP server. This is because when it sends the string “INBOX” to the server, the server interprets it to mean a special mailbox, which is not necessarily the mailbox that has the name “INBOX”. Each of the strings mbox, mail.txt, INBOX, and INBOX.MTX have a special meaning to clients and servers that are based on the University of Washington's c-client tools. It is best to not use any of these for a mailbox name, unless, of course, you want a mailbox to be interpreted in the special c-client way. The string “default” has a special meaning to the qmail MTA and it is safest not to use this as a mailbox name. The strings new, cur, and tmp are used in the internal structure of maildir-formatted mailboxes and, to avoid confusion, I recommend that you do not use any of these as a mailbox name. The strings Trash, Discard, Junk, Junkmail, Spam, and Ham have a special meaning on some systems. For example, in the default Tuffmail.com configuration, the Trash, Discard, and Junkmail folders are periodically auto-deleted; at Panix.com, “mail in the Trash folder will get thrown out automatically when it's a month old.” To ensure that you have complete control over your mailboxes, do not use any of the special mailbox names on your system. For example, I use a mailbox named MaybeSpam for my possibly-spam messages and I do not need to worry about my system auto-deleting these messages (because MaybeSpam is not a special mailbox on my system). Do not use a forward slash (/), dot (.), or backslash (\) in a mailbox name because these are used as a namespace hierarchy separator on some IMAP servers. For example, Binc IMAP and UW IMAP use forward slash for the hierarchy separator; Lotus & Exchange use backslash; and Dovecot (when using Maildir-format), Mirapoint, Courier, & Cyrus all use dot. (Note that most IMAP servers can be configured to use a non-default hierarchy separator. For example, Tuffmail.com has configured their Cyrus server to use forward slash as the hierarchy separator.) Avoiding these characters in mailbox names makes it possible for you to have a consistent naming style across different IMAP systems, and should make your life easier if your system's IMAP server changes or if you move your mailboxes to a system using a different flavor of IMAP. You might consider not using a dollar sign ($) or exclamation mark (!) in a mailbox name because some IMAP servers, for example Cyrus, do not allow these characters. However, Courier and UW IMAP do seem to allow these characters in a mailbox name. You might consider not using a caret (^) in a mailbox name because Cyrus IMAP will convert any dot (.) in a mailbox name to a caret if the hierarchy separator is changed from slash to dot. If you use ^ in your mailbox names, you will not be able to distinguish between carets that have been converted from dots and carets that have always been carets. I have been wondering if there is any problem with using tilde (~) as the 1st character in a mailbox name? If you have any thoughts about this, please post in this comp.mail.imap thread that I started 2005 Nov 18. If your mail system allows dual-use names (which are aka hybrid folders), i.e., allows a mailbox and a directory (hierarchy) to have the same name, I recommend that you not use the same name for both a mailbox and a directory. This will make your life easier if you move your mailbox hierarchy to a system that does not support dual-use names. For example, if you use UW IMAP with mix-format mailboxes, Dovecot IMAP, Cyrus IMAP, our Courier IMAP, it is possible to create a container named Archive that contains both mailboxes and messages. My advice is to use this Archive container as either a container of only mailboxes (a directory) or as a container of only messages (a mailbox). Some of the other characters and strings to avoid in mailbox names that I list above were suggested in the 2001-July-09 comp.mail.imap message Re: Mailbox Names (Was: Which IMAP Server is better?) and in the 2002-Sep-23 comp.mail.imap thread characters allowed in mailbox names. Do not start a mailbox name with the hash character (#) because some IMAP servers interpret a string that begins with # as an IMAP namespace token. This is discussed in RFC 3501, section 5.1 (Mailbox Naming), which says “Two characters, "#" and "&", have meanings by convention, and should be avoided except when used in that convention.” The next item discusses the problem with ampersand (&). You might also consider not using an ampersand (&), a space, or an upper-case character in a mailbox name for the reasons discussed in the gmane.mail.imap.dovecot thread Ampersand in Folder names causing folders to disappear?, in the gmane.mail.imap.courier.general thread IMAP Folders and the ampersand, in the EMD thread Ampersand in Folder Name, and in the FastMail.FM FAQ about How do I use FastMail.FM's subdomain addresses and "plus- addresses"?  Tip: If you use a Procmail recipe to deliver to a mailbox that contains a space in its name, you need to escape the space by preceding it with a backslash (\). Another reason not to use an ampersand (&) in a mailbox name is that it might be interpreted as a MUTF7 string, rather than an ASCII string, by an IMAP server or client. Some systems have problems with a special mailbox name — such as SENT, DRAFTS, SPAM, and TRASH — if it is created in a non-standard way, e.g. via procmail. This is discussed in this and this thread at EmailDiscussions.com.

My Latest Mailbox-Naming Experiment

Request:  I'm revising my mailbox naming style and I'd like to learn about names that other people use, especially any mailbox-naming scheme that is in common use. I'm thinking about changing the prefix of my incoming mailbox names from IN- to something that will sort above all alphabetic characters in an ASCII sort. I'm leaning towards prefixing these box names with a dash (-), so, for example, incoming procmail mailing-list messages would be put in --procmail. And then I'd put the archive of that mailbox in zz-procmail-YYYY-MM-DD or ==procmail-YYYY-MM-DD because the leading zz- or == will make it sort below my incoming mailboxes. I realize that these are kind of odd-looking names so I'm interested to hear what ideas other people have — please let me know what you think and if you have suggestions.

Note 1:  Someone emailed me and mentioned that if a mailbox name starts with a dash, it is a pain to manipulate from a Unix shell. That is true, but it provides a bit of security if a lamer breaks into your account and tries to read your mailboxes using shell commands such as more or less or pine -f mailboxName (and it's not that much of a pain if you know the trick)!

Note 2:  I'm now using this naming scheme and so far I haven't had problems with Procmail or with any mail client or IMAP server that I've used. For example, here are mailbox names that could be used with my reverse-spam-filtering system:

  --magenta		
  --violet		
  --blue
  --green
  --limegreen		
  --yellow
  --red
  ==archive1		
  ==archive2		
		

To force my most important incoming mailboxes to the top of an ASCII sort, I prefix each mailbox name with an appropriate number of dashes. For example:

  -------green
  ------limegreen		
  -----yellow
  ----blue
  ---red
  --violet		
  -magenta
  ==archive1		
  ==archive2
			

I actually use a modified version of this and don't really have a mailbox name that begins with seven dashes! If you are interested in why I name my incoming mailboxes after the colors of the rainbow, see Reverse Spam Filtering: Winning Without Fighting.

Ordering Procmail Flags

I recommend that you order Procmail flags using the style that Jari Aalto suggests on his Procmail Tips page in the section called The order of the flags. I use this style throughout this page, for example in the SpamAssassin recipes below.

Managing Spam

A lot of people are learning Procmail because they want to automatically separate non-spam messages from spam messages. My strategy is to do the following, in this order:

1. Have my mail hosting provider do SMTP-level spam rejection using, for example, Greylisting, DNSBLs, or SPF (Sender Policy Framework).
2. Quarantine or delete viruses.
3. Deliver or forward known non-spam messages, possibly also setting appropriate flags/labels/keywords.
4. Pipe the possibly-spam (not known non-spam) messages through a spam-detection tool, such as SpamAssassin, to assign a spam score or spam probability.
5. Set appropriate flags/labels/keywords on possibly-spam messages.

To implement step number . . .

1. see IMAP Service Providers page, which describes over 400 providers, many of whom offer SMTP-level spam rejection.
2. see Snagging Viruses below.
3. see some or all of the following:
   1. Step by Step Through Setting Up and Testing Procmail below, which includes recipes that catch and deliver specific “Solicited Bulk Email” (mailing-list and newsletter) messages.
   2. Alternatives to the ^TO_ Macro below, which includes more recipes that catch and deliver specific Solicited Bulk Email messages.
   3. Generic SBE Sorting below.
   4. Setting Keywords or Labels below.
   5. Minimizing False Positives With Greenlists and Bluelists below.
4. see Using SpamAssassin below.
5. see Setting Keywords or Labels below.

For more about my spam-deflexion strategy and about Unsolicited Bulk Email (UBE or spam), see my Reverse Spam Filtering: Winning Without Fighting page.

For other people's spam-deflexion strategies, see the December-2005 gmane.mail.procmail discussion about Spam and Procmail, which takes place in these two threads. Especially useful is this message from G.W. Haywood, in which he describes the system that he uses on his mail servers and says:

I am collecting Spam-related links at del.icio.us/Deflexion.com/Messaging/Spam.

Before You Begin: Note About MH, maildir, and Traditional Unix Spool (mbox) Mailbox Formats

Overview

The step-by-step instructions below guide you through setting up Procmail to deliver to mailboxes that are in traditional Unix mail spool format (mbox). To deliver to a mailbox that is in maildir format, you need to append a forward slash (/) to the mailbox name. The table below gives examples of the various types of mailbox specifications that can be used in the action line of a Procmail recipe.

	If Procmail tries to deliver a message to a maildir, MH, or mbox mailbox that does not exist, Procmail will create the mailbox in the appropriate format and then deliver the message to it. If a maildir or mbox mailbox does not exist, it is safest to let Procmail create it (see previous item) rather than trying to create it yourself using Unix commands. In order for Procmail to successfully deliver to a pre-existing mailbox, the mailbox must be in the appropriate format. For example, a maildir or MH mailbox must be a directory and a Unix mbox mailbox must be a file. If you deliver to maildir mailboxes, use procmail 3.15.2 or 3.22+.  3.14 has serious problems delivering to maildir mailboxes and 3.15, 3.15.1, 3.20, and 3.21 have minor problems. (*) If you use MH mailboxes, you will probably want to use the MH/nmh rcvstore program to deliver messages. Details are in Beedub's Wiki in the section Using the procmail program. If you want to deliver to another type of mailbox, consult the relevant documentation. For example, to deliver to . . . Dovecot IMAP mailboxes, see Gerhard Wiesinger's 2007-March-17 message titled Re: deliver to subfolder with dovecot's LDA - procmail filtering at dovecot.org or at gmane.org. (Note: Gerhard's top page is a cool example of DHTML and, if you have JavaScript enabled in your browser, I recommend you check it out!) Cyrus IMAP mailboxes, see Nicholas Petreley's 2002-March-18 article titled How to set up IMAP on the cheap: How to set up Cyrus, Postfix & Procmail to block spam at LinuxWorld.com or at Archive.org. C-client mbx or mix mailboxes, see Jauder Ho's Implementing IMAP with UW imapd. Jauder's report discusses dmail, which is part of the UW IMAP Toolkit. Also see Jauder's procmailrc and Faisal N. Jawdat's Using mbx Format Mail Files, each of which includes examples of using dmail in Procmail recipes. Note the following, which is on Faisal's page: “The dmail command understands normal unix [mbox] mailboxes as well as mbx [and mix] ... so it's probably wise to migrate all your procmail recipes to using dmail before converting to mbx [or mix] format”

Details

An explanation of how Procmail delivers to various folder formats is in the latest (3.22) procmailrc man page. Here is the relevant section.

  [ . . .]
  Anything else will be taken as a mailbox  name  (either  a
  filename  or a directory, absolute or relative to the cur-
  rent directory (see MAILDIR)).  If it is a  (possibly  yet
  nonexistent) filename, the mail will be appended to it.
  
  If  it  is  a  directory,  the mail will be delivered to a
  newly created, guaranteed to be unique file named $MSGPRE-
  FIX* in the specified directory.  If the mailbox name ends
  in "/.", then this directory  is  presumed  to  be  an  MH
  folder;  i.e.,  procmail will use the next number it finds
  available.  If the mailbox name ends  in  "/",  then  this
  directory  is presumed to be a maildir folder; i.e., proc-
  mail will deliver the message to a file in a  subdirectory
  named  "tmp"  and  rename  it  to be inside a subdirectory
  named "new".  If the mailbox is  specified  to  be  an  MH
  folder  or maildir folder, procmail will create the neces-
  sary directories if they don't exist,  rather  than  treat
  the  mailbox as a non-existent filename.  When procmail is
  delivering to directories, you can specify multiple direc-
  tories  to  deliver  to  (procmail  will  do  so utilising
  hardlinks).

When delivering to a maildir or MH format mailbox, you do not need to use a lock file so the first line of a recipe does not need the second colon. For example, instead of using this as the first line:

   
:0:   

you can use this:

   
:0   

There are examples of recipes that deliver to maildir-formatted mailboxes in Step 7D and in Using SpamAssassin below.

Details Specific to maildir Format

	If you want your system mailboxes $ORGMAIL and $DEFAULT to be in maildir format, it is best to specify this in the procmail source code and recompile procmail. This is discussed in the following messages, which were posted to the Procmail mailing list. Philip Guenther's message Re: Maildirs & "Couldn't rename bogus..." David W. Tamkin's message Re: Maildir format

If you are delivering messages to maildir-formatted mailboxes that will be served to mail clients by Courier IMAP (which supports only maildir format mailboxes), put a dot (.) in front of the mailbox name in your Procmail recipe. This leading dot is a hierarchy separator and will ensure that IMAP clients can see these mailboxes. IMAP clients will display the mailbox name without the leading dot (or the trailing slash, which is just used to tell procmail the format of the mailbox and is not part of the name that's used on the Unix system).

Technical Note:  On Courier IMAP, private mailboxes reside under the “INBOX.” hierarchy. On the server, this hierarchy is usually in the directory $HOME/Maildir but users do not need to tell their IMAP clients the actual name of this directory because the IMAP protocol will pass that information to the IMAP client.

|preline /usr/local/bin/procmail ~/.procmailrc || exit 111
|preline procmail -m -p myprocmail.rc
|preline procmail

See Also

• Sam Varshavchik's message Re: Adding keywords to incoming mail in gmane.mail.imap.courier.general, in which he says “With procmail [delivering to a maildir-formatted mailbox], ordering of new messages is likely to be completely random.”
• wiki.dovecot.org: Questions And Answers -- the first few questions include useful information about Procmail and maildir-formatted folders
• era eriksson's Procmail FAQ: Appendix A -- Folder Formats
• Using maildir format by D. J. Bernstein
• maildir man page
• Courier IMAP Server
• David W. Tamkin's message Re: /var/spool/mail, MailDir and quota in the Procmail list
• Sam Stern's message Re: Maildir... in the Procmail list
• Philip Guenther's message Re: Rerun procmail filters on maildir folder
• Bart Schaefer's 2004-Dec-11 message Re: cannot create /IMAPdir//INBOX/: is a directory, which begins with this:

  “Firstly, the word MAILDIR as a variable name in procmail predates, and has absolutely nothing to do with, the mail storage format called "Maildir" which was invented quite some years later.
  
  Secondly, . . .”

• Step 7D and Using SpamAssassin below, each of which includes examples of Procmail recipes that deliver to maildir-formatted mailboxes

Step by Step Through Setting Up and Testing Procmail

These 18 steps walk you through ensuring that procmail is installed on your system, setting it up to sort your incoming mailing-list messages, and testing your setup.

Preliminaries

	Important:  For the following steps to work, you must be a regular user, not root. If you are a system administrator who is learning Procmail, I highly recommend that you step through this procedure before you try to set up a global procmail rc file. You are much more likely to understand the basics of Procmail if you learn by doing (rather than by reading only). And your system & users will be much less likely to experience an email nightmare if you do your Procmail learning — and making mistakes — as a regular user!

0. To find out if procmail is on your system and, if it is, what the path to it is, type one of the following commands.

    

   Shell	Command
   csh, tcsh	which procmail
   sh, bash, ksh	type procmail
   various	whereis procmail
   various	where procmail
   various	locate procmail

   Make a note of the path to procmail because this is needed in Step 9 below.
   
   
   

1. To find out procmail's version, locking strategies, default rcfile, and your system mailbox (aka your default INBOX), type:

   procmail -v 
   

   	If you are using a version earlier than 3.11pre4, which was released on 29 October 1995, you need to use the procmail expression ^TO rather than the expression ^TO_ in Step 15. If you are using version 3.13 or earlier, procmail cannot deliver to maildir mailboxes (unless you apply a patch to procmail or pipe a message to a program that can deliver to a maildir mailbox). If you deliver to maildir-formatted mailboxes, it is best to use procmail 3.15.2 or 3.22+.  3.14 has serious problems delivering to maildir mailboxes and 3.15, 3.15.1, 3.20, and 3.21 have minor problems.

   
   
   If your system does not have procmail or does not have the latest version, 3.15.2 or 3.22+, ask your system administrator to install it. You can find out what is the latest version of procmail and where to get it at www.procmail.org.
   
   
   
   
2. Rename (move) ~/.forward, ~/.procmailrc, and the global procmailrc, if they exist. This will ensure that you do a clean Procmail setup.
   1. To find out if ~/.forward exists and what its contents are if it does, type:

      cat ~/.forward

      If ~/.forward exists, rename it using a command similar to this:

      mv -i ~/.forward ~/.forward-2007-11-27

      This will ensure that you use the optimal ~/.forward file for your system, which is discussed in Steps 8 and 9 below.
      
      
      
   2. To find out if ~/.procmailrc exists and what its contents are if it does, type:

      cat ~/.procmailrc

      If ~/.procmailrc exists, rename it using a command similar to this:

      mv -i ~/.procmailrc ~/.procmailrc-2007-11-27

      Steps 3 and 4 (in the next section) walk you through setting up a minimal ~/.procmailrc file. If you would like to use some of the recipes that are in your old procmailrc file, I recommend that you call them by using an INCLUDERC command in your new minimal ~/.procmailrc. For an example of an INCLUDERC command, see the end of Step 4 below (in the #### Processing Section ####).
      
      
      
   3. To find out if a global procmailrc exists and what its contents are if it does, type:

      cat /etc/procmailrc
      cat /usr/local/etc/procmailrc
      

      If a global procmailrc exists and if you are the administrator of your system, rename it using a command similar to this:

      sudo mv -i /etc/procmailrc /etc/procmailrc-2007-11-27
                 ^^^^^^^^^^^^^^^
                change to the path
                  on your system

      After you have experience setting up and using a personal procmailrc, you can, if you want, re-install this renamed global procmailrc. Details about using a global procmailrc are in See if There is a Global procmailrc below.
      
      Note: As discussed here, viaVerio VPS accounts use /usr/local/etc/procmailrc for the global procmailrc file. Also note that you can view the VPS default files and directory structure in the /skel directory. (Verio is where I host the Infinite Ink web pages, including this page.)
      
      Warning: As discussed in this December-2005 gmane.mail.procmail thread, it seems that there is a Linux distribution that “out of the box” includes a broken /etc/procmailrc.
      
      
      

Set Up Your Procmail Files for Testing

3. Create ~/.procmailrc by typing:

   cd
   pico -w .procmailrc
   

   	Throughout this article I use pico for editing files. Pico is the Pine composer and is a simple easy-to-learn text editor that is available on most Unix-type systems. Pico's -w command-line argument tells pico to not wrap lines, that is, do not insert extra line breaks. To learn more about Pico, see the pico man page and What is Pico at the University of Washington's Pine Information Center. You can, of course, replace pico with your favorite text editor. In any case: Make sure your editor creates Unix ASCII text files with Line Feed (LF) line terminators. For details about this, see Step 10 in this Step-by-Step section and see Make Sure Your Configuration Files Use LF (Line Feed) Line Terminators in the Troubleshooting section below. On some systems the procmailrc file has a non-standard name or non-standard location. For example: Sonic.net uses both $HOME/.procmailrc-first and $HOME/.procmailrc. Verio Signature accounts use $HOME/etc/users/UserID/.procmailrc for sub-users (i.e., users other than the primary user). This is discussed in the message Procmail on Signature in the viaVerio Reseller Forums (registration required). Make sure you read the mail-filtering documentation for your system before you create your procmail files. Some people will do anything to avoid the “satanic syntax” of Procmail, but others embrace—and even promote—it !  :-)

   
   

 

4. Enter the following in your ~/.procmailrc either by typing it or by using copy & paste*. But first note these warnings.
   

   	Procmail does not support the expansion of tilde (~). You must use $HOME rather than the ~ character within your .procmailrc. This is documented in the procmailrc man page. Make sure you replace  MAILDIR=$HOME/Msgs  with the directory where your personal (non-system-spool) mailboxes are stored and that this directory exists and is writable by your LOGNAME. The MAILDIR variable specifies Procmail's working directory and all relative paths are relative to this directory. When you first test Procmail, do not set the PATH variable. As Philip Guenther said in this message, “you shouldn't alter PATH unless there's something actually missing from it or the order is wrong.” As I discuss in Simplicity above, do not explicitly set any variable unnecessarily. In addition to not explicitly setting the PATH variable, which is discussed in the previous item, it is almost always best to not explicitly set the DEFAULT variable. For examples of problems caused by explicitly setting variables, see: 2006-Jan thread about procmail with mutt 2005-July thread about procmail is creating Maildir files instead of mbox 2005-July thread about Can't change the Subject: The first two of these threads discuss problems caused by explicitly setting DEFAULT and the third thread discusses a problem caused by setting a user-invented variable named FORMAIL. To learn about the Procmail default environment variables on your system, see man procmailrc on your system.

   
   
   

   # Note: Anything after a hash character (#) is a comment & is ignored by procmail.
    
   #### Begin Variables Section ####
        
   # It is essential that you set SHELL to a Bourne-type shell if
   # external commands are run from your procmailrc, for example if
   # you use rc.spamassassin, rc.quarantine, or other advanced recipes.
   # Setting SHELL should not be needed for the simple sorting recipes in 
   # this step-by-step section, but to be safe and to future proof your
   # procmailrc, set it anyway! Details are in Check Your $SHELL and $PATH.
   SHELL=/bin/sh
     
   # Directory for storing procmail configuration and log files
   # You can name this variable anything you like, for example
   # PROCMAILDIR, or don't set it (but then don't refer to it!)
   PMDIR=$HOME/Procmail
   
   # LOGFILE should be specified ASAP so everything below it is logged
   # Put ## before the next line if you want no logging (not recommended)
   LOGFILE=$PMDIR/pmlog
     
   # To insert a blank line between each message's log entry in $LOGFILE, 
   # uncomment the next two lines (this is helpful for debugging)
   ## LOG="
   ## "
     
   # Set VERBOSE to yes when debugging; VERBOSE default is no
   ## VERBOSE=yes
     
   # Replace $HOME/Msgs with the directory where your personal (non-system-spool) mailboxes reside 
   # Mailboxes in maildir format or served by Courier IMAP are often in $HOME/Maildir
   # Mailboxes served by UW IMAP are sometimes in $HOME, sometimes in $HOME/mail, & sometimes elsewhere
   # MAILDIR default is the value of $HOME
   # Make sure that $MAILDIR exists and that it is a directory!
   MAILDIR=$HOME/Msgs   
     
   # IMPORTANT:
   # * On most systems $MAILDIR is a subdirectory of $HOME
   # * Upon reading a line that contains MAILDIR=
   #     Procmail does a chdir to $MAILDIR
   #      ...and $MAILDIR becomes the Procmail working directory
   #       ...and relative paths are relative to $MAILDIR
   # * Do not include a trailing slash in your MAILDIR setting
   # * The $MAILDIR directory must exist and must be writable by your LOGNAME
   # * The MAILDIR variable is an entirely different entity from maildir mailbox format
        
   #### End Variables Section; Begin Processing Section ####  
     
   INCLUDERC=$PMDIR/rc.testing
   INCLUDERC=$PMDIR/rc.subscriptions
     
   # Messages that fall through all your procmail recipes are delivered
   # to your default INBOX. To find out yours, run 'procmail -v' 
    
   #### End Processing Section #### 
   # EOF
   

    
   *Copy & Paste Tip: A concise version of the above procmailrc is in Step 17 below. If you have problems pasting this into the vim editor, see Vim Tip #330: how to stop auto indenting at vim.org.
   
   
   
5. To create the directory where you will store your procmail-related files (this is $PMDIR that you set in Step 4), type:

   cd
   mkdir Procmail
   

6. Create an rc file for testing by typing:

   cd Procmail
   pico -w rc.testing

7. Enter the following 5 lines in ~/Procmail/rc.testing:

   :0:
   * ^Subject:.*test
   IN-testing
    
   # EOF

    
    
   

   	The first line contains a zero (0), not the letter O. For now do not worry about the meaning of this recipe; it is explained in Dissecting a Recipe in the Understanding and Refining Your Procmail Recipes section below. Log File and Mailbox Creation If a log file or mailbox does not exist, Procmail will create it. It is usually best to let Procmail create a non-existent mailbox rather than creating it yourself using an external tool. Procmail will create a log file or mailbox using a umask of 077, unless you or your system administrator changed the Procmail default umask, for example, by setting the UMASK variable in a procmailrc file. IN-testing is a traditional mbox-format mailbox and is located in your $MAILDIR directory ($HOME/Msgs in this example), which you defined in Step 4 above. Important: Procmail will not create the $MAILDIR directory, so make sure that it already exists. However, as discussed in the previous item (B), Procmail will create the IN-testing mailbox if it does not exist. To deliver to a maildir-format mailbox that will be served by a Courier IMAP server, use the following recipe instead of the mbox recipe in Step 7 above. # In the next line, note the missing 2nd colon (:), which means "do not use a lockfile" :0 * ^Subject:.*test .IN-testing/ # In the previous line, note the leading dot (.) and trailing slash (/) Details about this and other mailbox formats are in Before You Begin: Note About MH, maildir, and Traditional Unix Spool (mbox) Mailbox Formats above. Important maildir Note 1:  Do not specify the cur, new, or tmp subdirectories because these are dealt with automatically by Procmail (as long as there is a slash (/) after the mailbox name in the Procmail recipe). Important maildir Note 2: If you want your default inbox to be a maildir-format mailbox, it is best to specify this in the procmail source code and recompile procmail. Important maildir Note 3: You must be using Procmail version 3.15.2 or version 3.22+ for Procmail to be able to correctly create and deliver to maildir-formatted mailboxes.

Set Up Your .forward File (not required on most systems)

In 1994 when I wrote the Filtering Mail FAQ, which is what this document evolved from, most Unix systems used sendmail as the message transfer agent (MTA) and Procmail was invoked via the .forward file. Nowadays, on most systems, the .forward file is not used to invoke procmail. Instead, either:

• The MTA is already set up to use Procmail as the local delivery agent (LDA) and if ~/.procmailrc exists, it will be used (regardless of the existence of ~/.forward).
  
  This is the case for the default Debian Linux, which uses exim as the MTA, and the default RedHat Linux (since at least version 4.1), which uses sendmail as the MTA. According to this 2007-March-18 comment in my blog from Jim Syler (aka Calion), Mac OS X uses Procmail as the default LDA for fetchmail, but does not use Procmail as the default LDA for postfix.
  
  
• Or, the MTA does not respect the .forward file and calls procmail using some other method. For example:
  • The qmail MTA uses a dot-qmail file (discussed above) to pass incoming messages to procmail.
  • DreamHost, which I discuss on my IMAP Service Providers page, uses ~/.forward.postfix to pass incoming messages to procmail.

	Philip Guenther, the current Procmail maintainer, recommends that you first try testing your .procmailrc without a .forward file -- that is: Skip Steps 8 and 9. If that does not work, come back here and create a .forward file. If creating a .forward file also does nt work, ask your system administrator what is your MTA and what you need to do to set up Procmail as your LDA. A default Verio VPS2 account does not have procmail installed. To install procmail on a VPS2 system, you need to run vinstall procmail-lda. Details are in Procmail - E-Mail Sorting and Filtering Utility at viaVerio.com.

8. [not required on most systems] Create a ~/.forward file by typing the following. (Pico's -w command-line argument tells pico not to auto-wrap lines, that is, don't insert extra line breaks.)

   cd
   pico -w .forward
   

9. [not required on most systems] Enter the following in your ~/.forward:

   "|/absolute/path/to/procmail"
   

   The vertical bar (|) is a pipe and is typed as SHIFT-backslash on a standard US keyboard. Make sure you replace /absolute/path/to  with the correct path for procmail (see Step 1) and include both the leading and trailing double quote mark (").

	If sendmail is configured to use smrsh (sendmail restricted shell), you will only be able to invoke the version of procmail that is listed as an smrsh-allowed program. For more about this, see What are some typical installation problems? in era eriksson's Procmail FAQ and Q3.34 -- What does "foo not available for sendmail programs" mean? in the Sendmail FAQ.

Check File Types and Set File & Directory Permissions

10. Make sure that .forward (if it exists), .procmailrc, and rc.testing are in Unix ASCII text format by typing

    cd
    file .procmailrc
    file ~/Procmail/*
    file .forward
    

    For each of these that exist, the response should be ASCII text or ASCII English text. If it is anything else, for example ASCII text, with CRLF line terminators, go to Make Sure Your Configuration Files Use LF (Line Feed) Line Terminators in the Troubleshooting section below and fix this problem.
    
    
11. Starting with version 3.14, Procmail will not trust your .procmailrc if either it or your home directory is either group or world writable. Also, on some systems your .forward file, if it exists, needs to be world-readable & not executable and your home directory needs to be world-searchable in order for the message transfer agent to “see” the .forward file. To set these permissions, type the following:

    cd
    chmod go-w,a+x .
    chmod go-w .procmailrc
    chmod 644 .forward
    

    If you skipped Step 8 and 9, you obviously don't need to chmod your non-existent .forward file. (But it won't hurt if you do!)
    
    To check these file and directory permissions, see Check File and Directory Permissions in the Troubleshooting section.

	Once your .forward (if it is needed to invoke procmail) and .procmailrc files are in place, procmail is alive. To kill it, see Pulling the Plug on Procmail below.

Test Your Procmail Setup

12. Send yourself two test messages: one that contains procmail testing in the Subject and one that does not contain the string test in the Subject.
    
    
13. Use your mail client, such as pine or mutt, to check that the messages were delivered correctly. The one with procmail testing in the Subject should be in the mailbox $MAILDIR/IN-testing and the one without test in the Subject should be in your default inbox. Ways to view your mailboxes are described in Accessing Your Mailboxes below.

If these two messages were not delivered correctly, see Troubleshooting below.

Create Recipes to Sort Solicited Bulk Email

14. After you successfully test procmail, create rc.subscriptions, or rc.sbe if you prefer that name, for sorting solicited bulk email (newsletters and mailing lists) into mailboxes. To do this, type:

    cd
    cd Procmail
    pico -w rc.subscriptions
    

15. In rc.subscriptions create a recipe, like the two example recipes below, for each mailing list to which you are subscribed.

    :0:
    * ^TO_procmail@lists.rwth-aachen.de
    IN-S-procmail
      
      
    :0:
    * ^TO_vim@vim.org
    IN-S-vim
    

    The first recipe puts messages that are sent to the procmail mailing list into a mailbox named IN-S-procmail and the second recipe puts messages that are sent to the vim mailing list into a mailbox named IN-S-vim. The following table describes the meaning of the second recipe.

    Notation	Meaning
    :0	Begin a recipe
    :	Use a lock file
    *	Begin a condition
    ^TO_	Look for a header that begins with To, Cc, Resent-To, or another destination specification, followed by any or no characters, followed by an email address that begins with the string...
    vim@vim.org	vim@vim.org
    IN-S-vim	If there is a match, put the message in the mailbox $MAILDIR/IN-S-vim and stop processing this message; if there is not a match, pass the message to the next recipe

    
    
    
    

    	Important Notes About Procmail's  ^TO_  Macro
    	^TO_ is not a normal regular expression; it is a special procmail expression designed to catch an email address — or the beginning of an email address — that is in any destination header (To, Cc, Resent-To, etc.). For details, see the MISCELLANEOUS section of the procmailrc man page and Jari Aalto's TO_ Macro Deciphered. Sometimes the actual destination address is not in any destination header, for example if the message was Bcc'd (blind courtesy copied). For more about this, see the following sections below: Important Warning: Procmail Cannot Reliably Route Users' Messages in a Virtual-Domain Dropbox Alternatives to the ^TO_ Macro To use the special ^TO_ expression, you need to be using at least procmail version 3.11pre4 (see Step 1 for how to check the version). If you are using an earlier version, replace ^TO_ with ^TO, which works essentially the same as ^TO_ but is not as precise. Do not put a space between the caret (^) and TO or between TO and the underscore (_) in ^TO_. Both the letter T and the letter O must be capitalized in ^TO_. Do not put a space or any character between ^TO_ and the email address you are matching on; it must be ^TO_beginning-of-address or ^TO_fully-qualified-address. An exception is if you are using Procmail's extraction operator (\/) to assign the address to a variable. Do not put .* or left angle bracket (<) or both (.*<) immediately after the underscore in ^TO_. Equivalents of these are built into the ^TO_ expression. For an exception to this (and an excellent use of the word "eviscerate"), see the discussion between David W. Tamkin and Martin Ward in the message Re: ackmail.rc not acking correctly? in the Procmail list. Most of the time ^TO_ is better to use than ^TO. For exceptions to this, see Ruud H.G. van Tol's Re: Error on recipe, and the discussion between Martin McCarthy and Nuno Teixeira in the message Re: Newbie question: filters. The expression ^TO_address will match on any of these strings: address, foo+address, bar+address, anything+address, etc. For details, see this 2004-Nov-25 message and the rest of the thread.

     

    	Put the busiest mailing lists at the beginning of your rc.subscriptions file so those messages will be caught and delivered before they have been checked by a lot of recipes. This both speeds up processing and makes it less likely that a little-used recipe might accidentally match on a message that would better be handled by a more frequently used recipe. I discuss alternate — and more accurate — ways to sort mailing lists in Alternatives to the ^TO_ Macro below.

     

16. Repeat the procedure given in Test Your Procmail Setup (Steps 12 and 13) to make sure things are still working.
    
    
17. "Unplug" rc.testing 1) so that it won't catch non-testing messages that contain the string test in the Subject and 2) to save processing power. To turn the line into a comment, put # at the beginning of that INCLUDERC line. Your .procmailrc will now look like the following (possibly with the other comments that I include in Step 4 above or your own comments).

    #### Begin Variables Section ####
      
    # Next line is needed if programs are invoked from your procmailrc
    SHELL=/bin/sh
    			
    PMDIR=$HOME/Procmail      # Make sure this directory exists!
    LOGFILE=$PMDIR/pmlog
    ## LOG="
    ## "
    ## VERBOSE=yes
    MAILDIR=$HOME/Msgs        # Make sure this directory exists!
      
    ####  End Variables Section; Begin Processing Section  #### 
      
    ## INCLUDERC=$PMDIR/rc.testing  
    INCLUDERC=$PMDIR/rc.subscriptions
     
    # Messages that fall through all your procmail recipes are delivered
    # to your default INBOX. To find out yours, run 'procmail -v'
     
    #### End Processing Section #### 
    # EOF
    

Tips for Managing Your Procmail Files

• Leave the commented out LOG (2 lines), VERBOSE, and rc.testing lines in your ~/.procmailrc so you can use them for future debugging or experiments.
  
  
  
• Tips for managing your Procmail log files are in Tracking Your Incoming Messages below and in Scott Wiersdorf's Log file rotation made easy (man savelogs).
  
  
  
• To make it easy to see and manage all my Procmail-related files from one directory, I create a symbolic link in my ~/Procmail directory that points to ~/.procmailrc . You can do this by typing:
  

  cd ~/Procmail
  ln -s ~/.procmailrc L.procmailrc
  ls -l
  

  Now when you are in your ~/Procmail directory, L.procmailrc is a “shortcut” or “alias” for ~/.procmailrc. If the file /etc/procmailrc or the directory /etc/procmailrcs exist on your system, I suggest that you also make symbolic links to those so that when you are in your Procmail directory you'll remember that those exist. You might also want to make a symbolic link to other mail-related files and directories, such as the directory where your mailboxes are stored. To learn more about symbolic links, see symbolic link at FOLDOC, man symlink, man ln, and Lew Pitcher's The difference between hard and soft links. For another example of symbolic linking, see Using Multiple Clients to Access Your Mailboxes below.

	Do not try to deliver mail to a symbolic or hard link. Because of potential locking problems, a mailbox that Procmail delivers to should be an actual file and the $ORGMAIL mailbox, which is discussed in the Troubleshooting section below, must not be a symbolic link or a file with more than one hard link.

Accessing Your Mailboxes

After procmail delivers messages to your mailboxes, you can access your mailboxes using

• a local mail client, such as pine or mutt, or a local NNTP client, such as nn, that runs on the system where your messages are delivered and accesses your mailboxes using local file system calls.
  
  
• local Unix tools such as grep, more, and less.
  
  
• a local or remote messaging client that can access your mailboxes using IMAP, the Internet Message Access Protocol (assuming your mailboxes reside on a server that supports IMAP access). This can be a most satisfactory approach, since it lets you run a PC program such Mulberry, SeaMonkey Suite, Outlook Express, or PC-Pine, and still get all the benefits of procmail sorting and message processing on your Unix host.
  
  
• a local or remote tool for searching through mailboxes

If you are looking for an IMAP service provider that has procmail installed, see my list of Free and Reasonably-Priced IMAP Service Providers -- most of the providers in the list that give Unix shell access also have procmail installed.

Mailbox Locking

If you use a mail or news client to change a mailbox that is receiving messages — for example, to delete a message — make sure that both procmail and your client use the same locking scheme. To determine what locking strategies are used by your installation of procmail, and the order in which these locking strategies are used, run:

procmail -v

and look at the value of “Locking strategies.”

To learn about file locking, see:

• 11.4 How does folder locking work? and questions 11.5, 11.6, & 11.16 in the UW Pine Information Center's Info for systems administrators, developers, etc. This is useful to read even if you do not use Pine.
• File locking in the Era's Procmail FAQ
• 4.11 Local lockfile usage and 4.12 Global lockfile in Jari's Procmail Tips
• The thread "Lock files - confused" in the Procmail mailing list archives
• Configuring Netscape Mail on Unix by Jamie Zawinski — this is useful to read even if Netscape is not your mail client.

	Make sure that you are not delivering to mailboxes in Unix spool (mbox) format on an NFS or any other network file system since locking does not help on those systems. For a possible workaround to this, see the comp.mail.imap message Re: UoW IMAP, lock files and multiple servers by Mark Crispin. The Netscape mail client does no locking of its local mailboxes, meaning that there is no safe way to deliver directly to them. Kmail seems to be flaky about locking local mailboxes. For details, search the KDE Documentation for procmail.

Commands for Specific Clients

Here are instructions for how to use some clients to open a mailbox that is in that client's default mailbox directory. (Please send me information about other mail and news clients!)

If a client does not use a lock file, you should not use that client to change the mailbox while it is receiving messages. To learn more about your mail or news client, see your client's documentation. I describe how to set up Pine mailbox collections in Setting Up Local and Remote Folder Collections in my Power Pine.

Using Multiple Clients to Access Your Mailboxes

Most Unix mail clients store mailboxes in a directory named either "Mail" or "mail" and most Unix NNTP clients use a directory named "News." If you want to use more than one client to access your mailboxes (or make it easy to do that in the future), you can use the Unix ln command to link the directory where your mailboxes are stored to the names of the mailbox directories used by other clients. Here are instructions for merging the mailboxes in your existing mailbox directories into a new directory named Msgs, and then linking this directory to the names mail, Mail, and News. In Steps 3-5 below replace mail with the name of one of your existing mailbox directories.

Of course, you can also link the Msgs directory to any other name that a program uses for its mailbox directory. Now all your mailboxes are in one place and you can easily save to and access them from different news and mail clients.

	The above procedure works only if your mailboxes are in a single-file format, such as traditional Unix mbox format, and your mail and NNTP clients can all access the same mailbox formats.

	In Step 3 above, I put slash dot after the target directory name (i.e., Msgs/.) to ensure that a directory named Msgs rather than a file named Msgs is the target.

Tracking Your Incoming Messages

The Procmail log is located in $LOGFILE, which, if you used the instructions in Step 4 above, is $HOME/Procmail/pmlog. The contents of $LOGFILE depend on the values of $VERBOSE, $LOG, $LOGABSTRACT, and $TRAP, which you can read about in the Environment section of the procmailrc man page. You can use many commands to view the log including  cat, more, less , and my favorite, tail, which I discuss in the next section.

Following Your Log with tail -f

If you want to continually follow your log, you can use

tail -f $HOME/Procmail/pmlog

To start tailing with the last 50 lines of the log, use

tail -n 50 -f $HOME/Procmail/pmlog

which on my system is equivalent to

tail -50 -f $HOME/Procmail/pmlog

To quit live monitoring your log, type

CTRL-C

If you want to be able to run other commands while the tail is happening, use & to put it in the background:

tail -f $HOME/Procmail/pmlog &

To learn about tail, see man tail.

Another tail option is to use Paul Chvostek's ProcMail Log Watch (pmlw), which is an “awk script that tails your procmail log file, summarizing results and giving you basic traffic statistics, live.”

Summarizing Your Log with Mailstat

You can use mailstat, a script that is part of the procmail package, to get a concise summary of your procmail log file. To see if it is on your system type one of the following commands:

which mailstat
type mailstat
whereis mailstat
where mailstat
locate mailstat

If mailstat is on your system, type:

mailstat $HOME/Procmail/pmlog

This moves pmlog to pmlog.old and displays a concise description of the contents that looks like the following:

  Total  Number Folder
  -----  ------ ------
   9964    3    IN-S-procmail
   3408    1    IN-S-vim
  -----  ------
  13372    4

You may want to put the mailstat command in your .login or .profile so that each time you log in you will see a list of how many messages you have received since the last time you ran mailstat, and what mailboxes these messages were delivered to. Or, if your system lets you use the cron command to run programs on a regular schedule, you could automatically run mailstat once a day or once a week and either save the results in a file or mail the results to yourself. A typical cron command line to collect the stats at 1:00 each morning and mail them to yourself would be:

0 1 * * * mailstat $HOME/Procmail/pmlog | mail YourAddress

To get a listing of pmlog.old, the previous log file, use the -o command-line argument:

mailstat -o $HOME/Procmail/pmlog

To get help on the mailstat command-line arguments, type one of the following:

mailstat -h
man mailstat  (I think this works only on a Debian Linux system)

	Your procmail log file will keep growing unless you regularly delete it by using mailstat, the Unix rm command, or some other technique. To minimize logging, put VERBOSE=no (which is the default) and LOGABSTRACT=no in your .procmailrc. To turn logging off, which I do not recommend, turn the LOGFILE setting in your .procmailrc file into a comment by preceding it with #, for example:    ## LOGFILE=whatever

Summarizing Your Log with Other Scripts

Other scripts that you can use to summarize a Procmail log are:

• Aaron Turner's ProcLog, A Procmail Log Reporting System
• Christian Stigen Larsen's Gubby
• culley harrelson's LogSum at Mudhut.org
• Parv's chkmail, which "combines fetchmail and mailstat (from procmail)" and is available on his Bourne Shell Scripts page
• Jeremy Mates' fm.pl (man page and script) - procmail logfile parser and reporter
• Max J. Werner's mailstat
• winnetou's match.sh and procstat.c
• Scott Wiersdorf's histospam - perl program that creates a histogram of email messages received by procmail

Customizing What's Logged

Some ideas for changing what is logged and where it is logged are in

• Jari's Keep simple header log
• David W. Tamkin message Re: Extra information when logging in the Procmail list
• the thread Re: Procmaillog and LOG variable in the Procmail list
• Bart Schaefer's message Re: help with a simple counter file and David W. Tamkin's followup, which contains a correction, in the Procmail mailing list.

Sending Log Entries or Messages to an RSS/Atom/XML Feed

To be notified about incoming email messages in an RSS/Atom/XML feed reader, you can use a service or tool such as one of the following. Note that most feeds can be accessed by anyone, so make sure that you either secure your feed or send only non-private messages to your feed.

Services

To use one of these services, you could set up Procmail to forward messages (or message headers or other message fragments) to the appropriate email address at the service provider.

• Bloglines email-to-RSS service, which is discussed in the Bloglines FAQ Why does Bloglines Offer Email Accounts?
• Any Blogger blog can be set up with a secret mail-to-blogger email address and an Atom feed of either short (~255 characters) or full blog items.
• As discussed on the Gmane page about RSS Feeds, any mailing list that uses Gmane, the mail-to-news/RSS/web gateway, has four RSS feeds. An example is the Procmail mailing list, which I discuss in detail in Procmail Discussion Groups below.
• Any mailing list hosted at Google Groups has two Atom feeds, one for topic summaries and one for message summaries.
• Any non-commercial public mailing list hosted at Yahoo! Groups has an RSS feed. Details are at Yahoo! Groups RSS Help.
• If you have a Google Mail account, messages in your Inbox are available via the username/password-protected Atom feed at https://gmail.google.com/gmail/feed/atom and messages labeled label are available at https://mail.google.com/mail/feed/atom/label/. I discuss using Procmail to forward messages to Google Mail in Forwarding and Plus Addressing below.
• MailBucket, which is discussed in this ThrowingBeans.org blog item and available at www.mailbucket.org
• mail2rss.org -- “free disposable email, read your email in your favorite RSS feedreader!”; released 2006 Feb 28

Tools

• To find a bunch of tools that transform email to RSS/Atom/XML, you can use a search engine to search for mail2RSS or search for the phrase mail to RSS
• Gmane.org: Gmane Software: “twine is a command line based program that converts mail into RSS. It's based on libxml2 and libgmime.”
• Johan Fitié's quickmail -- “quickmail is a PHP script that reads email from an IMAP server and outputs it in RSS format that can be displayed in a browser (as XML, using included CSS) or RSS aggregator or WML for use on WAP devices (like mobile phones or PDAs).”
• imap2rss.php -- simple PHP application that generates an RSS 2.0 feed from the messages within an IMAP folder.
• DynamicObjects.com/d2r: atomflow and atomflow-templates and atomflow-spider
• EightLinks.com: RSSMail - E-Mail to RSS Script -- a Perl script that will read your POP3 email and create a feed of message that match criteria you specify
• Wonko.com: MailFeed -- a PHP script that checks a POP3, IMAP, or NNTP mailbox on demand and returns an RSS feed containing the messages in the mailbox; “no longer being actively maintained”

I list more tools and services that can be used to deflect email messages to RSS/Atom/XML feeds — and vice versa — on the following pages. These pages include links to tools & services that can be used for doing many different types of message deflexion.

• Deflexion.com: Internet Message Deflexion: Intertwingling IMAP, SMTP, NNTP, IM, RSS, & More
• Deflexion.com: Watching the Deflexions and Reflexions (aka What's a Feed)
•  My del.icio.us bookmarks about Messaging/Deflexion and Messaging/Convergence

Please tell me about other tools or services that can be used to send a message or Procmail log entry to an RSS/Atom/XML feed.

	If you are looking for a desktop feed reader, check out Opera, which is free/gratis, ad-free, and registration-free for all. To me, it's especially nice when the Newsfeeds are set up with View > Display > List and message on right (as opposed to the default, which is List and message below). You can, of course, configure Newsfeeds (and all of Opera) however you like! Opera is also a multi-platform standards-based customizable HTTP, IRC, BitTorrent, NNTP, & IMAP client.

Using General Email Notification Tools

In addition to the Procmail-specific notification tools that I discuss above, there are many other programs, such as biff, that can notify you about incoming messages. For details, see:

• mailutil, which is a command-line utility distributed with the University of Wasington (UW) IMAP Toolkit and with both Unix- and PC-Pine 4.50+. It can be used to check local and IMAP-accessible mailboxes for RECENT messages and to do a lot of other things too. For details, see Using mailutil on the Power Pine page and man mailutil at the UW.
• About.com: Product Reviews: Windows Mail Checkers: page 1 and page 2
• EmailAddresses.com: Check Your POP3 Email Account -- many of these can also check IMAP-accessible mailboxes
• WebAttack.com: Freeware > E-Mail Tools > E-mail Notification
• TUCOWS > Linux > Internet > E-mail checkers
• Sven Guckes' Mail Notification
• Johan van Selst's chk4mail
• Apple.com > Downloads > Dashboard Widgets > Email & Messaging > eCheck
• Forkit.org's MacBiff - A GPL'd Mac OSX 10.3+ Biff Program
• FastCheck checks your FastMail.FM mailboxes in the background and notifies you when you receive new messages. [If you sign up with Fastmail and want to help Infinite Ink, set the Referer to iiiiiii@fastimap.com (that's i typed seven times or in geek speak i^7) or use this link to initiate your registration.]
• man biff and man comsat
• Deflexion.com: IMAP Tip: Use a backup-all mailbox -- discusses how to use a backup-all mailbox to monitor all your incoming email.

Also, some IMAP clients can be set up to monitor mailboxes on IMAP servers. For example, I run Mulberry along with my primary mail client, Pine, mainly for its customizable monitoring of mailboxes and mailbox-specific alerts.

On the Power Pine page, I discuss techniques for Checking for Recent Messages in Pine. Especially useful are the “newmail-fifo” and “newmail-window” features, which are available in Pine 4.62+.

Understanding and Refining Procmail Recipes

Anatomy of a Recipe

A Procmail recipe looks like this:

:0 flags: LockfileName  
* condition1-PMRE    (PMRE = Procmail regular expression)
* condition2-PMRE  
exactly one action item

where flags, the second colon (:), LockfileName, and the condition lines are optional. So a minimal Procmail recipe looks like this:

:0
exactly one action item

which means perform the exactly one action item on every message that is passed to this recipe and don't do any mailbox locking (that's what the missing second colon means).

A recipe that uses no flags and does not explicitly specify the lock file, but instead uses the implicit lock file, looks like this:

:0:
* condition1-PMRE
* condition2-PMRE  
exactly one action item

In this case, the header only is passed to the conditions and if both condition1-PMRE AND condition2-PMRE are satisfied, the whole message (header & body) is passed to the exactly one action item and procmail stops processing the message.

	If no flags are specified, Procmail uses the Hhb flags. Because of a Procmail bug, it is best to not explicitly specify the default flags, especially the H flag. For details about this bug, see Philip Guenther's message Re: Cannot get recipes to work properly in the Procmail list and Known Procmail Bugs below. Procmail conditions are case insensitive by default. If you want the conditions to be case sensitive, use the D flag. To specify a Procmail regular expression that will match leading whitespace, begin the PMRE with \ or () followed by the leading whitespace you want to match. David W. Tamkin wrote about this in his 2004-07-21 message Re: Debugging procmail rules and his 2004-04-24 message leading backslash problem (was persistent lock files). Exactly one action item can be one of the following: a mailbox delivery, a forward action, or a pipe. To do something else, such as a simple variable assignment or to perform more than one action, you need to use a nesting block. Nesting-block examples are in the sections Using a Nesting Block to Optimize the SpamAssassin Recipes and Deleting a Message below.

ORing Conditions

In the previous section, we saw that it is easy to create a Procmail recipe that ANDs conditions. If you want to OR conditons, see Jari's ORing traditionally, ORing and score recipe, and ORing by using De Morgan rules. There is an example of “ORing traditionally” in Using the Extraction Operator (\/) and $MATCH to Sort Solicited Bulk Email below. Dallman Ross's virus-snagger recipes include elegant examples of ORing with scores.

Dissecting the rc.testing Recipe

The recipe in rc.testing is:

:0:
* ^Subject:.*test
IN-testing

The following table explains the meaning of this recipe.

Matching a Word

If you want a recipe to match only the word test, rather than the string test, surround test with the Procmail word delimiters \< and \> as in the following:

:0:
* ^Subject:.*\<test\>
IN-testing

This recipe would not match on a message that contained procmail testing in the Subject but it would match on a message that contained procmail test in the Subject. The next section includes another example of a Procmail recipe condition that uses \< (the example is in the Tip).

To learn more about Procmail regular expressions, see regular expression in the Terminology section above and Jari Aalto's Procmail and Egrep Differences. Jari includes technical details about Procmail's word delimiters.

The most frequently asked question about Procmail is “How can I use Procmail to separate messages that were sent to different addresses?” And the answer is: “That is a job for the MTA (message transfer agent).” Unfortunately, it is now common for people to have a virtual (aka vanity) domain and to not be able to use an MTA to route the domain's mail. Instead, the domain's mail is delivered to a “dropbox” and then people try to use Procmail to do the routing long after the messages pass through the MTA that is running on the domain's purported mail server (the server pointed to by the DNS MX record for the domain). This is fraught with problems as you will discover if you try to do it.

The current Procmail maintainer, Philip Guenther, summed it up well in a message in the procmail mailing list in which he said:

Sometimes the MTA is set up to record the original envelope recipient address in one of the following headers.

And then you can use a Procmail recipe, such as the following recipe, to process messages sent to a specific address in your virtual domain.

:0:
* ^X-Rcpt-To:.*secretBccAddress@my\.domain
IN-sent-bcc

Note that the condition in this recipe uses \. (backslash dot) to specify an actual dot. If you use . without a preceding backslash, it is a metacharacter that means any character.

	I discuss reasons why you might want to use a secret Bcc address on my IMAP Service Providers page in the section "What to Look For in an IMAP Service Provider" in this auto-Bcc item, and in my blog in the item titled Server-Side Sent Messages. If you use a recipe similar to the above X-Rcpt-To example, I recommend that you use a regular expression that is more accurate than .* to match the characters between the colon (:) and the beginning of the email address. The example recipe above will also catch an address, such as anotherSecretBccAddress, that contains the address that you want to match as a substring. The best regular expression depends on the format of the header that your incoming MTA injects and on the characters that are allowed in the left hand side of an email address in your domain. A reasonably generic option is this: * ^X-Rcpt-To:(.*\<)?secretBccAddress@my\.domain ^^^^^^^ this PMRE is discussed in this section of Jari's PM Tips Because of dictionary spam and blowback (aka backscatter or outscatter) from spam that was sent with your domain name in a forged From header, I recommend that you set up your domain's email so that it does not use a catch-all mailbox. Instead, email sent to a non existent recipient should be rejected at the SMTP level. Most email hosting providers have a control panel where the domain administrator (e.g. you!) can turn off the catch-all for your domain(s). Important: Do not try to use Procmail to reject messages. If you do, you will be contributing to the spam problem and your system will very likely be added to blocklists. This is discussed in Rejecting or “Bouncing Back” a Message below. In addition to the original envelope recipient address, the original envelope sender address is also useful. On some systems, this address is in a Return-Path header. Here is an excerpt from a Tuffmail Help page about this topic: "The envelope sender may or may not be the 'From' address displayed by your email client. The envelope sender is the sender address provided in the SMTP 'MAIL FROM:' command. This address is also known as the return address or error address. Mail delivered to a mailbox on [the Tuffmail] system will have a 'Return-Path:' header added that contains the envelope sender address issued in the SMTP 'MAIL FROM:' command."

	Even if your mail-hosting provider has set up their MTA to inject a header such as X-Rcpt-To into incoming messages, there can still be problems. For example, A spammer might have constructed the original message with a fake “envelope” header line, for example a fake X-Rcpt-To header. The message may have been routed through another system and that system's MTA injected the state of the envelope (as it was at that point) into the header. For example, as discussed below, messages sent to the Vim mailing list contain a  Delivered-To: vim@vim.org  header. If a message is for multiple local recipients, some MTAs do not inject this type of header.

List of Links: “Why Headers Don't Matter” and More

Once SPF (Sender Policy Framework) is adopted, some headers from some users will matter, i.e., be trustable. Currently no header should be trusted.

For exhaustive information about this, see:

1. chromatic's weblog entry titled One Question Certification Tests for E-Mail Filter Authors and subtitled “I'm normally skeptical of certification programs, but anyone who writes a program intended to scan and to respond to incoming e-mail should be required to pass a one question test before proceeding.”
   
   I recommend that everyone who creates email filters take chromatic's one question test, especially if your filters use auto-replies or white/green/red/black/block lists! If you pass that test, here's my one question super advanced followup test:

   If a@b.c is the recipient of an email message, which headers will contain the address a@b.c?

   To check your answer, see the links below.
   
   
2. era eriksson's Procmail FAQ: Why Headers Don't Matter
   
   
3. era eriksson's Procmail FAQ: Why can't I match on the BCC: header?
   
   
4. era eriksson's Procmail FAQ: How do I implement a virtual domain? Or, how can I let several local users share the same POP mailbox at the upstream? Or, what is the most frequently made wrong assumption about mail delivery?
   
   
5. Jari Aalto's Procmail Tips: Bcc lecture by Alan Stebbens
   
   
6. Jari Aalto's Procmail Tips: Bcc lecture by Philip Guenther
   
   
7. Jari Aalto's Procmail Tips: What is that first From_ header?, which includes this: "A problem arises however when people start creating virtual domains. When sendmail does the aliasing (usually by mailertable I believe?), it totally loses the original envelope recipient address in the rewriting. All the addresses get rewritten to the same thing, and sendmail thus has no reason to differentiate them. Having lost their independent identities, the now-same multiple recipients are merged to form one call to the local mailer. The key point here is that once the envelope recipient is lost by the virtual domain alias, THERE IS NO WAY TO GET IT BACK!"
   
   
8. Bart Schaefer's 2004-Dec-11 message to the Procmail list, in which he discusses that it is possible to “have procmail at the provider insert the special header information that's recognized by the downstream procmail for local routing. That would, however, mean the dropbox would get one copy of the mail for every downstream recipient rather than one copy for all downstream recipients...”
   
   
9. sendmail.org: Virtual Hosting with Sendmail and Q3.29 -- How can I add a header specifying the actual recipient when having multiple users in a virtual domain go to a single mailbox?
   
   
10. Andrzej Filip's Deliveries to Shared Mailboxes (sendmail)
    
    
11. Philip Guenther's sendmail-related message Re: Procmailers beware: BCC handling in the Procmail mailing list
    
    
12. Dave Sill's Life with qmail: Configuring Virtual Domains
    
    
13. Fetchmail
    • FAQ: Multidrop-mode problems, especially M2. I can't seem to get fetchmail to route to a local domain properly and the Q&As below this one
    • Man Page: The Use and Abuse of Multidrop Mailboxes
    
    
    
14. Getmail FAQ
    • What is a "domain mailbox"?
    • What's this "envelope sender" and "envelope recipient" stuff?
    
    
    
15. the Procmail mailing-list thread virtual domain multiple users filter
    
    
16. Jason R. Mastaler's TMDA Pre-Configuration page, which includes information about getting envelope information from various MTAs. Jason says “TMDA receives much of its information about the envelope of an incoming message from environment variables set by the MTA. Most importantly, SENDER (the full envelope sender address), RECIPIENT (the full envelope recipient address), and EXT or EXTENSION (the recipient address extension).”
    
    
17. Ken Lucke's Reading Email Headers
    
    
18. EmailDiscussions.com > Discussions about Email Services > The Technical Zone... > Guide to email headers? How easily can these be forged?
    
    
19. SpamAssassin Bugzilla Bug 2142: Need code to infer MAIL FROM: address from message (this bug discussion includes a link to this page!)
    
    
20. John Conover's E-mail "Received: " Header IP Address Auditing
    
    
21. Jonathan de Boyne Pollard's 2004-July-5 message in comp.mail.misc Re: received header field
    
    
22. Sean B. Straw's 2004-July-14 message in the procmail list Re: local procmail filtering before using sendmail smarthost, in which he says

    “There is a procmail mantra. Please recite it with me:

    PROCMAIL IS NOT AN MTA.”

    
    
23.  My del.icio.us bookmarks about email envelopes.

Alternatives to the ^TO_ Macro

The recipes that I suggest in Step 15 for sorting solicited bulk email use the special procmail expression ^TO_, which catches any message that contains the list address in the To, Cc, Resent-To, or some other destination header. If someone sends a message to a mailing list by putting the mailing list address in the Bcc header, this address will usually not appear in any of the destination headers that the ^TO_ expression checks — that's why it is called Blind Carbon Copy!

You can catch these Bcc'd messages, and all other messages that are sent to the list, by determining a header that all messages that are passed to the list contain, and messages that are not sent to the list do not contain. Such a header depends on either the mailing-list management software that the list is using or the address that you used to subscribe to the list (this will work if you use a unique address for each list you join). Examples of such headers are:

• List-ID:     (can be used by Majordomo, Majordomo2, and Mailman; see RFC 2919 for details about this header)
• List-Post:     (see RFC 2369 for details about this and related List-* headers)
• Mailing-List:     (used by ezmlm and Yahoo! Groups)
• X-Mailing-List:     (used by SmartList)
• X-BeenThere:     (used by Mailman)
• X-MDMailing-List:
• Sender:
• Resent-Sender:
• Return-Path:
• Delivered-To:  (discussed in previous section and in the Vim list example below)
• ‘From ’  (without the quote marks;  this is sometimes referred to as the  mbox format 'From ' line  or the  From_ line)

:0:
* ^Delivered-To:.*vim@vim.org
IN-S-vim

:0:
* ^List-ID:.*procmail.lists.RWTH-Aachen.DE
IN-S-procmail

(This procmail recipe works on messages sent to the procmail mailing list after 18 May 2000, which is when the mailing list management software changed and started including the List-ID header.)

	Default Procmail conditions are not case sensitive so you do not need to worry about the case you use in a condition. For example, headers that begin with List-ID, List-Id, and any variation in capitalization will all match on the beginning of the second condition above. Most regular expressions, including Procmail regular expressions, use dot (.) to match any character. To match an actual dot, you need to use \. (backslash dot). For example, to be precise, use vim@vim\.org rather than vim@vim.org, which would match on vim@vimborg.net (and lots of other addresses with any character where the dot (.) is in vim@vim.org). Many auto-responder and solicited-bulk-email messages include one of these headers: Precedence: bulk Precedence: list Using a Procmail recipe condition that checks for a Precedence header might be useful... if you are doing a simple bulk/non-bulk sort of your email; or if a mailing-list recipe's conditions are not strong enough and are incorrectly matching on some messages that were not sent to the list; or in an auto-responder recipe. For details about these and other Precedence headers, see List of precedence codes in Jari Aalto's PM Tips. Details about the above and other message headers are in the RFCs mentioned above (2919 & 2369), RFC 2076 - Common Internet Message Headers, RFC 4021 - Registration of Mail and MIME Header Fields, and IANA's Permanent Header Registrations.

	Some people prefer to use the ^TO_ expression to catch mailing-list messages because: A message that is Bcc'd to a list is often spam and since the ^TO_ expression does not catch Bcc'd messages, these spam messages won't be put in your mailing-list mailbox. A message that is sent both to you personally and to the list will be put into the list's mailbox twice, rather than once in the list's mailbox and once in your personal-correspondence inbox. To get rid of duplicate messages, you can use the duplicate-catching recipe that is described in the procmailex manual page (but make sure you read the paragraph that begins with the word “Beware”!). If you interact with the mailing-list server by sending, for example, an unsubscribe or help request, the auto-reply message will not be put in the mailing-list's mailbox (because it will be sent directly to your address rather than the list address). Some other list-catching Procmail recipes, for example one that uses the List-ID header, might put these list-management auto-replies into the list box where it could get lost in the crowd of list messages. If the the mailing-list management software changes but the mailing-list address stays the same, your procmail recipe will still work. Details about the ^TO_ expression are in Step 15's Important Note About Procmail's ^TO_ Macro.

Advanced Recipes

Using the Extraction Operator (\/) and $MATCH to Sort Solicited Bulk Email

If you subscribe to a lot of mailing lists and each one has its own recipe like this:

  
:0:
* ^TO_vim@vim.org
IN-S-vim

You can speed up processing by combining all your subscription recipes into a single recipe like this:

  
:0:
* ^TO_\/(subscription1|subscription2|subscription3)
IN-S-$MATCH

Here is what this means:

The disadvantage of this recipe is that your mailbox will be named whatever you specify as subscriptionn. For example, if you use this:

  
:0:
* ^TO_\/(procmail@lists.rwth-aachen.de|vim@vim.org)
IN-S-$MATCH

Your mailboxes will be named:

  
IN-S-procmail@lists.rwth-aachen.de
IN-S-vim@vim.org

which are ugly and would be a pain to type in your mail client or at a command-line prompt. So you might want to use the following:

  
:0:
* ^TO_\/(procmail|vim)
IN-S-$MATCH

This produces nice mailbox names but has the disadvantage that it might match on messages that were not actually sent to the mailing lists. For example, it would match on a message sent to vimala@example.com. (FYI, Vimala is an Indian name that means "pure.") You can refine this recipe by appending the "at" character (@) to the condition:

  
:0:
* ^TO_\/(procmail|vim)@
IN-S-$MATCH

This should be pretty good about matching only messages that were actually sent to the list but now your mailbox names look like this:

  
IN-S-procmail@
IN-S-vim@

To delete the trailing @ from the mailbox names, you can use the following recipe (assuming you are using Procmail version ??? or later):

  
:0:
* ^TO_\/(subscription1|subscription2|subscription3)@
* MATCH ?? ()\/[^@]+
IN-S-$MATCH

Here is what this recipe means:

Downcasing a Mailbox Name

Most mailing lists are sorted fine by the above recipe, but there is a problem if the string subscriptionn in the mailing-list address is not consistent in the capitalization of its letters. For example, debian-user messages are delivered to mailboxes named

IN-S-debian-user
IN-S-Debian-user

and all other combinations of upper- and lower-case letters that list participants use in the ^TO_ address. This happens when mailing-list-management software does not rewrite the list address into a canonical form and instead leaves it in the form that the message's author used.

To solve the problem with the inconsistent capitalization of the string 'debian-user' and any other non-canonicalized list addresses, you can do one of the following.

• Create an individual recipe for the list. For example, the recipes described in
  • Step 15 above, or in
  • Alternatives to the ^TO_ Macro above
• Use the adjust-case technique described in
  • the 1999-Dec-14 message Re: questions about ^TO_ and MATCH by David W. Tamkin.
  • Converting value to lowercase in Jari's PM Tips.
  • the 2005-Jan-24 message Re: Nested recipes... (sorry if this is sent twice.. ) by Ruud H.G. van Tol; also see David W. Tamkin's follow-up message with Subject recasing (was Nested recipes... ) and the entire thread.
  • the 2005-April thread case insensitive $MATCH, especially this message by Ruud H.G. van Tol
• Use the generic-SBE-sorting recipes described in the next section.
• Use the technique described in Forwarding and Plus Addressing below.

Generic SBE Sorting

One way to avoid the inconsistent-capitalization problem described in the previous section is to extract the list name from a header that is inserted by the mailing-list software, such as Sender, List-ID, List-Post, Mailing-List, or one of the other alternatives to ^TO_ that I list above. For a simple example of extracting the mailing-list name from the Sender header and using $MATCH to construct the mailbox name, see the Recipes section at the bottom of this FreeBSD Diary Procmail page. This contains Neil Blakey-Milner's recipe for sorting all the FreeBSD lists that he subscribes to.

Sean B. Straw, of Professional Software Engineering, has a much more generic SBE sorting system. He created listname_id.rc, which extracts the list name from the headers of a message and assigns it to a variable named LISTNAME. He then uses something like the following in his .procmailrc.

INCLUDERC=$PMDIR/listname_id.rc
 
:0:
* ! LISTNAME ?? ^^^^
IN-S-$LISTNAME

For more about this, see:

• 2004-Sep-11: Ruud H.G. van Tol's message Re: Prioritizing matching of "best" header and the rest of the thread
• 2004-Feb-17: the thread Duplicates sent from lists
• 2004-Feb-10: Sean's message Re: Procmail CPU Time and the rest of the thread
• 2003-June-04: Mads Laursen's message Re: mailing list recipes and the rest of the thread
• Jari Aalto's Using subroutine pm-jalist.rc to detect mailing lists and the sections immediately below this
• Ruud H.G. van Tol's listmail.rc, which is a variation of Sean's listname_id.rc

Note that I do not currently use these generic-SBE recipes but I plan to try them one of these days.

	If the extracted $LISTNAME contains a dot (.), forward slash (/), or backslash (\) and the mailbox IN-S-$LISTNAME is accessed using IMAP, the IMAP server may interpret one of these characters as an IMAP namespace hierarchy separator. For details, see Notes About Mailbox Names above.

More Examples of the Extraction Operator

In the sections Using SpamAssassin and Using a Nesting Block to Optimize the SpamAssassin Recipes below, there is an example of using the extraction operator to prepend text at the beginning of the Subject of a message. This technique can also be used to append text to the end of the Subject.

Dallman Ross's Virus Snaggers include recipes that extract the values of the Content-Type and From headers and assign them to variables named CTYPE and VFROM, respectively.

Also see Philip Guenther's 1997-Jan-15 message Re: Wierd regex Behaviory [sic] in the Procmail list.

Forwarding and Plus Addressing

In addition to the techniques for downcasing a mailbox name listed above, another option is to use forwarding and plus addressing. I discovered this because I deflect some of my mailing-list messages to my Tuffmail.com and Gmail.com accounts and use plus addressing to deliver directly to an appropriate mailbox or “label” at Gmail. For example, instead of delivering to a mailbox named IN-S-$MATCH as in the extraction-operator recipe above, I forward these messages by putting the following in the recipe's action line:

! MyUID+$MATCH@tuffmail.com

or this:

! MyUID+$MATCH@gmail.com

or to route these messages to both Tuffmail and GMail, I use this:

! MyUID+$MATCH@tuffmail.com MyUID+$MATCH@gmail.com
                           ^
                           Note that these 2 addresses are separated by a space 
                           If this doesn't work try using comma space (', ') or comma (',') instead

If a mailbox named $MATCH exists in my Tuffmail mailstore, the message is automatically delivered directly to it. In other words, all Debian-user messages are delivered to a mailbox named debian-user (if it already exists) regardless of how the string dEbIaN-uSeR was capitalized in the original message or in the envelope recipient address that Tuffmail received.

To learn more about “plus addressing,” which is also known as “subaddressing,”  “user-extension addressing,”  and “detailed addressing,” see my del.icio.us bookmarks about plus addressing and this section of my IMAP Service Providers page.

To learn more about Gmail (aka Google Mail), see my del.icio.us bookmarks related to Gmail and the Gmail discussion groups, especially the threads about The PLUS (+) feature and Filtering for list mail. At Gmail you need to create a Gmail filter to automatically label incoming messages. The Gmail MTA records the original envelope recipient address (for example MyUID+$MATCH@gmail.com) in a Delivered-To header, but unfortunately the Gmail filter cannot filter on this header. I describe how I get around this in a 2004-July-20 message with the subject Tip: Simultaneously file a message in an IMAP mailbox and assign a Gmail label that I posted to the Procmail list — make sure you read the entire thread.

Gmail Tips: 

• ExtremeTech.com: Gmail Power Tips: Plus Addressing and Filtering by Ben Hammersley
• If you forward mail to your Gmail account, you will have the option of reading it in an RSS/Atom/XML client. This is discussed in Sending Log Entries or Messages to an RSS/Atom/XML Feed above.
• To learn how to use Pine to “bounce forward” (redirect) archived messages to Gmail, see the Gmail Tips on my Power Pine page.

The Problems With Forwarding

Doing MTA-style forwarding with Procmail can be problematic, most notably there can be problems with:

• mail loops, which are discussed in
  • man procmailex (search for Loop)
  • Jari Aalto's Procmail Tip about using an X-Loop (or an X-YourDomain-Loop) header to keep track of where a message has been
  • cornerhost wiki: ForwardingMailToGmail
• losing a message if 1) the message is forwarded to an account that it already passed through and if 2) that account (the target account of the forward) is on a system that suppresses (discards) any message that has already been through that system. (FYI, duplicates are often detected by a header, such as Message-ID, Received, or Delivered-To, and possibly a database that tracks messages that have been through the system.)
• the correct routing of auto-replies, SMTP-level bounce-backs (5xx rejection messages), and other error messages from the final destination back to the original sender
• the target machine not being able to do robust SMTP-level spam detection on the incoming forwarded messages using Greylisting, SPF (Sender Policy Framework), or the other techniques described in Wikipedia's Stopping e-mail abuse page. Note that this problem might be solvable using one of the following:
  • a recipe similar to the one Bart Schaefer posted in the 2005-July-13 message Re: Forwarding and Barracuda Spam Appliance
  • SRS (Sender Rewriting Scheme), which is discussed at pobox.com and Wikipedia.org.
  • RPR (Return Path Rewriting)
• the forwarding machine being blocklisted if it forwards spam or viruses — an example of this is discussed in this 2004-June-27 message at grc.spam
• the issues discussed in Redirection of Email Messages is Dangerous in the OSAF (Open Source Applications Foundation) wiki

For more about this and related problems, see:

• Greg Hewgill's email forwarding considered harmful
• the section Rejecting or “Bouncing Back” a Message below, especially the Note at the bottom of that section

Snagging Viruses

Nowadays many mail service providers give their users the option to have incoming mail scanned for viruses, but if either . . .

• your mail service provider does not offer virus scanning,
• or you have chosen not to use your provider's virus scanner because
  • you don't trust that it will always be up-to-date and catch only viruses,
  • or it bounces (possibly defanged) virus messages back to forged From: addresses,
  • or it sends you, the intended recipient of the virus, auto-notification for each virus it catches that includes an ad for the virus-scanning software,

then you can use Procmail — preferably running on your incoming mail server — to detect and snag viruses. There are many Procmail recipes that (attempt to) do this, including:

• Dallman Ross's Virus Snaggers. Important: To use vsnag 2.x, you need Procmail 3.15+.
• John Conover's “Quarantining Malicious Outlook Attachments” recipes
• Barton E. Schaefer's: Email Tools: viriirc.txt recipes; also see the message from Bart Re: virus recipe for MyDoom
• John D. Hardin's “E-mail Sanitizer” recipes
• Nikos K. Kantarakias's “Yet Another antiVirus Recipe” (YAVR), which is available at Nikos' site and at FreshMeat.net. Some details are in this 2004-02-29 message from Nikos.
• Scott Wiersdorf's Blocking Windows Worms at the Server with Procmail on a VPS
• Ruud H.G. van Tol's recipe that he posted on 2003-09-23 to the Procmail mailing list
• Catherine Hampton's SpamBouncer, which checks for both viruses and spam
• Robert Allerstorfer's Softlabs AntiVirus, which he first announced in this 2004-02-29 message
• a recipe that pipes a message through an external virus scanner such as the following:
  • Clam AntiVirus (via clamscan, clamdscan, or TrashScan; note that as of April 2004, clamscan is more stable than clamdscan)
  • Jem Berkes RenAttach (via a procmail recipe on this page)

 
I am collecting Virus-related links at del.icio.us/Deflexion.com/Viruses.

Using SpamAssassin

SpamAssassin is free/libre open-source software (FLOSS) that can be used to analyze a message and tag it with a “spamminess” score. The higher the score, the more likely it is that the message is spam. After a message is tagged, Procmail can be used to filter the message to an appropriate mailbox, assign an IMAP keyword (label), or delete it.

	The recipes below are specific to my system and my configuration files. Before using these recipes, you need to... Edit the recipes below so that they work with your system. For example: If you do not use maildir-formatted mailboxes, change the delivery recipes so that they use the correct delivery syntax for your system. Edit your SpamAssassin configuration file (usually local.cf or user_prefs) so that an X-Spam-Level header is inserted into your SpamAssassinated messages. The X-Spam-Level header is used in the SpamAssassin Procmail recipes below. Be aware that SpamAssassin 3.x is not backward compatible with SpamAssassin 2.x so make sure that you 1] read the UPGRADE documentation and 2] run  spamassassin --lint --debug  after you upgrade! The SpamAssassin Procmail recipes below should work with both SpamAssassin 3.x and 2.x, provided you have configured SpamAssassin to add_header all Level _STARS(R)_ ^ this character is the "X-Spam-Level character" I use R (for Red/Rubbish) as the X-Spam-Level character because 1] in Procmail, and many other filtering tools, it is easier to match R than * (asterisk), which is the default; and 2] spammers are more likely to forge SA headers using the default spam-level character (*). Tip 1: If you want the spam score to be injected into the Subject header (or From or To header) of spammy messages, for example messages with a spam score of 4 or more, put settings like the following in your SpamAssassin configuration file. required_score 4 rewrite_header Subject {_SCORE(0)_} ^ ^ I use squiggly braces, {}, because | square brackets, [], are special in IMAP | zero padding, details here If you use these two settings, it will be easy to use an email client to sort (order) your messages by spamminess and find false positives (non-spam messages that were flagged as spam). I discuss this, and more, in my blog item titled Using a MaybeSpam Mailbox. (Note that required_score is called required_hits in old (outdated!) versions of SpamAssassin.) Tip 2: If you want to use an email client — either instead of or in addition to Procmail — to manage your spammy email, make sure that you use the spam headers that your email client is looking for. For example: For Thunderbird, see Thunderbird's Junk Mail Controls (JMC), especially the discussion about the option called  Trust junk mail headers set by. For Mutt, see the Spam detection section of the Mutt manual. For more information about configuring SpamAssassin, see the SpamAssassin Docs, especially   man Mail::SpamAssassin::Conf (which is also known as perldoc Mail::SpamAssassin::Conf). The message Re: add headers by Theo Van Dinter is also useful.

Using a Nesting Block to Optimize the SpamAssassin Recipes

We can improve the SpamAssassin-related recipes in the previous section by immediately delivering any {* -BIG- *} message. This will stop the processing of the message and avoid checking if it has an X-Spam-Level header, which it should not have because it was not piped through spamc. One way to do this is to replace the first recipe above with a recipe whose exactly one action item (discussed in Anatomy of a Recipe above) is a nesting block. For example, you could replace it with a recipe similar to the following.

   
# This outer "container" recipe is a non-delivering recipe and does
# not need a lock file. That's why there is no colon (:) after :0 in 
# the next line.
:0
* > 256000
{ 
 :0 fh w
 * ^Subject:\/.*
 | formail -I "Subject: {* -BIG- *} $MATCH"
  
 :0:
 --yellow
}
  
# IMPORTANT: In this example, the --yellow mailbox is in mbox format
#  and thus a lockfile is needed. If you use a different mailbox format,
#  make sure that you change the delivering recipe so that it is 
#  appropriate for your system.
  

This is an example of a nesting block, which you can read about in man procmailrc and see examples of in man procmailex. For information about and links to the Procmail manual pages, see The Fine Man Pages below.

See Also:

• John Hardin's antispam/spamassassin.procmail
• My Del.icio.us bookmarks about SpamAssassin at del.icio.us/Deflexion.com/Messaging/SpamAssassin and del.icio.us/Deflexion.com/SpamAssassin

Setting Keywords or Labels

Inspired by cheap disk space and Opera Mail's & Gmail's search-don't-sort approach to email, some people are starting to keep all their messages in one giant universal mailbox and then using a combination of keywords (aka labels or flags or tags) and saved searches to zoom in on various groups of messages (“create virtual mailboxes”). This system has a number of advantages over using separate mailboxes, including:

• being able to give one message multiple labels
• being able to do a merge or intersection of groups of messages, for example,
  • view all messages with label-N OR label-M
  • view all messages with label-N AND label-M
• not needing to remember what mailbox you filed a message in
• not needing to search through multiple mailboxes when you are looking for a message
• not needing to spend time filing messages
• not needing to monitor lots of mailboxes for recent messages

I have not yet switched to this one-mailbox strategy, but I have started delivering my “yellow” (unknown, possibly spam) messages, which I discuss in the Using SpamAssassin section above, to my $DEFAULT mailbox. To set an IMAP keyword on these messages, I use a Procmail recipe like the following.

Note: The dollar sign ($) is escaped with a backslash (\) so it will not be interpreted as a Procmail metacharacter.

# this recipe will usually work with an mbox-formatted mailbox on a UW IMAP server	
:0
* condition that matches a message that is yellow (unknown, possibly spam)
{
  :0 fh w
  | formail -i "X-Keywords: \$Label5" 
   
  :0:
  $DEFAULT
}

I then have my mail clients, Alpine, Pine 4.60+, Cone, Mulberry 4.0.1+, and SeaMonkey Suite, set up so that messages that have the $Label5 keyword are colored yellow.

If you use Opera Mail, the $Label5 IMAP keyword is not supported and you instead need to use an IMAP keyword named XOPL-N. For example, if you would like your possibly-spam messages to be labeled “Funny” () in Opera Mail, substitute the following in the recipe above:

| formail -i "X-Keywords: XOPL-6"

I have tested this only with the Opera 9.0 Preview. Please let me know what, if any, IMAP keywords are supported in other versions of Opera.

See Also:

• Joseph Scott's blog: Email Tags -- this discusses Procmail, IMAP keywords, and more
• IETF Internet Draft: Common IMAP keywords by Alexey Melnikov, which proposes the following standard keywords: $Forwarded, $AutoJunk/$AutoNotJunk/$AutoMaybeJunk, $Junk/$NotJunk, $Work, $Personal, $ShouldReply, $Important.
• IETF Internet Draft: SIEVE Mail Filtering Language: IMAP flag Extension by Alexey Melnikov
• Infinite Ink: Power Pine: Setting Up Keywords (Labels) -- includes a tip about using keywords to implement David Allen's Getting Things Done strategy.
• Infinite Ink: Compartmentalizing and Sharing Your Pine Configuration: Sample Configuration: Keywords — discusses the 22 user-defined keywords that I use
• Infinite Ink: This item and the item below it in the section What to Look For in an IMAP Service Provider on the IMAP Service Providers page. These items discuss servers, clients, and delivery agents that support IMAP keywords.
• Infinite Ink: IMAP Arena 2: SeaMonkey Suite and Thunderbird versus Pine on the All About Pine page, which discusses support for keywords/labels/flags in Mozilla-based clients.
• The comp.mail.imap thread IMAP keywords (labels) / Annotate[more].
• The comp.mail.pine & comp.mail.imap thread Questions on Keywords.
• Deflexion.com:
  • Using a MaybeSpam Mailbox -- discusses using an LDA to set the IMAP \Deleted flag on incoming messages that have a high spam score
  • Server-Side Message Labels -- describes how I use labels to help me manage my email
  • Sometimes a Mailbox is Just Not a Mailbox -- discusses virtual mailboxes
  • One-Click Tagging in Mulberry -- discusses an easy way to use the Mulberry IMAP client to set and unset labels
•   My del.icio.us bookmarks related to Keywords and GTD (Getting Things Done).

Deleting a Message

	Before you use a procmail recipe to delete a message, you should first have the recipe deliver to a 2delete or infrared mailbox (see the SpamAssassin recipe above for an example). Then monitor this mailbox — and delete the messages by hand — for about a month. After you are sure that the recipe catches only messages that you want to delete, edit the recipe so that it deletes the messages.

One way to have procmail delete a message is to deliver it to /dev/null by using a recipe such as this:

:0
* ^X-Spam-Level: RRRRRRRRRR
/dev/null

Another way to delete a message is to set the HOST variable to anything other than the hostname of the machine on which procmail is running. This will make Procmail immediately stop reading the procmailrc and return an exitcode of zero (success). For example:

:0
* ^X-Spam-Level: RRRRRRRRRR
{ HOST = adios }

Another option, which is a variation of the previous option, is to unset the HOST variable by using a recipe like this:

:0
* ^X-Spam-Level: RRRRRRRRRR
{ HOST }

This last recipe is more cryptic to someone — including you — who might try to understand it in the future so I recommend explicitly setting HOST to something such as adios or byebye or hasta.la.vista.baby, which will make for better self-documenting code (aka literate programming). For more about the HOST variable, see Jari Aalto's Procmail Tips page in the section called Variable HOST and Dallman Ross's message to the Procmail list Re: How to tell promail to stop processing recipes?

	Setting the HOST variable is more efficient than “delivering” to /dev/null. A variable assignment, such as the HOST assignment above, is not considered an action item and so needs to be done within a nesting block (i.e., inside curly braces). None of these three recipes need a lock and using one will cause procmail to complain about an “extraneous lockfile.” For details about why it does not make sense to lock /dev/null, see David W. Tamkin's 2004-03-14 message Re: looking for simple procmail recipe in comp.mail.misc. Tip: If you use IMAP, you might want to set the IMAP \Deleted flag rather than actually deleting a message. This way you can use an IMAP client to monitor and expunge \Deleted messages. I discuss this in Using a MaybeSpam Mailbox in my blog.

Rejecting or “Bouncing Back” a Message

By the time Procmail gets a message it is too late to reject it at the SMTP level (aka “block it at the SMTP level” or “send a 5xx error code”). It is sometimes possible to use Procmail to “bounce back” a message or to send a fake SMTP-rejection message, but this is not recommended because...

1. your system's receiving MTA will have already told the sending MTA that the message was accepted,
2. it is often impossible for Procmail to determine the correct address to send the bounce-back message to,
3. if you do use Procmail to send a bounce-back message, it will often be filtered or ignored by the recipient, and
4. your system might be put on a block list.

For more about all this, see:

• Reverse Spam Filtering page, especially the Email Deflexion Flowchart
• SpamLinks.net > Prevent > Secure > Backscatter > Fake Bounces
• Jari Aalto's:
  • Procmail plan for spam: Bouncing messages do no good in which Jari says: "The MTA is the only place where rejects may be handled intelligently"
  • PM Tips: List of exit codes and UBE and bouncing message back
• era eriksson's:
  • Procmail FAQ:  I've seen all these recipes which use EXITCODE= but how can I tell what exit codes to use?
  • Moronic Mail Autoresponders (A FAQ From Hell) at www.iki.fi/era/mail/autoresponder-faq.html
• SpamCop.NET: SpamCop FAQ: Help for abuse-desks and administrators: Robots: Mailing lists and autoresponders: Why are auto-responders bad?
• Slashdot.org: Why Do Email Admins Make Viruses Worse? (2004 January 27)
• O'Reilly Developer Weblogs: One Question Certification Tests for E-Mail Filter Authors by chromatic (2003 August 19)
• gmane.mail.procmail: Re: EXITCODES, bounce, might be off topic by David W. Tamkin (2002 April 19) in which David says: “Bouncing spam from procmail can be pointless or mean, but it can never help. Don't do it.”
• gmane.mail.spam.spamassassin.general:
  • Re: Rejecting spam with SpamAssassin and Qmail by Simon Byrnand (2004 April 6) in which Simon says: “remember that even if you reject at an SMTP level, you have still received the ENTIRE contents of the message (the DATA phase of the SMTP transaction) BEFORE you reject it, so you don't save any network bandwidth.”
  • the thread A really good reason not to bounce emails, which discusses the New Scientist article Email attack could kill servers (2004 April 6)
• FastMail.FM: What does the Backscatter Action mean?
• Taint.org: Justin Mason’s Weblog » How to deal with joe-jobs and massive bounce storms
• section on this page (above): List of Links: “Why Headers Don't Matter” and More
  

	The word bounce is sometimes used to mean bounce forward, which is also known as redirect, resend, remail, relay, or MTA-style forward. Because of the potential for confusion, I try to always use either the phrase bounce forward or bounce back instead of the single word bounce, unless it is absolutely clear from the context which type of bounce I mean. Note that bouncing forward, which can be done using ! in a procmail action item, also has problems. For details about bounce forwarding, see Forwarding and Plus Addressing above, especially the subsection called The Problems With Forwarding. I am collecting links about both bouncing forward and bouncing back in my Del.icio.us bookmarks under the tag Messaging/Bouncing.

Minimizing False Positives With Greenlists and Bluelists

If you use recipes, such as the SpamAssassin recipes above, to try to sort spam into a “red” (rubbish) box, you will no doubt have experienced a

false positive (FP) — a non-spam message that was falsely identified as spam and delivered to a rubbish box.

A key to avoiding this type of false positive is to first sort messages that are almost surely not spam. For example, I use a bluelist and a greenlist, along with Procmail, to pre-sort my solicited-bulk-email (SBE) and my trusted (good) non-SBE senders.¹

To be able to optimally use this strategy, you need...

1. recipes that identify and sort “green” and “blue” messages into appropriate mailboxes. For example, I use a recipe similar to the Solicited-Bulk-Email recipe above to deliver my blue (bulk) messages to appropriate blue boxes and a recipe similar to this or this to deliver my green messages to my green box (which, of course, could be set up to be the $DEFAULT box).
   
   
2. to invoke your bluelist and greenlist recipes after virus-detection tools are invoked.²
   
   
3. to invoke your bluelist and greenlist recipes before spam-sorting recipes are invoked.²
   
   
4. tools and automated systems that make it easy for you — and only you — to view and update your greenlists and bluelists using any mail client running anywhere.³ Possibilities include:
   1. Doing updates by having every message that you send auto-Bcc'd to a secret magnet-update address. The addresses in the To and Cc headers of these messages would then be automatically added to your greenlist by Procmail. In order to correctly identify and process these Bcc'd messages, procmail needs to be able to determine the original envelope recipient address (i.e., the address that was put in the Bcc header) of incoming messages.
   2. Doing an update via commands that you send in an email message or via a message that you bounce forward to specific secret magnet-update addresses. These messages could be processed by procmail recipes that do different things depending on the original envelope recipient address (i.e., which secret magnet-update address the message was bounce forwarded to).
   3. Using the IMAP COPY command to copy representative messages into your 2greenlist, 2bluelist, 2ungreenlist, and 2unbluelist IMAP mailboxes and then periodically run a script against these mailboxes to update your procmail-usable greenlist and bluelist.
   4. Using an IMAP client to compose a DRAFT message that contains magnet-update commands and periodically run a script against your DRAFT mailbox to update your procmail-usable magnets.
   5. A generalization of the previous technique (#4D) is to place messages in a magnet-update mailbox and use IMAP keywords or flags to specify what type of magnet the message represents (e.g., whether it should be used to create a green, blue, or red magnet). For an example of magnet keywords, see the keywords that I use, which include my greenFrom and sbeRecip keywords.
   6. Using server-side address books to maintain your greenlist and bluelist and then periodically run a script against these address books to update your procmail-usable greenlist and bluelist. Mulberry, Pine, Cone, and Mohogany are examples of email clients that support server-side address books. Pine uses IMAP to store & update remote address books and Mulberry can use either IMSP or the defunct ACAP. For some details about how I do this, see my 2005-Mar-01 message in the spamtools list.
   7. LOAF, which is a distributed-social-network greenlisting tool, that currently works with Pine (and Procmail of course!).
      
      
5. tools that makes it easy to create and populate your base greenlist and bluelist. For example, I do this by running a shell script (like this) against my SENT mailboxes and my Pine address books.

Soon I (and I hope others) will write about various ways to implement this in the upcoming Infinite Ink wiki-blog, which will probably be based on Dokuwiki. For now, see:

• Reverse Spam Filtering: Winning Without Fighting -- gives an overview of my email deflexion strategy and explains how I use my magenta, violet, blue, green, lime, yellow, and red mailboxes.
• The See Also section of Reverse Spam Filtering -- has links to lots of reverse-spam-filtering tools and strategies, including many that are based on procmail.
• The IMAP: What and Why section of IMAP Service Providers: A Step in Dealing with Viruses, Spam, and Email Overload
• Deflexion.com: Using a MaybeSpam Mailbox -- describes how I manage my MaybeSpam (“red”) mailbox, which sometimes contains "blue" or "green" messages
• Deflexion.com: Server-Side Address Books and Server-Side Greenlists
• The section Using the Extraction Operator (\/) and $MATCH to Sort Solicited Bulk Email above -- this describes recipes for sorting blue mail.
• Procmail list:
  • 2006-Sep thread about Running a script from procmail, which discusses a redlist (aka blacklist) recipe, but it is essentially the same as a greenlist (aka whitelist) recipe
  • 2006-June-11 message from Michael J Wise about Filter mail based on a text file list?
  • 2004-Nov-30 message from me about Whitelisting/Greenlisting (Was: help for a newbie)
  • 2003-Nov-Dec thread about Simplest Whitelist, which is archived at Gmane.org in this and this thread, and at xray.mpe.mpg.de starting with this message
  • 2001-Jul thread about some spammers sneaking into my greenbox, which starts with this message
  • Dallman Ross's 2004-Feb-22 message about whitelisting expected reply addresses
  • My 1998-May-27 message about my whitelist strategy - feedback welcome
• Scott Wiersdorf's Some Useful Procmail Recipes -- use your web browser to search for whitelist
• Eric A. Hall's Mail::SpamAssassin::Plugins:LDAPfilter, which “was developed for the purpose of being able to store blacklist and whitelist data in an LDAP server.”
• TextDrive/Joyent Forums » SPAM and Security » Using Procmail to filter whitelist - Not working, getting errors, which references this procmailrc and this procmailrc at the dpaste.com pastebin
• SPF (Sender Policy Framework) -- SPF is an open standard that can be used to ensure that only email that is truly from you is used to update your greenlists and bluelists (e.g. via technique 4A, 4B, or 4G above). Also, if your greenlisted correspondents use SPF, it will be much easier to correctly identify if a message is from one of your greenlisted addresses.
• My del.icio.us bookmarks related to Greenlisting and SPF.

---
¹ Some people and programs stratify messages using whitelists and blacklists, and call the messages that are in between these extremes “gray.” I prefer to use the metaphor of the full color spectrum to stratify my messages and that's why I deliver messages to magenta, violet, blue, green, lime, yellow, red, and infrared mailboxes. The terms “whitelist” and “greenlist” are discussed in this message and this message in the Procmail list.

² In the ideal world, mailing-list management software would detect (& block!) viruses & spam, and not send these to the list members. But because 1) everyone, including mailing-list management software, can be infected; 2) viruses can be engineered to sneak into green & blue boxes; and 3) viruses can be destructive, I recommend that you check all messages for viruses.

³ When thinking about how to implement tools and procedures for maintaining greenlists and bluelists, we need to remember that most users are not going to have (or want) direct access to their server or their procmail recipes. We also cannot assume anything about their email client and we do not want to force them to do updates through a web browser (although that could be an option). [It sure would be nice to be able to assume clients and servers that support IMAP mailboxes, IMAP keywords, IMAP annotations, and server-side address books though!]

Pulling the Plug on Procmail

Once the .forward and .procmailrc files are in place and readable by your message transfer agent, procmail is plugged in and will be invoked each time a message arrives for you. If you have a problem, such as losing messages or a mail loop, you can quickly unplug procmail by getting rid of your .forward file. One way to do this is to rename it by typing the following:

  
cd
mv .forward .forward.OFF

Just in case procmail is your system's local delivery agent (or becomes the LDA in the future), it is a good idea to also rename your .procmailrc file by typing:

  
mv .procmailrc .procmailrc.OFF

To turn procmail back on, type the following:

  
cd
mv .forward.OFF .forward
mv .procmailrc.OFF .procmailrc

	Did you notice I violated my naming style by using capital letters in a file name?! Bear in mind the wise words of Ralph Waldo Emerson in his essay Self-Reliance: A foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines.

Troubleshooting

If messages are not delivered correctly, here are steps you can use to try to solve the problem.

Look at Your Log File

Look at your $LOGFILE ($HOME/Procmail/pmlog) to see if you can determine what the problem is.

Look for Typos & Misplaced Comments in Your Configuration Files

Check these, and any other Procmail configuration files that you are using, for typos.

~/.forward
~/.procmailrc
~/Procmail/rc.testing

Also, if you have comments within a recipe, move them completely outside the recipe, for example, like this:

# comment
# another comment
:0:
rest of the recipe with no embedded comments

Comments are not allowed in some parts of a recipe and it is safest to not use them at all within a recipe. For example, if you specify a conditon like this:

* condition             # comment

Make Sure Your Configuration Files Use LF (Line Feed) Line Terminators

Make sure that each configuration file is a Unix ASCII text file with Line Feed (aka \n or ^J or ASCII character 10 or LF) line terminators. To view all characters in a file, both printing and non-printing characters, run this command:

cat -vet filename | less
     |||
     ||+--display tab as '^I'
     |+--display end of each line as '$'
     +--display non-printing characters

If filename contains a Carriage Return character, which will almost always cause a problem in a Procmail rc file, it will be displayed as ^M by this variation of the cat command.

Alternatively, on some (‡) sytems, you can run this command:

file filename

Or, to check all files in your Procmail directory, run this:

file ~/Procmail/*

Each Procmail configuration file should be listed as one of the following.

ASCII text 
ASCII English text

If a file was created or edited on a non-Unix system and transferred using Copy & Paste, Samba, or a non-ASCII FTP transfer, the file might be listed as one of these:

ASCII text, with CRLF line terminators
ASCII text, with CRLF, CR, LF line terminators

Each of these means that there are lines in the file that are terminated with something other than the Line Feed character.

	(‡) Not all versions of the file command distinguish between the different types of ASCII files -- does anyone know which versions of file make this distinction??? To learn more about the  file  command, see the  file  man page at gsp.com, linux.gr or (preferred) on your system.

Since Unix uses only the Line Feed character to terminate a line, the Carriage Return character (aka \r or ^M or ASCII character 13 or CR) is interpreted to be a regular character by Procmail and it will almost always cause a problem. For examples of this causing a problem in Procmail and suggestions for how to fix it, see the following:

• Procmail Mailing List
  • Procmail won't use existing mailbox, especially David W. Tamkin's reply
  • Ways to check for CRLF? (Was: Error???), especially Jason Tishler's suggestion to use od piped through fgrep (FYI, od is octal/decimal/hex/ASCII dump)
• Vim Online
  • Vim Tip #26: Getting rid of ^M - mixing dos and unix
  • Vim Tip #379: 1,$ s/^M//g gets rid of control-Ms (windows carriage returns)
• sed
  • HANDY ONE-LINERS FOR SED (Unix stream editor)
  • Seders grab bag

Tip 1: An easy way to fix the carriage-return problem in a single file is to open it in either nano or pico and then simply writeOut and eXit. For example, run  pico -w filename  and then:

Press...  To...
========  =====
^O        writeOut
ENTER     accept the filename
^X        eXit

This will convert the file to Unix format (LF line terminators) if it's not already in Unix format.

Tip 2: Another option is to use the command-line tool dos2unix, which is also known as tofrodos.

Tip 3: To learn about ASCII file transfers, see Dan Tobias's Use ASCII mode for HTML FTP transfers.

Check File and Directory Permissions

Use the instructions below to check the file and directory permissions of your .forward file. (These were set with the chmod commands in Step 11.)

	Not all systems require the .forward file to be world readable so you might be able to set your home directory permissions to drwx------ and your .forward file permissions to -rw-------. On many systems the .forward and .procmailrc files will not be used if they are group or world writable. Whether your system has this limitation or not, you probably do not want other people to be able to edit your .forward and .procmailrc files!

Turn on Full Logging

LOG="
"
VERBOSE=yes

Test procmail by sending yourself messages as described in Steps 12 and 13 again. Look at your $LOGFILE, which will contain verbose entries and a blank line between each message's log entry (that's what the two-line LOG setting does), to see if you can now determine what the problem is.

If you can't figure out what's going on in your log because messages are arriving simultaneously and their log info is intertwined, see Bart Schaefer's message Re: Procmail log intelligibility and David W. Tamkin's followup message in the Procmail mailing list.

After you decipher your log and get procmail to work, you probably will want to set VERBOSE=no and "unplug" the LOG line by preceding it with # (or ## if you are using my style).

See if There is a Global procmailrc

If there is a global procmailrc file on your system, its settings and recipes are used by all users on the system and it is read before your personal procmailrc file. It often resides in one of the following files:

/etc/procmailrc
/usr/local/etc/procmailrc

If neither of these files exist on your system, it is possible that your system administrator configured Procmail to use a different global rc file. The location of the global rc file may be documented in the Procmail manual page on your system. To see how your local Procmail is configured, type

man procmail

followed by

/global

The second command searches forward in the man page for the string  global  and will jump to a line that looks something like this:

/etc/procmailrc        initial global rcfile

If the manual page is from the same install as the Procmail binaries, it should contain the correct path to the global rc file. If neither the procmail man page nor exploring your system turns up a global procmail rc file, ask in your system's discussion group or ask your system administrator.

Another thing to look for on your system is the directory /etc/procmailrcs. To see if this directory is on your system and to list the directory's contents if it exists, type

ls -l /etc/procmailrcs
                     ^
                     make sure you type this s! 
  

To learn more about the global procmailrc file, the files in the procmailrcs directory, what user the recipes in these files are run as, and how they relate to the user procmailrc file, see:

• Step 2c in Step by Step Through Setting Up and Testing Procmail above
• man procmail
• Scott Wiersdorf's 2004-March-22 message Re: Running user procmailrc, which includes a nice description of “How Procmail Fits into Local Mail Delivery.”
• Sean B. Straw's 2003-June-03 message Re: procmailrc flow -- global, per-user?, which discusses root vs. user privileges, using DROPPRIVS=yes, SWITCHRC, and a lot more
• Ruud H.G. van Tol's 2005-Feb-04 message Re: exit current script before end of aka goto|return|exit|jump and the entire thread, which discusses SWITCHRC and more. Especially useful is this comment by Ruud:

  “SWITCHRC is like a GOTO, INCLUDERC like a GOSUB.”

Check $SHELL and $PATH

If you invoke programs from your .procmailrc, it is possible that either the SHELL or PATH environment variable that procmail is using is causing problems. To check the settings that procmail is using, put the following three lines in your .procmailrc after the line that sets the LOGFILE environment variable.

LOG="My PATH is currently $PATH
My SHELL is currently $SHELL
"

The third line contains only the double quote character. It closes the beginning quote, which is on the first line, and inserts an extra linefeed (and thus a blank line) in your log to make it more readable.

Next, send yourself a message so that the commands in your .procmailrc are run and the above lines are appended to your $LOGFILE. Now view the tail of your log using for example

tail -n 30 $HOME/Procmail/pmlog

This displays the last 30 lines of your log. If $SHELL is anything other than the Bourne shell (sh), the Bourne again shell (bash), or another Bourne shell derivative, put a line like this at the top of your .procmailrc file.

SHELL=/bin/sh

Of course you should make sure that /bin/sh exists on your system! To determine the path to sh on your system, run one of the following:

which sh
type sh
whereis sh
where sh
locate sh

For some information about the problems with non-Bourne-derived shells, see the 2004-Jan-29 message Re: forward to multiple users by David W. Tamkin.

If your $PATH does not contain all the directories from which you run programs and if you don't always specify the absolute path to programs, you need to add the needed directory (or directories) to your $PATH. For example, if you installed a CVS version of SpamAssassin (SA) in your $HOME directory and if you want to invoke that version of spamassassin or sa-learn from Procmail, you may want to add a line like this

 PATH=$HOME/SAusr/local/bin:$PATH

This line should be put near the top of your .procmailrc. For example, immediately after the SHELL setting.

	“... you shouldn't alter PATH unless there's something actually missing from it or the order is wrong.”
	-- Philip Guenther in a message in the procmail mailing list

Check $DEFAULT and $ORGMAIL

In this Quick Start, I suggest that you do not explicitly set the Procmail DEFAULT and ORGMAIL environment variables, which are used to specify the default inbox (defined above) and the ORiGinal default inbox, respectively. Instead I suggest that you use Procmail's default DEFAULT and default ORGMAIL settings because doing this . . .

• helps to ensure that Procmail will politely handle delivery errors, and
• satisfies my goal of simplicity, which I discuss above.

If you want to override the default DEFAULT setting, include lines similar to the following in your .procmailrc:

MAILDIR=$HOME/Msgs        # Make sure this directory exists!	
DEFAULT=IN-catchall

Note that the mailbox IN-catchall will be located in the MAILDIR directory (as long as the value of DEFAULT is specified using a relative path rather than an absolute path). If you want DEFAULT to be a maildir-formatted mailbox (rather than mbox-formatted), append a trailing slash (/) to the mailbox name. For example, use this:

DEFAULT=IN-catchall/

DEFAULT=.IN-catchall/
        ^
        Note the leading dot
 				

In any case, the Procmail MAILDIR variable, which is an entirely different entity from maildir mailbox format, should be specified in your .procmailrc without a trailing slash.

If you want your DEFAULT maildir-formatted mailbox to be at the top level of your Procmail MAILDIR directory, you can put something like these two lines in your .procmailrc:

MAILDIR=$HOME/Maildir        # Make sure this directory exists!	
DEFAULT=$HOME/Maildir/
                     ^
                     Note that DEFAULT includes a trailing '/' and MAILDIR does not

Alternatively, you could use these two lines:

MAILDIR=$HOME/Maildir
DEFAULT=$MAILDIR/
        ^^^^^^^^
          Note

Or these two lines:

MAILDIR=$HOME/Maildir
DEFAULT=./
        ^^
        Note

I prefer to explicitly set DEFAULT to $HOME/Maildir/ rather than $MAILDIR/ or ./ because there are sections of my procmailrc where I change Procmail's current working directory ($MAILDIR), and I want DEFAULT to have a fixed value, i.e., be independent of the cwd.

On a Courier IMAP server with a default configuration, these settings will make your Procmail “default inbox” or “system mailbox” (i.e., the mailbox that is specified by the Procmail DEFAULT variable) be the mailbox that is served when an IMAP client sends a request for the IMAP INBOX. Note that INBOX is a special IMAP name and the IMAP protocol translates this special name to the actual default mailbox on that particular IMAP server. Examples of actual default mailboxes include:

/var/spool/imap/user/username
/var/spool/mail/username 
/var/mail/username
/usr/spool/mail/username
/usr/mail/username
$HOME/Maildir (often used on a Courier IMAP server).

[Yes, it is confusing that the word maildir has three different meanings — the name of the mailbox format (maildir), the name of the Procmail variable (MAILDIR), and the name of the directory (Maildir)!]

You can check $DEFAULT and $MAILDIR, which $DEFAULT may depend on, by using the technique described in the previous section, i.e., by putting the following three lines in your .procmailrc after the lines that set the LOGFILE, MAILDIR, and DEFAULT environment variables.

LOG="My MAILDIR Directory is currently $MAILDIR
My DEFAULT Inbox is currently $DEFAULT
"

If Procmail is unable to deliver to $DEFAULT, for example if there is no space available, it will attempt to deliver to $ORGMAIL. To determine $ORGMAIL, run:

procmail -v

and look at the value of “Your system mailbox.”

	Make sure that Procmail is able to both create and write to the mailboxes $DEFAULT and $ORGMAIL. If you want to change the value of $ORGMAIL, it is best to specify this in the procmail source code and recompile procmail. The $ORGMAIL mailbox must not be a symbolic link or a file with more than one hard link. If you are using maildir-formatted mailboxes, it is best to specify both $ORGMAIL and $DEFAULT in the procmail source code and recompile. This is discussed above in Details Specific to maildir Format.   Question: What, if any, problem will occur if you set DEFAULT to a maildir-formatted mailbox in your .procmailrc rather than in the source code???

Where is my Mail?

The first thing to do when you are trying to figure out where Procmail delivered your mail is to look at your log file. If it is hard to interpret your log file, or if you don't have a log file, good places to look for missing messages are your $DEFAULT and $ORGMAIL mailboxes. If Procmail cannot deliver a message to the intended mailbox, it will try to deliver it to $DEFAULT. If it cannot deliver to $DEFAULT, it will try to deliver to $ORGMAIL. Details are in the previous section, Check $DEFAULT and $ORGMAIL.

Does ~/.forward Exist?

To see if ~/.forward exists and display it if it does, type the following

cat ~/.forward

If this file exists and if your system is set up to use it, it is invoked before Procmail. If you want your mail to be processed by the recipes that are in your ~/.procmailrc file, set up your system so that your ~/.forward file either

• does not exist (this is the best solution on most modern systems), or
• contains only the one line that is needed to invoke Procmail.

If you want to both forward messages and use Procmail, do the forwarding via a Procmail recipe.

For more about the ~/.forward file, see Set Up Your .forward File (not required on most systems) above and the next section.

Try Alternate .forward Files (not relevant on most systems)

If you invoke procmail via a .forward file and you are still having problems, try these alternate .forward files and see era eriksson's Debugging Procmail Recipes at www.iki.fi/era/mail/procmail-debug.html.

	If your system uses procmail as its local delivery agent or uses an MTA that does not respect the .forward file, this section is not relevant to you.

The command you put in your .forward file depends on the version of Unix your system is running, the message transfer agent your system is using, and how your system administrator configured the system. On some systems the procmail manual page contains information that is specific to your system, including the best command to put in your .forward file. To see what is suggested in your system's procmail manual page, type the following at your system's Unix prompt:

man procmail | less

To search forward in the manual page for the word kludge, type:

/kludge

The command in the paragraph below the paragraph that contains the word kludge may be the best command to use on your system. If you want to read earlier text in the manual page, you can go up half a page by typing u.

If neither the .forward command in Step 9, which is

"|/absolute/path/to/procmail"

nor the one suggested in your system's procmail man page work, try a modified version of the following

"|IFS=' ' && exec /absolute/path/to/procmail -f- || exit 75 #nancym"

	Make sure you include all the quotation marks, both double (") and single ('). The vertical bar (|) is a pipe, and on a U.S. keyboard is typed as SHIFT-backslash. Replace /absolute/path/to with the correct path for procmail (see Step 0). Replace nancym with your user ID. You need to put your user ID in your .forward so that it will be different from any other .forward file on your system. This circumvents a design error in sendmail. If your system uses an MTA other than sendmail, the comment (what's after the #) is not needed but does not hurt. If procmail resides below your home directory, type the absolute path. Do not expect $HOME, ~, or a relative path to work in your .forward file, since commands in a .forward file are run from sendmail without setting up your normal login environment. The .forward file must be a real file

If that doesn't work, try these (with nancym replaced by your login name of course!):

"|exec /usr/local/bin/procmail #nancym"
"|exec /usr/local/bin/procmail USER=nancym"
"|IFS=' ';exec /usr/local/bin/procmail #nancym"
"|IFS=' ';exec /usr/local/bin/procmail USER=nancym"
"|/usr/local/bin/procmail #nancym"
"|IFS=' '&&p=/usr/local/bin/procmail&&test -f $p&&exec $p -f-||exit 75#nancym"

These formats can be tried in different combinations; and the leading "| can be tried as |", or vice versa. Make sure you replace nancym with your user ID!  era eriksson's Procmail FAQ includes a detailed explanation of the last .forward command.

Known Procmail Bugs

• As I discuss in Snagging Viruses above, I recommend that you do not use a recipe that has an H or B flag in a line that starts with :0. This will help you avoid a bug that is described (along with how to work around it) in the following messages.
  • Cannot get recipes to work properly by Philip Guenther, 2002 February 10
  • Re: Procmail lock by Dallman Ross, 2003 September 22
  • proctip0 - the H and B built-in variables by Scott Wiersdorf
  If you decide to use a recipe that uses the B flag in the :0 line, make sure that there is not — and never will be — an earlier recipe that has the H flag in the top line.
  
  
• The “Dropped F off From_ header” problem is discussed in the SpamAssassin-talk mailing list in the thread From being cut off; especially useful are this and this message. If you are having this problem, you might want to try procmail-3.23pre, which you can get at Barton E. Schaefer's Email page, and see if this fixes the problem.
  
  
• On 2005-Feb-28, Robert Allerstorfer started a thread about recommendation: always end a rc file with a newline. Ruud followed up with a nice way to deal with this bug, which is to always have the last line of a Procmail rc file be something like this:

  # EOF (workaround for procmail-v3.22 bug)
  	

For more about Procmail bugs and patches for fixing them, see Philip's TODO list for procmail, Philip's list for procmail warts, the Procmail HISTORY file, and the October 2004 thread in the Procmail list about Who is the procmail maintainer?

More Troubleshooting Links

If you are still having problems, see:

• era eriksson's
  • Debugging Procmail Recipes at www.iki.fi/era/mail/procmail-debug.html
  • “Help, I get this error message ...” section of the Procmail FAQ at www.iki.fi/era/procmail/mini-faq.html#debugging
  
  
  
• Sean B. Straw's Procmail documents, especially his Procmail sandbox and his diagnostic script, procdiag.sh
  
  
• Prof. Timo Salmi's How can I test individual procmail recipes? I do not wish to disturb my regular ~/.procmailrc recipes file in the process. and I tried out your tips, but some of them failed on my system. What next?
  
  
• Dallman Ross's vsnag.point-n-shoot.sh shell script, which is part his Virus Snaggers (vsnag) package

See Also

The Fine Man Pages

Make sure you read the six Procmail man pages on your system. If the manual pages are from the same install as the Procmail binaries on your system, they will document the configuration that is specific to your system. To find out which Procmail version is documented, type man procmail and then type  /SOURCE  to search for the SOURCE section. For example, here is the SOURCE section of man procmail at linux.gr and monkey.org.

If you want to read or link to hypertext man pages, here are the apple.com 3.22, gsp.com (version 3.22 I think), linux.gr 3.15.2 and monkey.org 3.15.1 man pages. Some links in the monkey.org man pages are broken so I recommend that you use the gsp.com or linux.gr pages.

• man procmail at apple.com, gsp.com, linux.gr and monkey.org
• man 5 procmailrc at apple.com, gsp.com, linux.gr, and monkey.org
• man 5 procmailex at apple.com, gsp.com, linux.gr and monkey.org
• man 5 procmailsc at apple.com, gsp.com, linux.gr and monkey.org
• man formail at apple.com, gsp.com, linux.gr and monkey.org
• man lockfile at apple.com, gsp.com, linux.gr and monkey.org
• man mailstat at apple.com, annys.eines.info

The following man pages are not part of the Procmail package, but may be useful to you.

• Scott Wiersdorf's Advanced mail filtering for Virtual Servers:  man 5 filtering at gsp.com, viaverio.com, and digitaldaze.com
• Scott Wiersdorf's Log file rotation made easy:  man savelogs at viaverio.com

Many more man pages are available at the FreeBSD Hypertext Man Pages, where you can choose from a drop-down list of over 100 flavors and versions of Linux and Unix. Some have recent Procmail man pages and some have ancient ones.

Other Docs Shipped with Procmail

You can get concise help for procmail, formail, lockfile, and mailstat by typing the following at the Unix prompt on your system

   procmail -h
   formail -h
   lockfile -h
   mailstat -h
 

More Procmail docs are sometimes in /usr/share/doc/procmail, but these are very outdated and you can get more current and robust Procmail information in the documents listed in the next two sections.

Links to Pages that Link to this Page

This is one of the most popular pages here at Infinite Ink. I am collecting a list of pages that link to this page so I can find out where everyone is coming from, to provide some links that might be interesting to readers of this article, and as a way to thank everyone who links to this page.

• www.procmail.org
• Infinite Ink (ii.com)
  • Processing Messages with Procmail (I haven't updated this page for a long time)
  • Reverse Spam Filtering: Winning Without Fighting
  • IMAP Service Providers: A Step in Dealing with Viruses, Spam, and Email Overload
• Google Web Directory > Computers > Software > Internet > Clients > Mail > Unix > Procmail (uses dmoz.org data and includes a Google PageRank for each page; this Procmail Quick Start has a high PageRank thanks to everyone who links to it!)
• Open Directory (dmoz.org): Software: Internet: Clients: Mail: Unix: Procmail
• Netcape > Computers > Software > Internet > Clients > Mail > Unix > Procmail (uses dmoz.org data)
• era eriksson's Introduction to Procmail, Procmail FAQ, and Procmail Links
• Jari Aalto's Procmail Tips
• Scott Wiersdorf's Proctut: Procmail Tutorials and his other Procmail docs
• Paul Chvostek's Procmail things (and more)
• Sven Guckes' Procmail
• Timo's Procmail Links
• The FreeBSD Diary: Procmail - A Simple Example
• University of California, Irvine > School of Information & Computer Science > computing support > email faq > Procmail Tips
• The Procmail Companion, Martin McCarthy
• Æleen Frisch's
  • More Mail Filtering with procmail at Linux Magazine | August 2001 | GURU GUIDANCE
  • Essential System Administration, 3rd Edition, published by O'Reilly & Associates; the “link” to this Quick Start is on page 614 (look for McGough in the book's index!)
• comp.mail.misc FAQ: Fighting email spam and anti-UBE pointers is archived at faqs.org and cs.uu.nl
• Apple Developer Connection > Mac OS X Server > Fighting Spam on Mac OS X Server (Feb 2004)
• Josh Tolbert's Installing and Using procmail as the LDA for sendmail under FreeBSD
• Matthias Friedrich's Mail Handling under Linux at SuSE > Private customers > Support > HowTos and FAQs
• L i n u x P e r f o r m a n c e T u n i n g : Email Server Tuning
• The APK Net E-Mail Abuse How-To Page
• Nic Wolff's Rejecting spam with a procmail accept list; his spam-x discussion boards contain some good tips too
• SpamCon Foundation: to reduce spam
• Bruno Postle's Filtering mail, which is part of his nicely designed and nicely written My first mutt
• Linuxlinks around the world
• Beedub's Wiki: Using the procmail program -- written for MH users but includes instructions that are useful to all Procmail users
• "Alternatives to mhook Programs" in Jerry Peek's MH & nmh: Email for Users & Programmers
• OSU (Oregon State University) ACM Electronic Workshops: Section x. Mail Filtering
• Diego Kuonen's Linked Links
• Berkeley Open Computing Facility (OCF): How to filter your OCF mail through shell access
• KBase > DreamHost > Email > procmail, sendmail, qpopper > Can I use procmail to filter my mail  [If you sign up with DreamHost and want to help Infinite Ink, use this link to initiate your registration and when you Create a New Dream Network ID, set  Referred By  to  iiDOTcom.]
• Csoft.net: Procmail micro-howto
• Dr. Pierre Barthelemy's Anti-virus scanning on Unix mail servers using public domain software (PDF)
• YourHosting.NL: Procmail webmail hosting / Email features op uw hosting account
• Parker Morse's Alternatives -- includes sections called Malicious-Code Scanning, Spam Blocking, and General Reading
• Tucows > Internet > E-mail Utilities > Procmail
• The Notes section of the procmail man page mentions the incarnation of this page that was in the Filtering Mail FAQ. (When you are looking for information about a Unix tool such as Procmail, make sure you look at the man pages because sometimes they have clear concise explanations!)
• Use Yahoo! to search for pages that link to www.ii.com/internet/robots/procmail/qs/
• Use Google to search for pages that link to www.ii.com/internet/robots/procmail/qs/
•   The people who have added www.ii.com/internet/robots/procmail/qs/ to their del.icio.us bookmarks

More Links

• era eriksson's Procmail Quick Reference Guide
• Dallman Ross's 2003-Feb-22 message Re: furrin character sets
•   My del.icio.us bookmarks that are tagged Messaging/Procmail (which is how I tag all my Procmail-related bookmarks)
•   All bookmarks from all del.icio.us users that are tagged procmail
• 
  

Procmail Discussion Groups

Public discussion groups are a tremendous resource and they — along with the fine manual — are where I learned almost everything I know about Procmail. For details about the Procmail mailing list, including how to subscribe and unsubscribe, see mailman.rwth-aachen.de / mailman / listinfo / procmail. In addition to participating in the list via email, you can access — and in some cases participate in — the list using any of the following.

• HTTP client (e.g., a web browser):
  • www.xray.mpe.mpg.de/mailing-lists/procmail/ -- MHonArc style, no address munging, archives go back to 1995 and are searchable
  • mailman.rwth-aachen.de/pipermail/procmail/ -- Mailman style, archives go back to 2000
  • mhonarc.org/archive/html/procmail/ -- MHonArc style, addresses are munged, archives go back to 1995 and are searchable
  • marc.theaimsgroup.com/?l=procmail -- white text on black background, archives go back to 1995 and are searchable
  • info.ccone.at/INFO/Mail-Archives/procmail -- MHonArc style, no address munging, archives go back to 2000 and are searchable from the top-level Mail-Archives page
  • archive.netbsd.se/?ml=procmail
  • news.gmane.org/gmane.mail.procmail -- newsreader style (collapsed threads, frames), archives go back to 2002 and are searchable, option to post new or reply to old messages
  • blog.gmane.org/gmane.mail.procmail -- blog style (user-configurable layout and skin, no frames), archives go back to 2002 and are searchable, option to post new or reply to old messages
    
    
• NNTP client:  nntp://news.gmane.org/gmane.mail.procmail or news://news.gmane.org/gmane.mail.procmail -- posting to the list via NNTP is allowed, archives go back to 2002 and can be searched by your NNTP client
  
  
• RSS/Atom/XML feed reader:  Gmane.org provides four feeds of the Procmail list; details are at Information about gmane.mail.procmail
  
  

For an overview of the various Gmane incarnations of the Procmail list, including a form for searching the Gmane archive of the list, see Information about gmane.mail.procmail. Currently the Gmane archive of the Procmail list goes back to February 2002, but it would be great if someone would submit the entire Procmail list archive to Gmane. If you would like to support Gmane and get rid of the Gmane Google ads, donate money to Gmane.

There are a lot of other places around the Net where people discuss Procmail, including the groups that show up in the results of these searches:

• Search Gmane for procmail
• Search Google Groups for procmail

If you visit the Usenet group comp.mail.misc (c.m.m), please ignore the anonymous coward, his many aliases (one of which is AC), and any thread started by AC or an AC alias.  Tip: To maintain (or restore!) your sanity, I recommend avoiding c.m.m altogether.

	Many archives munge email addresses in order to hide them from email-address harvesters. But this is a problem when the discussion is about an email address (or about a fragment of an email address). For example, a 2004-Nov-26 message from Ruud H.G. van Tol with Subject “Re: alleged bug in ^TO_ macro: character '+' *is* allowed in emails” was archived in many different ways, including: Unmunged:  www.xray.mpe.mpg.de/mailing-lists/procmail/2004-11/msg00231.html Unmunged: info.ccone.at/INFO/Mail-Archives/procmail/Nov-2004/msg00231.html Munged (“@” converted to “(_at_)”):  mhonarc.org/archive/html/procmail/2004-11/msg00230.html Munged (“@” converted to “ at ”):  mailman.rwth-aachen.de/pipermail/procmail/2004-November/021038.html Munged (“@” converted to “ <at> ”):  article.gmane.org/gmane.mail.procmail/12984 Unmunged:  article.gmane.org/gmane.mail.procmail/12984/raw (you need to type /raw yourself because Lars said “You shouldn't link to raw articles.”) Tip:  At Gmane you can view the unmunged version of a message by appending /raw to the message's URL. This is discussed in the Gmane FAQ.

Search

In addition to searching discussion groups, which is discussed in the previous section, you can use the following links to search for Procmail-related tools and documents.

• Search for procmail in projects at FreshMeat.net
• Search SourceForge.net for procmail
•   If you have an account at del.icio.us and if you are currently logged in at del.icio.us, you can search your del.icio.us bookmarks for procmail

You can also use this form to search the web, the Infinite Ink site, or the Deflexion.com site (my blog) for information about Procmail (or about anything!).

Ads

Questions to You the Reader

• If you know about Procmail and maildir-formatted mailboxes, I'd appreciate your feedback on the Maildir section above, especially the questions I ask.
  
  
• What flavors of Unix, other than Red Hat Linux 6.0 and later, use procmail as the LDA? I'd like to add this information to the .forward section above.
  
  
• Is $HOME/nsmail the default mailbox directory for for all versions of Netscape Messenger? Are there any special instructions for using Procmail to deliver to Messenger mailboxes other than the link I give in the Mailbox Locking section above?
  
  
• What is the default mailbox directory for other mail or NNTP clients? I am especially interested in the default remote IMAP-accessible mailbox directory that different IMAP clients use. Does the default remote IMAP-accessible mailbox directory depend on the IMAP server software? If so, what are the defaults for different IMAP servers? Do clients automatically detect the default directory or does the user need to specify this in the client?
  
  
• What do you think is the best command for people to try first in their .forward file? Is what I've got in Step 9 the best?
  
  
• Does anyone have a URL for a page describing dmail or tmail, which are part of the UW imap-utils package (ftp.cac.washington.edu/mail/imap-utils.tar.Z)? These are used to deliver new mail directly to MBX-format mailboxes. tmail is used for direct sendmail calls and dmail is used for calls from procmail.
  
  
• I am using this page as a testing ground for how to do layout and design for technical writing on the Web. I am still not that happy with the design so if you have some ideas or pointers to good tech writing on the web, please let me know. Once I figure out a design I like I am going to update all my pages.

Author Blurb for the Book (feedback is welcome on this too!)

Nancy McGough started Infinite Ink, www.ii.com, in 1992. Infinite Ink is dedicated to writing, teaching, and publishing about mathematics, science, computing, and philosophy. Before she started Infinite Ink, Nancy worked at Microsoft first as a technical writer and then as a program manager designing tools and strategies to improve communication and information access within Microsoft. The project, called Integration of Information Access Tools (IOIAT), included WinBBS, a MS Windows front end to Usenet News. Nancy has taught mathematics and computer science courses at universities, colleges, and at a grade school. She is a fan of standards-compliant, modular, open-source software, and she is especially interested in intertwingular software and software that helps people communicate and collaborate. Her primary software tools are Pine and Mulberry for IMAP messaging; nn and Pine for NNTP messaging; Dreamweaver and Vim for web page development; Vim for powerful text editing; SecureCRT for ssh connections & port forwarding; Powermarks for intertwingular bookmark management; Info Select for intertwingular information management; and (of course!) Procmail for automated message processing. Her favorite operating systems are Mac OS X & FreeBSD and her favorite mail- and web-hosting providers are Verio and JohnCompanies.com.

Thanks

Thanks to Philip Guenther, the current former Procmail maintainer, for sending me feedback on the 23 April 2001 incarnation of this page and for all the work he's done on Procmail.

Thanks to John Levine of Taughannock Networks for giving me an account on his system which is running Courier IMAP, Qmail, and Procmail. It's because of this account that I was able to learn about and write about delivering to maildir-formatted mailboxes.

Thank you to these people who sent me feedback on this Procmail Quick Start:

• era eriksson (also sometimes at www.iki.fi/era/ or www.iki.fi/era/procmail/ or en.wikipedia.org/wiki/User:Eeera)
• Rik Kabel
• Christopher Lindsey
• Jeffrey McDonald
• Michael Meltzer
• Léon Planken
• Herur Venkataramaiah Krishna Prasad
• Dallman Ross
• Brock Rozen (brozen@tora.org)
• Andrew Starr
• David W. Tamkin
• Uriel Wittenberg

And to these people who sent me feedback back in 1994 when I first wrote the Filtering Mail FAQ:

• David L. Miller
• Cookie Monster
• Jim Showalter
• David W. Tamkin
• Rick Troxel
• Stephen R. van den Berg, the creator of procmail
• Syd Weinstein

Thank you also to the developers of procmail, especially the creator, Stephen R. van den Berg, and the current maintainer, Philip Guenther; the people who have written procmail documentation; the members of the Procmail mailing list; and everyone else who has sent me feedback. Please let me know if I have left you out of my thank-you lists, or if you know the URL of any of the unlinked people in the lists.

Thanks to Norm Walsh and Lenny Muellner for letting me use these DocBook admonition graphics:

  Tip
  Note
  Important
  Warning

Finally thank you to my friends, family, and Internet acquaintances who have helped me to make Infinite Ink what it is. Special thanks to Bob Tinkelman of New York Net for giving me a free T1 line to my apartment (wow!) and to Sean Shapira of Jazzie Systems for giving me free web space and bandwidth for years.