spamheader.muse

<contents>

* An analysis of the X- headers in my spam

** Introduction 

[[http://www.faqs.org/rfcs/rfc822.html][RFC 822]] allows people to define and use additional header fields in
their emails.  These fields start with =X-= in their names.  For
example, =X-Quarantine-ID= or =X-Spam-Score=.  Usually these headers are
not too interesting.  I wanted to see if the spammers use any custom
fields in the spam messages that they send out.

* Analysing spam

I took a spam message and checked its X- headers.  The following are
the X-headers that I think can be ignored

 - X-MimeOLE
 - X-Spam-Status (added by spamassassin)
 - X-Account-Key, X-Mozilla-Status2 and X-Mozilla-Status are added by Thunderbird
 - X-Mailer
 - X-Original-To
 - X-Quarantine-ID

Any headers outside of the above list promises to be interesting.  

** Summary: X-headers in my spam

I ran a script on my spam mbox file (i.e. mails that have been marked
as spam by spamassassin) to see what I can dig up. The following X-headers
look interesting, because they occur so few times.  (caveat: I don't
think I have enough spam)

 X-DSNContext ::
 This seems to be header set by MS Exchange

 X-AntiVirus: Checked,  ::
 This seems to be header set be spammers to confuse the mail filter

 X-ME-UUID ::
 Looks like header set by some mail server

 X-Source, X-Source-Args, X-Source-Dir, X-Spam ::
 These are headers set by [[http://www.phplist.com/][phplist]]

 X-Failed-Recipients ::
 This is a header generated by exim when it handles mailing lists.
 Looks like some spammer has a mailing list with all his targeted
 email ids in it.


** Summary: X-headers in non-spam email

I ran my script on a mbox which had emails from one of the mailing
lists I am on. Some interesting headers
 
 X-AOL-IP ::
 Self explanatory

 X-BigFish ::
 Ummm --- no clue
 
 X-Disclaimer ::
 Hehe
  
 X-Ninja-AttachmentFiltering, X-Ninja-PIM ::
 Seems like a rarely used mail client


** Script used in analysing the headers

This is a Perl script that uses regular expression to find the
interesting X-headers.  You can get the script from [[http://rajshekhar.net/download/spamheaders][here]].  The
interesting parts are explained below.

The =isinterestingheader= function filters out non interesting headers.

<example>
sub isinterestingheader($)
  {
    my $header = shift;
    my @ignore_x_headers = qw/Original-To Spam-Status Mozilla-.*  Quarantine-ID Virus-Scanned 
			     Spam-Score Spam-Level Spam-Flag Amavis-.* Mailer Priority Account-Key
			     Priority /;
    my $regex_ignore_x_headers = join( "|", @ignore_x_headers );

    unless ( $header =~ /X-($regex_ignore_x_headers)/i) {
      #print "[$header]  is interesting and matched $1\n";
      return 1;
    }
    return 0;
    
  }
</example>

If you want to ignore/unignore any particular X-headers, you can
add/remove it to the =@ignore_x_headers= variable.

The loop below is the main workhorse.  It iterates over the mbox file
and if it finds a X-header, it extracts that, checks if the header is
interesting and if the header is interesting, stuffs it into the
=%xheaders= hash.

<example>
while (<INFILE>) {
    my $mline = $_;

    if ( $mline =~ /(^X-.*).*:\s(.+)/ ) {
      if (isinterestingheader($1)){
	countthisheader($1,$2);
	
      }
    }
  }

</example>



 
* Raw data

** Spam 

Result of running the script on my spam folder (2704 mails).  Another
note - UIDL is *Unique IDentification Listing*, hence the number X-UIDL
in the results below gives a count of the number of emails in the mbox
file.


**Header** | **Count** | **sample value** (comma seperated list)
X-Accept-Language    | 22 | -
X-AntiAbuse          | 15 | -
X-AntiVirus: Checked | 1 | 29.02.2004]
X-AntiVirus: Checked | 1 | 23.00.2006]
X-AntiVirus: Checked | 1 | 21.10.2007]
X-Antivirus          | 87 | -
X-Antivirus-Status   | 87 | -
X-Authenticated      | 1 | #08894883
X-BeenThere          | 3 | denmark@vendome-corniche.com, denmark@vendome-corniche.com, denmark@vendome-corn
X-DSNContext         | 1 | 335a7efd - 4523 - 00000001 - 80040546
X-Eagle-Notice: Send | 1 | [news #38922] Your doctor\222s advice
X-ME-UUID            | 2 | 20071018140851841.CD4AC1C00876@mwinf3106.me.freeserve.com, 20071018140851841.CD4
X-MIMEOLE            | 32 | -
X-MSMail-Priority    | 1817 | -
X-Mailman-Version    | 3 | 2.1.9, 2.1.9, 2.1.9
X-MimeOLE            | 1939 | -
X-OriginalArrivalTim | 3 | Thu, 13 Sep 2007 05:59:30 -0600, 07 Nov 2007 00:30:21.0042 (UTC) FILETIME=[612B7
X-Originating-Email  | 1 | [rajlist@rajshekhar.net]
X-Originating-IP     | 30 | -
X-Originating-Server | 2 | chilean.ebaystatic.com (labshost.com.moebelheinrich.de [110.96.122.184]), chilea
X-SMTP-Server-Queue- | 1 | 782D21C00887
X-SMTP-Server-Sender | 1 | rfc822; rajlist@rajshekhar.net
X-Scanned-By         | 1 | Digested by UGA Mail Gateway on 128.192.1.75
X-Sender             | 41 | -
X-Source             | 1 | /usr/local/cpanel/3rdparty/bin/php
X-Source-Args        | 2 | /usr/local/cpanel/3rdparty/bin/php /usr/local/cpanel/base/3rdparty/squirrelmail/
X-Source-Dir         | 2 | :/base/3rdparty/squirrelmail/src, agitatudo.com:/public_html/site/components/com
X-Spam               | 1 | Not detected
X-UIDL               | 2704 | -
X-USER_IP            | 1 | 74.0.178.152
X-Version            | 5 | 2.0.2, 2.0.2, 2.0.2, 2.0.2, 2.0.2
X-Warning            | 2 | yahoo.it is listed at postmaster.rfc-ignorant.org, yahoo.it is listed at abuse.r
X-Yahoo-Calendar-Iid | 1 | IxA1ic3%40o23I%40bQzFhIPy%40dabUuc%40KhH
X-Yahoo-Newman-Id    | 1 | tinaakimrs36#ib-1191688048-tinaakimrs36#ib:6
X-Yahoo-Newman-Prope | 1 | calendar-invite
X-me-spamlevel       | 1 | med
X-me-spamrating      | 1 | 77.426110
X-remove             | 1 | 66958

I ran the script on the mbox file that had been marked as spam by
Thunderbird (979 mail).  The results are

**Header** | **Count** | **sample value** (comma seperated list)
X-Accept-Language    | 1 | en-us
X-AntiAbuse          | 5 | This header was added to track abuse, please include it with any abuse report, P
X-AntiVirus: Checked | 1 | 21.10.2007]
X-Antivirus          | 43 | -
X-Antivirus-Status   | 43 | -
X-DSNContext         | 1 | 335a7efd - 4446 - 00000001 - 80040546
X-ELNK-Trace         | 1 | cfd75b53fdaf59ca3366623b7f6a8e4a89e6754a908ffd693db58eff436b9b054f89210dab4adbbb
X-Eagle-Notice: Send | 1 | [news #38922] Your doctor\222s advice
X-Failed-Recipients  | 1 | raj@rajrudolph.co.uk
X-MIMEOLE            | 18 | -
X-MSMail-Priority    | 713 | -
X-MimeOLE            | 734 | -
X-OriginalArrivalTim | 4 | 29 Oct 2007 08:27:42.0578 (UTC) FILETIME=[93234520:01C81A05], 03 Nov 2007 06:20:
X-Originating-Email  | 2 | [raj@rajshekhar.net] , [saurabh@live.com]
X-Originating-IP     | 4 | 209.86.224.50, [79.210.6.45] , [122.161.25.59], [193.220.212.16]
X-Originating-Server | 1 | [12.114.117.239] (HELO milftape.com)
X-Sender             | 1 | raj@rajshekhar.net
X-UIDL               | 979 | -


** Non Spam

**Header** | **Count** | **sample value** (comma seperated list)
X-AOL-IP             | 14 | - 
X-Accept-Language    | 5 | en-us, en, ja, en-us, en, ja, en-us, en, en-us, en, en-us, en, ja               
X-AntiAbuse          | 10 | - 
X-Authenticated      | 1 | #15864248                                                                       
X-Authenticated-Send | 1 | arackcheev@aktivsystems.ru                                                      
X-BeenThere          | 756 | - 
X-BigFish            | 3 | V, V, V                                                                         
X-Brightmail-Tracker | 24 | - 
X-Brightmail-scanned | 24 | - 
X-Disclaimer         | 1 | Added                                                                           
X-ELNK-Trace         | 95 | - 
X-EN-AuthUser        | 4 | skmpu@bhmk.com, skmpu@bhmk.com, skmpu@bhmk.com, skmpu@bhmk.com                  
X-EN-UserInfo        | 4 | 5af75cd46c1ca8979d7582ce798d9964:1d2a11e68e3a5cb29e3c548bab12dcb9, 5af75cd46c1ca
X-Enigmail-Version   | 97 | - 
X-Env-Sender         | 29 | - 
X-Flow-Control       | 1 | Sendmail Flow Controller v1.6.3 bemtal03.swift.com                              
X-Google-Sender-Auth | 11 | - 
X-IronPort-AV        | 10 | - 
X-IronPort-Anti-Spam | 4 | true, true, true, true                                                          
X-IronPort-Anti-Spam | 4 | Ah4FAP4Mg0aWZZpy/2dsb2JhbACBT4EY, Ao8CAA+3m0aWZZpx/2dsb2JhbACBSYEY, AgAAALLv+kaW
X-MB-Message-Source  | 14 | - 
X-MB-Message-Type    | 14 | - 
X-MDRemoteIP         | 1 | 192.168.101.13                                                                  
X-MIME-Warning       | 3 | Serious MIME defect detected (), Serious MIME defect detected (), Serious MIME d
X-MIMEOLE            | 22 | - 
X-MIMETrack          | 5 | Serialize by Router on DOMMEL01/AU/MYOB(Release, Serialize by Router on ghdom_ap
X-MS-Has-Attach      | 3 | yes, yes, yes                                                                   
X-MSMail-Priority    | 108 | - 
X-Mailman-Approved-A | 4 | Tue, 17 Oct 2006 14:51:47 -0700, Mon, 13 Nov 2006 13:41:57 -0800, Mon, 13 Nov 20
X-Mailman-Version    | 756 | - 
X-MimeOLE            | 252 | - 
X-Msg-Ref            | 29 | - 
X-NAIMIME-Disclaimer | 6 | 1, 1, 1, 1, 1, 1                                                                
X-NAIMIME-Modified   | 6 | 1, 1, 1, 1, 1, 1                                                                
X-Ninja-AttachmentFi | 1 | (no action)                                                                     
X-Ninja-PIM          | 1 | Scanned by Ninja                                                                
X-Nokia-AV           | 13 | - 
X-OriginalArrivalTim | 105 | - 
X-Originating-Email  | 9 | [cbeiter@hotmail.com], [rmcilree@hotmail.com], [rmcilree@hotmail.com], [rmcilree
X-Originating-IP     | 142 | - 
X-Proofpoint-Spam-De | 1 | rule=notspam policy=default score=0 spamscore=0                                 
X-Proofpoint-Virus-V | 1 | vendor=fsecure engine=4.65.5446:2.3.11, 1.2.37,                                 
X-Provags-ID         | 1 | V01U2FsdGVkX1+D+zVCqUODw33xCMTOmmtG9xwhGTZTbjCAY9UNoU                           
X-Provags-ID2        | 1 | V01U2FsdGVkX18ARz/emDKd0zOFIeqrkoNaQg3vKbE987xNH1i                              
X-Return-Path        | 1 | arackcheev@aktivsystems.ru                                                      
X-Scan-Signature     | 3 | 59fbbbeddf57f03b6f4e4ac3778d6a97, 95f51ce3d4c4e0f1e1d6f3eecbbcfce0, 95f51ce3d4c4
X-Scanner            | 1 | InterScan AntiVirus for Sendmail                                                
X-Sender             | 9 | cbeiter@hotmail.com, rmcilree@hotmail.com, rmcilree@hotmail.com, rmcilree@hotmai
X-Sender-IP          | 1 | 128.143.197.234                                                                 
X-Spam-Processed     | 1 | aktivsystems.ru, Mon, 28 May 2007 16:46:53 +0400                                
X-StarScan-Version   | 29 | - 
X-TM-AS-Product-Ver  | 42 | - 
X-TM-AS-Result       | 42 | - 
X-Tiga.PelayanWeb.co | 2 | Found to be clean, Found to be clean                                            
X-Tiga.PelayanWeb.co | 2 | pwawood@mango.net.my, pwawood@mango.net.my                                      
X-Tiga.PelayanWeb.co | 2 | Please contact the ISP for more, Please contact the ISP for more                
X-UIDL               | 756 | - 
X-URL                | 2 | http://mail2web.com/, http://mail2web.com/                                      
X-UVA-Virus-Scanned  | 3 | by amavisd-new at fork2.mail.virginia.edu, by amavisd-new at fork6.mail.virginia
X-UVa-Vac-OK         | 1 | 1                                                                               
X-Virus-Status       | 35 | - 
X-VirusChecked       | 29 | - 
X-VisionSystems-Mail | 7 | Found to be clean, Found to be clean, Found to be clean, Found to be clean, Foun
X-VisionSystems-Mail | 7 | david.growse@vision-fs.com, david.growse@vision-fs.com, david.growse@vision-fs.c
X-WiganSS            | 1 | 0100000001001Fwhexchmb03.bsna.bsroot.bear.com                                   
X-Y-GMX-Trusted      | 1 | 0                                                                               
X-YMail-OSG          | 4 | IDs9bdsVM1lXZWLQ70iCLuQtkVuem5I9pruupVhilWv27I1aV34UeauN0WeYW.SZt4b8nZGDwkxfOP9R
X-bsdisclaimer       | 1 | standard                                                                        
X-eXpurgate-Category | 3 | 1/0, 1/0, 1/0                                                                   
X-eXpurgate-ID       | 3 | 149371::070228103941-0F893BB0-01554729/0-0/0-1, 149371::070311184930-3F13DBB0-4A
X-imss-approveListMa | 2 | *@optus.com.au, *@optus.com.au                                                  
X-imss-result        | 2 | Passed, Passed                                                                  
X-imss-version       | 2 | 2.047, 2.047