SPAM SLAM by pengxuebo

VIEWS: 10 PAGES: 6

									COVER STORY                       Spam Test




Comparing antispam appliances and services


SPAM SLAM




                                                                                                                                                BMW AG
Spam filters can help smooth the waves in your inbox, as long as they

are reliable and don’t have too many side effects. We’ll show you

what we found when we tested five antispam appliances and two                                 mail delivered to the account junk will
                                                                                              make it past the filter. Some bulk mail
service providers. BY TOBIAS EGGENDORFER                                                      programs even integrate SpamAssassin
                                                                                              [3] in order to test mass mail before
                                                                                              sending it on its way. The spammer will



F
        iltering spam as it enters the net-   spam. However, spammers constantly              modify the text until the message finally
        work keeps control in the hands       develop new obfuscation methods with            slips past popular filters.
        of IT professionals, increasing the   the aim of tricking existing filters, and          Thus, the detection rate for known
effectiveness of the filter and improving     they continually test their junk mail           spam does not say much about the qual-
user productivity by reducing the glut of     against known filtering techniques.             ity of a filter’s heuristics. It is hard to say
mail. In previous issues of Linux Maga-         Spammers often set up email accounts          how a filter will react to the daily flood
zine, we have outlined various software-      with major mail providers to see if junk        of spam based on historic data. This is
based scenarios for integrating spam fil-
tering with a mail server. In this article,                                  Mail Exchange in DNS
we’ll look at some alternatives to con-
                                               A Mail Transfer Agent (MTA) first discov-       the messages on to the internal mail
ventional software-based filters such as       ers the IP address to which it is permit-       server with the highest priority.
SpamAssassin. In particular, we’ll report      ted to deliver an email message [4]. To         In productive environments, this typi-
on some hardware-based spam filtering          do so, it extracts the domain name from         cally means that the highest priority MX
appliances, and we’ll examine a repre-         the mail address and sends an MX                is located on the company premises and
sentative pair of Internet spam filtering      Query (Mail Exchange) to a DNS server.          managed by the local IT department.
                                               The server responds with the IP address         Strict spam filtering typically applies
service providers.
                                               of the authorized mail server. For redun-       here. The carrier often has additional
                                               dancy, domains are allowed to have
The Test Environment                           multiple MX entries with different priori-
                                                                                               backup MX servers. These servers do
                                                                                               not usually apply filters, so this target is
The typical approach to testing spam           ties, where a lower value denotes a             far more lucrative for a spammer. If the
products is to take known spam from            higher priority.                                spammer succeeds in delivering their
large archives [2] and run the filters         The sending MTA establishes a connec-           junk mail, they have reached their target,
against it (see the “Filtering Techniques”     tion to the top priority MX and attempts        or at least managed to evade a number
box). At best, this will tell you whether      to deliver the message to the MX. If the        of filters, such as IP-based blacklists. The
the vendors have done their homework,          attempt fails, the MTA contacts the MX          blacklists will view the provider’s legiti-
                                               with the next highest priority. The sec-        mate, low-priority MX as the source,
and whether their filters have been
                                               ondary mail servers then attempt to pass        rather than as the spammer.
tested and optimized to combat known




32        ISSUE 76 MARCH 2007                    W W W. L I N U X - M A G A Z I N E . C O M
                                                                                                  Spam Test          COVER STORY




                                                                         When the appliances         have revealed much about the appliance,
                                                                       started to arrive – some      since detection rate is defined by the
                                                                       accompanied by a ser-         virus scanner the appliance deploys.
                                                                       vice engineer and oth-        However, a virus scanner can also re-
                                                                       ers by snailmail – each       duce the volume of unsolicited mail by
                                                                       device was assigned a         removing any worms it detects from the
                                                                       domain and the MX IP          mailbox. Some products will even let
                                                                       address (Figure 2).           you deploy multiple scanners at the
                                                                       This avoided the need         same time to compensate for inadequa-
                                                                       to change the DNS             cies of individual scanners, a capability
                                                                       data, which might             that is reflected in the licensing fee.
                                                                       have then affected the
                                                                       test results.                 Spam Quality
                                                                                                     Not all of the vendors we contacted actu-
Figure 1: Five appliances took part in our lab. From bottom to         SMTP Proxy                    ally provided test equipment, and this
top: Symantec Mail Security 8260, McAfee Secure Content              Of course, we needed            left three test domains unused – we had
Management Appliance 3200, Ironport C10 Email Security               to change the MX                originally planned for one domain to
Appliance, Canit Anti-Spam, and IKU Sponts-Box.                      entry to accommodate            provide comparative results for filtered
                                                                     the service providers,          versus unfiltered domains. What we
what prompted us to let the filters tackle       Expurgate and Spam Stops Here. To use               originally considered to be a drawback
unknown spam fresh off the Internet.             these services, you need to enter the pro-          turned out to be a bonus, giving us the
Simply redirecting an existing mail ac-          vider’s IP as the MX for your own do-               ability to ascertain a more stable mean
count would not be sufficient, because           main. Expurgate and Spam Stops Here                 value for the spam volume.
the spam battle starts with the SMTP di-         analyze and filter the incoming mes-                   As all the email addresses were known
alog. Thus, each test candidate needed           sages and forward them to the target                only to spammers, the mail servers re-
its own domain and its own DNS MX                mail server. This approach outsources               ceived only spam at first. This is useful
entry (see the box titled “Mail Exchange         spam filter management to the service               for ascertaining a filter’s detection rate,
in DNS”).                                        provider, however, relying on an external           however, it does not tell you anything
                                                 provider to manage your mail environ-               about false positives; and false positives
Four Months of Fresh Spam                        ment does mean having a lot of trust.               are a major criterion for defining spam
The first thing we needed to do was to             The appliances we tested work like in-            filter quality. A single legitimate mail
guarantee a continuous supply of spam            coming SMTP proxies: the external mail              message incorrectly classified as spam
to multiple addresses. To do so, we regis-       server uses SMTP to contact the appli-              by the filter can cause more damage
tered ten domains and published a web-           ance and attempt to deliver mail. If the            than letting ten spam messages through.
site with four email addresses in each           mail passes the filter, the appliance then             To ascertain the false positive rate, we
domain. The domains went online six              uses SMTP again to route it to the local            bombarded the candidates with legiti-
months before the test started, and many         mail server. It’s business as usual for the         mate messages toward the end of the
links from other pages pointed to them.          local mail server,
We added a selection of keywords to              and for the users
whet the harvesters’ appetites, and we           on the internal
sat back to wait for the spammers to             network who re-                                                      Internet
                                                                             Before
take the bait.                                   main blissfully
   About six months later, the domains           unaware of all
and the two webmail accounts were per-           this. They can                       Mailserver: x.x.x.1 to x.x.x.10
fectly prepared; they each received              continue to either                   Webserver: x.x.x.1
enough mail and similar amounts of               fetch mail from
spam. During the main test phase, each           the server or read
victim address received 50 to 100 fresh          their mailboxes on
spam mails.                                      the server itself.          After                                             Internet
   The mail server for the domains was a           We deliberately
machine with ten IP addresses assigned           avoided testing
to it. Each of the 10 IPs was entered as         the virus filtering                  Mailserver: x.x.x.1
                                                                                      Webserver: x.x.x.1
the MX for one registered domain. We             features of the ap-
used Sendmail to pick up the messages.           pliances. Virus                                                  SMTP Proxies
Sendmail simply passed the messages to           checking was be-
a small Perl script, quoting the envelope        yond the scope of        Figure 2: During the harvester phase (top), a mail server collected
to address; the script separated the body        our test, and for        spam for 10 IP addresses in 10 separated domains. During the test
from the header and entered the results          that matter, our         phase, the appliances were introduced as SMTP proxies for the
in a database.                                   test would not           addresses.




                                                     W W W. L I N U X - M A G A Z I N E . C O M              ISSUE 76 MARCH 2007           33
COVER STORY                        Spam Test




                                                            3 GHz. 2 GB RAM and two             Brightmail in June 2004. Among other
                                                            Raid 1 mirrored 73 GB hard          things, it uses SPF, various blacklists and
                                                            disks provide sufficient stor-      whitelists, URL filtering, and Sender ID
                                                            age capacity. The doubly re-        for detection purposes. Additionally, Sy-
                                                            dundant power supplies un-          mantec uses signature detection and a
                                                            derline the fact that the box is    hash-based comparative detection algo-
                                                            clearly targeted at the profes-     rithm. Each individual filter returns a
                                                            sional market. The operating        value used to score the spam probability.
                                                            system is Red Hat Enterprise        The values all contribute to a fixed, and
                                                            Linux 3.0. Root login is not        non-customizable, total spam score.
                                                            envisaged. According to                There is no way for us to disable the
                                                            Symantec, the system uses a         different filtering mechanisms on an in-
                                                            hardened version of Postfix.        dividual basis. However, it is possible for
                                                               The configuration interface      us to set individual values for the spam
                                                            is web-based via HTTPS on           score for user groups (defined on an
Figure 3: This hidden menu appears on the Symantec          port 41433. Initial configura-      LDAP server).
appliance when the admin presses [Shift]+[A] on the         tion can either use a serial           In our lab, the Symantec system
web interface. The Rapid Release option was set manu-       console or the local keyboard       achieved a perfect score of no false posi-
ally by the Symantic engineer. The SMTP Greeting            and display. The initial config-    tives, but it identified less than 90 per-
settings are also quite interesting.                        uration simply sets up the          cent of all spam, thus letting over 10
                                                            hostname and network envi-          ercent of all spam through (Figure 4).
test. To allow this to happen, a large          ronment. The license key, and filter direc-
number of users let us use their inboxes,       tion setup (incoming or outgoing), both          McAfee Secure Content
thus ensuring a good representative             use the web interface. The Symantec en-             Management
sample of genuine ham (non-spam)                gineer dug into his bag of tricks and con-          Appliance 3200
messages. Users either forwarded mes-           jured up a secret menu by pressing a key-       McAfee also shops with Dell. The McAfee
sages from their own mailboxes, includ-         board shortcut [Shift]+[A] in the Settings      3200 appliance supplied the Symantec
ing newsletters, or wrote their own text        menu (Figure 3).                                appliance’s little brother: a 2.8 GHz Xeon
emails, some in foreign languages such                                                          and 1GB RAM are all McAfee needs for
as Bulgarian and Turkish. Of course, this       Secret Menu                                     up to 1000 users. The hard disk subsys-
is torture for filters that attempt to per-     The engineer enabled the Rapid Release          tem is a SCSI Raid 1 array. McAfee also
form language analysis.                         setting for the antivirus filter; according     uses Red Hat Linux, however, the appli-
   At the end of the test phase, we put         to the engineer, this setting is standard       ance has a lot more software on board. A
the systems through stress testing. For a       for other customers. As we were not in-         Secure Web Gateway was pre-installed on
couple of days, we had the MX records           terested in the virus filtering functional-     the test device. (This product is available
of two domains, which are subject to a          ity, we left the setting as is. The secret      individually, and was not taken into con-
volume of spam a couple of orders               menu gave us access to more interesting         sideration in our lab.)
greater than in our test environment, re-       settings. For example, you can modify              The McAfee appliance again used the
direct their junk mail to one appliance,        the SMTP greeting, which gives admins           local console for initial configuration, but
thus forcing it to handle somewhere in          the ability to hide the appliance from          it also supports a web interface-based
the region of 35,000 emails per day.            simple probes launched by attackers.            approach that uses a cross-over cable to
                                                   The spam filter is based on Bright-          connect the client to the appliance.
 Symantec Mail                                 mail’s Antispam; Symantec acquired              HTTPS is simply used as the transport
     Security 8260
All vendors were asked to supply an off-                                          Appliance Security
the-shelf product suitable for a small to        All the appliances we tested included the       line of code increases the probability of a
medium-sized business. Despite this, Sy-         word security in their names, however,          bug, and thus increases the likelihood of
mantec sent us its top-notch device, the         some devices worried us in this respect         a security hole.
8260. This machine is designed for en-           by using open HTTP for client server            One of our wishes that has remained un-
terprises with upward of 1000 mail ac-           communications, or by providing a jum-          fulfilled thus far was for the configura-
                                                 ble of tempting-looking Javascripts as a        tion to support local console-based ad-
counts, and it is said to be able to handle
                                                 web interface. Didn’t there used to be a        ministration, such as the ability to dis-
up to 10,000 accounts.                           rule for secure systems that said some-         able the web server and other remote
  The system supports clustering for en-         thing about reducing the services to a          services. This feature would give the
vironments that require higher perfor-           minimum and keeping the software as             customer the ability to opt for less secu-
mance. The hardware comprised a Dell             simple as possible? Spam filters are very       rity risk with (ostensibly) less conve-
Poweredge Server with a 19’‘ rack case           complex bits of software. Adding a web          nience. The fact that some vendors don’t
                                                 interface to keep users happy with gad-         even let users customize the spam filter-
and a front plate that was customized by
                                                 gets and gimmicks is very likely to in-         ing rules shows how highly they value
Symantec. The flagship product sports            crease their vulnerability. Every single        their customers’ skills.
two Xeon CPUs with a clock speed of




34        ISSUE 76 MARCH 2007                      W W W. L I N U X - M A G A Z I N E . C O M
                                                                                                            Spam Test                                                 COVER STORY




protocol; client-side, you either need a        port simply states
Java client program, or you need to run a       that the C10 can            25,0
                                                                                     Spam as Ham
Java applet in your browser. The applet         protect up to 1000          22,5
                                                                                     Ham as Spam




                                                                                                                                                                                      22,2 %
takes slightly longer than rendering a nor-     mail users against          20,0

mal HTML page. All in all, the clear-cut        spam. This puts it           17,5




                                                                                                                                                             18,0 %




                                                                                                                                                                                                               17,5 %
Java interface was fairly sluggish and          in the same league          15,0




                                                                                                                                            15,6 %
seemed to respond more slowly to user           as the McAfee ap-            12,5




                                                                                   11,4 %
                                                                            10,0
interaction than any other user interface.      pliance.




                                                                                                                                    9,2 %


                                                                                                                                                     9,0 %
                                                                              7,5
Unfortunately, the client program did              Ironport offers a




                                                                                                    3,0 %


                                                                                                                    7,1 %




                                                                                                                                                                                                       7,1 %
                                                                                                            7,0 %
                                                                              5,0
very little to improve performance.             choice of two




                                                                                                                                                                      4,7 %
                                                                                            0,0 %




                                                                                                                            0,0 %




                                                                                                                                                                                               0,0 %
                                                                                                                                                                              0,4 %
                                                                              2,5
   To run the McAfee spam killer, you           spam filters,
                                                                              0,0
first need an activation CD. To create the      Brightmail, which             % Symantec McAfee IronPort Canit    IKU   eXpur-  Spam     GMX
                                                                                                                         gate Stops Here
CD, you first need to download the              is also Symantec’s
image file (this is just a couple of MB)        choice, and a pro-       Figure 4: Separate error rates are shown for false negatives (spam
from the McAfee company homepage,               prietary product.        incorrectly identified as legitimate) and false positives (legitimate
then burn the image onto a CD and in-           We used the sec-         mail identified as spam). The false negative value for Sponts is
sert the CD into the appliance’s drive to       ond variant in our       extrapolated (estimated error rate of between -3 and +5 percent).
read the CD. Instead of this convoluted,        lab. The filter had
and environmentally-unfriendly ap-              a convincing range of granular settings,            pre-configure. The box even includes a
proach, it might be easier and more un-         which admins can configure in a web-                user manual. Although the manual may
derstandable to customers to simply up-         based menu. The fact that the web inter-            be difficult reading for those without
load the ISO image directly using the           face uses HTTP rather than HTTPS                    some prior knowledge, users who prefer
web interface. And it might be even eas-        slightly tarnishes the good impression.             to set up individual spam protection at
ier to allow the appliance to contact the          Again, we used the default settings for          least have the controls to do so.
McAfee site directly.                           Ironport. In addition to the typical spam             Unfortunately, the user interface is un-
   Like almost all the vendors in our test,     filters, Ironport uses an image filter and          intuitive, but once you have mastered
McAfee uses a mix of several filter sys-        an Ironport-specific system known as                the quirky interface, CanIt gives a wide
tems. Besides SpamAssassin, they include        Senderbase. According to the vendor,                range of options that might let you im-
a Bayesian filter, blacklists and whitelists,   Senderbase logs much of the global mail             prove the poor detection rate by tweak-
and sender authentication. In contrast to       traffic, and is thus capable of quickly de-         ing the settings.
Symantec, McAfee supports very granular         tecting new waves of spam and malware
filter configuration. Although we kept the      outbreaks. In our lab, ham detection                Immature
default settings for our test, the system       proved reliable – no false positives. Iron-         At various places, the system creates an
achieved a laudable result of 97 percent.       port had the best spam detection rate of            impression of being immature. For exam-
Unfortunately, it did so at the expense of      all system with a perfect false positive            ple, setup mode, which requires the ad-
the false positive rate of no less than 7       rate of zeros (just 7 percent of all spam           ministrator to attach a display and a key-
percent. This said, the device drops spam       got through).                                       board, has an option to change the root
into a quarantine folder, which the recipi-                                                         password and the password for the setup
ent can access if needed.                        CanIt Anti-Spam                                   user. You can use the menu to change the
                                                The CanIt Anti-Spam appliance, which is             root password, but to change the pass-
 Ironport C10 Email                            developed by Roaring Penguin, is sup-               word for setup, you need to log in to the
    Security                                    plied in a 1 HE 19”, half-depth case.               console as root and enter passwd setup
Ironport’s case makes a nice change             Under the hood, the system has a 3 GHz              manually at the command line.
from the typical gray masses: the silver        Pentium 4 CPU, 1 GB RAM, and a 80 GB                  It is also annoying that the web inter-
19 inch, 1 HE unit looks elegant, and the       IDE hard disk.                                      face for the box only uses HTTP by de-
unit is custom-designed as an appliance.           The manufacturer describes the                   fault. Referring to HTTPS, the manual
The vendor even fits a blind cap to the         Debian 3.1-based system as the “leading             says “Setting this up is beyond the scope
VGA output to keep inquisitive adminis-         anti-spam solution.” We would question              of this manual, but CanIT-Pro should op-
trators’ fingers firmly away from the box       this based on the measured values: The              erate with no changes over HTTPS.” The
(Figure 5).                                     CanIt appliance had the second-highest              second part of the sentence turned out to
   The interior design is just as unique:       false positive rate, but it failed to com-          be true.
the operating system the appliance runs         pensate by catching more spam. In all,
on is called Async OS. This is a mail fil-      CanIt had a spam detection rate that                 IKU Sponts
ter-optimized Linux that works far more         puts it firmly in the bottom half of the            Linux Magazine tested the IKU Sponts
efficiently than normal Linux according         test field.                                         Appliance a couple of years ago. Accord-
to Ironport. Apart from this, the vendor           On the other hand, the system sur-               ing to the description on the website, the
states that the system has two 40 GB            prised us with some useful features:                case should have been either a Mini ITX
disks in a RAID 1 array. Instead of pub-        users are allowed to define their own fil-          or a 19’‘ rack system, but what we got
lishing the CPU speed, or memory, Iron-         ter rules, which the administrator can              was a Mini ATX system. Apart from this,



                                                   W W W. L I N U X - M A G A Z I N E . C O M                                                ISSUE 76 MARCH 2007                                                        35
COVER STORY                             Spam Test




the other components seemed to match                sages temporarily on the Sponts Box,                   no help, because spammers sometimes
the Sponts specifications. The big advan-           and to relay them to the internal mail                 open connections to test mail addresses.
tage this system offers is that it has no           server via the Replay is also useful. The              Figure 4 shows a detection rate that we
wear and tear parts, apart from the                 box also runs a POP3 server to keep                    extrapolated by reference to the compar-
40 GB IDE disk, and this should mean a              email service available in case of an in-              ative values from the other domains.
longer product life. IKU quotes a limit of          ternal mail server outage. This is a use-              The ham tests revealed that the Sponts
550,000 messages a day for the Debian-              ful feature that other appliances should               Box is far too aggressive by default. A
based system, which puts it more or less            think about introducing.                               false positive rate of 18 percent put the
on par with the 1000 users quoted by                  The Sponts Box uses two special ap-                  box firmly at the bottom of the pack.
McAfee and Ironport.                                proaches to spam protection: the Sponts
  We plugged in a keyboard and a dis-               Effect and SMTP transmission timing                     Expurgate
play for the initial setup. The system ex-          analysis. In addition to this, the system              Instead of buying an appliance, updating
pects you to enter the network settings             has a number of standard filter tech-                  it regularly, and managing the system
at the console. We removed the key-                 niques in place, all of which support in-              yourself, you can outsource the job to a
board for the next boot, and the system             dividual weighting and customization.                  service provider. The Expurgate service
hung. The BIOS was set to interrupt the               In our lab, the Sponts Box was config-               provides external spam filtering. Expur-
boot process if the keyboard was miss-              ured to respond with a User unkown on                  gate is operated by a German company
ing. Fortunately, we soon identified the            receiving spam. Because of this, the mail              called Eleven. Customers simply set the
source of the bug.                                  server downstream did not get to see any               MX record for their domains to the Ex-
                                                    spam mails, apart from the ones that got               purgate mail server. To use Expurgate,
Neat Interface                                      past the filter, of course. We were unable             you need to set up four MX records with
The system’s user interface is well de-             to ascertain the detection rate due to                 the same priority. Each of these host-
signed, and the internal help function re-          this. Although the Sponts Box’s logfiles               names resolves to multiple IP addresses
ally does help. The ability to store mes-           record connection attempts, this value is              in various subnets, thus implementing

                                                         Table 1: The Candidates
 Company          Symantec               McAfee                Ironport         Roaring Penguin       IKU             Eleven          Greenview Data
 Product          Mail Security          Secure Content        C10 Email        Canit Anti-Spam       Sponts-Box      Expurgate       Spam Stops Here
                  8260                   Management            Security         Appliance
                                         Appliance 3200        Appliance
 Price            EUR 6,310 plus         EUR 17,680            EUR 2,000 for    EUR 2,690 for         Starts at     10 users cost     EUR 14 per mail
                  licensing charge.      perpetual license     50 users         300 users, EUR 4      EUR 664       EUR 25 per        box and year
                  Rates for virus        for any number of     and one year     for each additional                 month, EUR 42     (for 10 to 19 mail
                  protection and         protected systems,                     user                                with virus pro-   boxes). 10,000
                  antispam: EUR 35       no time limits.                                                            tection. For      mailboxes or
                  per license for 100    Includes one year’s                                                        1,000 users,      more, EUR 6 each
                  licenses or EUR 23     support.                                                                   EUR 970           plus domain.
                  per license for 500                                                                               without, and      Maximum
                                                                                                                    EUR 1,700 with     transfer
                                                                                                                    virus protec-     per mailbox and
                                                                                                                    tion. Discount    month.
                                                                                                                    for 3 year ad-
                                                                                                                    vance payment.
 Type             Appliance              Appliance             Appliance        Appliance             Appliance     Service           Service
 Accounts         More than 1,000        Up to 210,000         Up to 1,000      Not disclosed         550,000 Mails –                 –
                                         messages per hour                                            per day
 POP and/         No                     No                    No               No                    Yes           No                No
 or IMAP
 SMTP Relay       Yes                    Yes                   Yes           Yes                      Yes           Yes               Yes
 Administration   Web interface          Web interface         Web interface Web interface            Web interface Web page          Web page
 via                                     and Java applet
 Licensing        Account driven         No limits             Time and       Account driven          By appliance    Time and        Account and
 model                                                         account driven                                          account        domin driven
                                                                                                                      driven
 Filters          LDAP users only        No                    No               For every user        No              Exceptions      Yes
 configurable                                                                                                         for single
 per user                                                                                                             account
 Hardware         2 x 3 GHz Xeon,        2.8 GHz Xeon,         40 GB            3 GHz Pentium 4,      40 GB IDE,      –               –
                  2 GB RAM,              1 GB RAM,             Raid 1, not      1 GB RAM,             not disclosed
                  73 GB SCSI             SCSI Raid 1           disclosed        80 GB IDE             otherwise
                  Raid 1, redundant                            otherwise
                  power supply
 OS               RHE 3.0                RHE 3.0               Async OS         Debian 3.1            Debian          –               –




36          ISSUE 76 MARCH 2007                          W W W. L I N U X - M A G A Z I N E . C O M
                                                                                                  Spam Test           COVER STORY




                                                                             users have been            There is still a latent danger of com-
                                                                             known to return         parative filters classifying mailing lists
                                                                             phishing mails          with large numbers of targets as spam.
                                                                             tagged as spam          Whitelisting legitimate mailing lists can
                                                                             with notes to the       help to solve this problem, although the
                                                                             effect that the         sheer bulk of mailing lists makes this dif-
Figure 5: Ironport fits a cap to the VGA port; attaching a display is        mails were impor-       ficult to do. To compensate, vendors use
not an option.                                                               tant messages           additional filtering techniques to further
                                                                             from their banks.       reduce the risk of false positives.
redundancy via DNS round robin load                   Unfortunately, administrators have                Where vendors combine aggressive fil-
balancing. Redundancy is an important              more or less no options for configuring           ters with automatic error message gener-
advantage of a service provider in com-            the Expurgate filters. Administrators can         ation in the SMTP dialog, and this is
parison to an appliance-based solution.            except email addresses from filtering and         what Sponts does, a fatal combination
   The Eleven spam filters benefit from            define actions to execute when a mes-             can occur. The idea that the sender will
the fact that they protect multiple mail           sage is classified as belonging to a spe-         open, read, and understand Mailer-Dae-
servers, which means they come in con-             cific spam category. More granularity             mon messages is one that would only
tact with many spam messages. The ser-             would be preferable here. Despite this,           occur to an engineer: users are more
vice calculates checksums for all incom-           Expurgate still achieved the second high-         likely to complain of being spammed by
ing messages and uses the checksums to             est spam detection in our lab, at the ex-         Mailer-Daemon. If you actually read the
compare the messages. If a checksum                pense of 0.4 percent false positives.             message, you discover cryptic wording
occurs more frequently than the mean                                                                 that is more or less impossible to under-
value, the message can be assumed to be             Spam Stops Here                                 stand. Spam filters should tag, but not
spam. This approach is similar to Iron-            Canada’s Greenview Data follows an ap-            remove – they are just not accurate
port’s Senderbase approach, but it does            proach similar to Expurgate with their            enough to make final choices.
away with update cycles.                           “Spam Stops Here” product, except that
                                                   the configuration options are more gran-          Three Winners
Endangered Species                                 ular. For example, administrators can en-         If you take the false positive rate as a im-
Expurgate uses various criteria to pre-            able and disable filter modules or stop           portant criterion, and you expect a de-
vent the filter from falsely identifying le-       filtering for specific mail boxes. Surpris-       tection rate of more than 80 percent, the
gitimate newsletters as spam. One crite-           ingly, the spam detection rate for the de-        contenders left standing are Expurgate,
rion is the delivery path: spam typically          fault settings is very poor: 77.8 percent         Symantec, and Ironport. Due to its ASP
originates from multiple machines on a             puts Spam Stops Here right down at the            approach (Application Service Provider),
bot network, whereas newsletters origi-            bottom of the list. The product did not           Expurgate is hard to compare with the
nate from a single source. In addition to          produce any false positives.                      two appliances, and the configuration
this, the company uses what they refer                                                               options were more than spartan.
to as spamtrap addresses. These are                Compensation Business                                With respect to filter quality and us-
email addresses only published on web-             The results in Figure 4 are not surpris-          ability, the Ironport C10 wins against the
sites – much like the addresses used for           ing: the higher the spam detection rate,          Symantec solution. We found no big dif-
this test. When the spamtraps receive a            the higher the false positive tends to be.        ferences in filter quality (Symantec failed
message, it is very likely to be spam.             A better detection rate (less spam getting        to detect 11 percent, and Ironport 7 per-
   An additional feedback channel gives            through) means using more aggressive              cent of all spam), and Symantec could
the ability to tag mail individually as            filters, which may sacrifice ham mails.           easily overtake Ironport following the
spam or ham. To avoid misuse and er-                  It is interesting to see how filters that      next update. One positive aspect for
rors, Eleven states that a staff member            apply comparative techniques (such as             Linux fans is that all of the filters we
checks messages for legitimacy because             Senderbase and Expurgate) can improve             tested are based on Linux systems. ■
                                                   detection rates without affecting false
                 Support                           positive performance. Although both                                 INFO
  The complex test setup was only possi-           values are approximations due to the
                                                                                                      [1] Spam-o-Meter:
  ble thanks to the Bundeswehr University          way the two systems work, they do                      http://www.spam-o-meter.com
  in Munich, Germany. The Institute for            allow us to evaluate the filter quality.           [2] Spam archive: http://spamlinks.net/
  Information Technology Systems pro-              The improved results may also have                     filter-archives.htm
  vided the mail server, the network con-          something to do with a new breed of                [3] SpamAssassin:
  nection, and the rackspace for the appli-
                                                   spam that started to emerge during the                 http://spamassassin.apache.org
  ances, in addition to registering the test
  domains. Lieutentant Carsten Schulz set
                                                   test period: image spam hides the mes-             [4] RFC 0974, “Mail Routing
  up the test systems, and performed the           sage in arbitrarily mangled images. The                and the Domain System”:
                                                   filters had to become accustomed to the                http://www.ietf.org/rfc/rfc0974.txt
  measurements, as part of his thesis.
  Thanks also to Daniel Rehbein, who pro-          new spam, and rule-based approaches                [5] “Bot Posse: An insidious botnet
  vided access to two heavily spammed              tend to be at a disadvantage in compari-               attacks Charly” by Charly Kühnast,
  domains for the test series.                                                                            Linux Magazine, August 2006, pg. 68
                                                   son to comparative techniques at first.



                                                     W W W. L I N U X - M A G A Z I N E . C O M              ISSUE 76 MARCH 2007                37

								
To top