Lessons Learned while Writing a Java Mailserver by Semaj1212


									Lessons Learned while Writing a Java Mailserver
by Richard O. Hammer

about me
• 70s – BSEE, coded in Fortran and APL. • 80s – went to CS graduate school at UNC-CH, ABD. Started a building business. • 90s – started a libertarian think tank. • Y2K – returned to programming to make a living, focus on Java and Internet programming.

The Familiar Catch-22
• You can’t get a job without experience. • You can’t get experience without a job. • So I needed a project to work on at home.

Why this Project?
• Spam is an interesting problem to me. It is a “tragedy of the commons”, an example of lawless behavior in a public space. • An email service could eliminate spam in a novel way – by charging unlisted senders. • Along the way I would learn about programming at the level of sockets and network protocols – where the problem of Internet lawlessness seems to start.

Minimal Specification: to offer an email service which eliminates incoming spam
• Customers can manage a list of “accepted” senders • All messages from other “unlisted” senders are waylaid. Automatic email is sent back to these senders, explaining the situation, giving them the option to pay for delivery • Works with usual email client, such as Mozilla Thunderbird or Outlook Express.

Wish-List Specification: (postponed)
• Transform payment into bond, with easy way for customer to order refund. • Add other ways for unlisted senders to authenticate themselves. • Make the service scalable, to handle high volume of traffic. • Install defenses against denial of service attacks. • And many more features.

This Presentation Is Not
• Entirely up to date (by a few years) • Comprehensive

Yet It Is
• Basic material that anyone who programs email will need to know

Protocols Specification
• I want people to be able to have email accounts with me. What do I have to offer? • Protocols required? • Services required?


Email Client such as Thunderbird or Outlook Express simplified SMTP

How an Email message is Transmitted
Recipient’s ISP

Mailserver or Mail Transfer Agent
Sender’s ISP


Mailserver or Mail Transfer Agent
POP3 Email Client such as Thunderbird or Outlook Express


A look at SMTP
• RFC (821, 1982), superseded by 2821, 2001 • client opens a socket to port 25 on the SMTP server • SMTP server identifies itself

SMTP in Action telnet from DOS
C:\>telnet XDomain.com 25
220 gork.XDomain.com ESMTP Postfix helo mailscreen.net 250 gork.XDomain.com mail from:<sue@yrox.net> 250 Ok rcpt to:<fred@XDomain.com> 250 Ok data 354 End data with <CR><LF>.<CR><LF> Here is a message to you. . 250 Ok: queued as 63235619F2ED quit 221 Bye

Connection to host lost.

Content of SMTP data RFC (822) 2822
data 354 End data with <CR><LF>.<CR><LF> Received: from BILL ([]) by vm024.mailsvcs.com for rHammer@mailscreen.net; Wed, 03 Jan 2007 07:07:50 -0600 (CST) Date: Wed, 03 Jan 2007 08:07:49 -0500 From: "Bill Hill" <bHill@Verizon.Net> Subject: RE: UFO sighting To: "Richard Hammer" <rHammer@mailscreen.net> MIME-version: 1.0 MIME HEADERS Content-type: text/plain; charset=windows-1250 Content-transfer-encoding: 7bit RFC 2045 Hi Rich,


RFC 2822 headers end with a blank line

Interesting! Maybe E. Jackson should take a look at it Bill . 250 Ok: queued as 63235619F2ED

SMTP (RFC 2821) envelope vs RFC 2822 headers

mail from:<sue@yrox.net> Addresses in SMTP envelope 250 Ok rcpt to:<fred@XDomain.com> may differ from 250 Ok Addresses in headers data 354 End data with <CR><LF>.<CR><LF> From: "Bill Hill" <bHill@Verizon.Net> To: "Richard Hammer" <rHammer@mailscreen.net> My message to you. . 250 Ok quit 221 Bye

Socket Programming
• java.net.Socket • java.net.ServerSocket • On top of TCP

public class GetterServer implements Runnable{ boolean gracefulShutdownOrdered; ServerSocket receiverServerSocket; Socket receiverConnectionSocket;

public GetterServer() throws Exception{ receiverServerSocket = new ServerSocket(25); }

Top Level Code for a SMTP server

public void run(){ while(!gracefulShutdownOrdered){ try{ try{ receiverConnectionSocket = receiverServerSocket.accept(); }catch (IOException e){ if(gracefulShutdownOrdered) break; else{ logStackTrace("Exiting GetterServer.", e); break; } } Getter myMan = new Getter(); myMan.handleConnection(receiverConnectionSocket); }catch (RuntimeException runt){ logStackTrace(runt); } } } ...


protocol programming
• We will look at three methods from class Getter, adapted from: org.apache.james.smtpserver.SMTPHandler

class Getter{ //adapted from void handleConnection(Socket socket) { remoteHost = socket.getInetAddress().getHostName(); remoteIP = socket.getInetAddress().getHostAddress();

method handleConnection(Socket)

in = new BufferedInputStream(socket.getInputStream(), 7024); smtpCommandLineReader = new CommandLineReader(in); out = new InternetPrintWriter(socket.getOutputStream(), "Getter says: ", true); out.println("220 " + helloName + " ready "); String clientCommand = null; try { boolean getAnotherCommand; do { clientCommand = smtpCommandLineReader.readLine(); getAnotherCommand = parseCommand(clientCommand); } while (getAnotherCommand); } catch (Exception e) { if (shutdownOrdered) { //log normal closing return; } else if (e instanceof SocketException) { //log unusual condition } else {// might be major //log error } } closeConnections(); }

private boolean parseCommand(String command) throws Exception { if (command == null) return false; String argument = null; String argument1 = null; command = command.trim(); if (command.indexOf(" ") > 0) { argument = command.substring(command.indexOf(" ") + 1); command = command.substring(0, command.indexOf(" ")); if (argument.indexOf(":") > 0) { argument1 = argument.substring(argument.indexOf(":") + 1); argument = argument.substring(0, argument.indexOf(":")); } } if (command.equalsIgnoreCase("HELO")) doHELO(command, argument, argument1); else if (command.equalsIgnoreCase("EHLO")) doEHLO(command, argument, argument1); else if (command.equalsIgnoreCase("MAIL")) doMAIL(command, argument, argument1); else if (command.equalsIgnoreCase("RCPT")) doRCPT(command, argument, argument1); else if (command.equalsIgnoreCase("AUTH")) doAUTH(argument); else if (command.equalsIgnoreCase("DATA")) { doDATA(); } else if (command.equalsIgnoreCase("QUIT")) doQUIT(command, argument, argument1); else doUnknownCmd(command, argument, argument1); return !(command.equalsIgnoreCase("QUIT") == true || shutdownOrdered); }

method parseCommand(String)

method doMAIL(String, String, String)

private void doMAIL(String command, String argument, String argument1) { if (!gotEhlo) { out.println("503 bad sequence of commands, send ehlo or helo first"); return; } if (argument == null || !argument.equalsIgnoreCase("FROM") || argument1 == null) { out.println("501 Usage: MAIL FROM:<sender>"); return; } String sender = argument1.trim(); if (sender.length() < 2 || sender.charAt(0) != '<' || sender.charAt(sender.length() - 1) != '>') { out.println("501 Usage: MAIL FROM:<sender>"); return; }

//method doMAIL concluded MailAddress senderAddress = null; sender = sender.substring(1, sender.length() - 1); if (sender.length() > 0) {// the usual case try { senderAddress = new MailAddress(sender); } catch (ParseException e) { out.println("501 Failure to parse InternetAddress from " + "\"" + argument1.trim() + "\". " + e.getMessage()); return; }
// screen senders with mailscreen addresses if (Services.isToLocalDomain(senderAddress)) { if (authenticatedUser == null) { out.println("530 Authentication Required"); return; } else if (!CustomerSet.isMsAddress(senderAddress)) { out.println("550 No such address"); return; } } } else { // sender.length() == 0, must have been <> // let senderAddress remain null, as that is our signal of mail from <> } setEnvelopeFrom(senderAddress); out.println("250 Sender " + sender + " OK"); }

Mailscreen.net as it now runs
• A free offering of the service has been available since March 2004

Unlisted Sender T 0.0

T 1.0 Unlisted Sender Pays HTTPS


HTTPS URL TO: JOE@Mailscreen.net FROM: Unlisted Sender T 1.1 T 1.2 T 1.3

Payment Required Message
TO: Unlisted Sender FROM: customer-agent@mailscreen.net

T 0.2


Instant Payment Notificaton
HTTP POST Echo Parameteres


T 2.0 TO: JOE@Mailscreen.net FROM: Unlisted Sender

T 0.1

Payment Required Message
Dear Somebody@somewhere.org:
Mailscreen.net has received your email for a Mailscreen customer, service@mailscreen.net. Unfortunately, since your email address is not registered as an accepted sender to this customer, Mailscreen will not deliver your message to the customer until you make a payment as described in the next paragraph. The charge to send this message will be $0.50. To make this payment click on the link below. It will take you to PayPal.com where you can make payment, and where you can open a new PayPal account if you do not already have one. Mailscreen does not accept any other form of payment at present.

https://www.paypal.com/xclick/business=cashier@mailscreen.net&item_name=accept+email&invoice=06 0410.204815c5i0&amount=0.50&return=http%3A//mailscreen.net/thanks.jsp Please make your payment promptly. We cannot promise fulfillment if your payment comes after four calendar days, because we must limit the number of messages that we hold at Mailscreen.
If you would like more specific information to identify the message which we received from you and which we are holding awaiting payment, here are some identifying headers from that message.

Date: 10 April 2006 From: Somebody@somewhere.org Received: FROM nc-60-41-56-62.dyn.yahoo.net ([]) BY mailscreen.net; Mon, 10 Apr 2006 20:48:15 -0400 (EDT)

Sincerely, Mailscreen.net

<% Enumeration en = request.getParameterNames(); StringBuffer replyBuf = new StringBuffer("cmd=_notify-validate"); while(en.hasMoreElements()){ String paramName = (String)en.nextElement(); String paramValue = request.getParameter(paramName); replyBuf.append("&").append(paramName).append("=") .append(URLEncoder.encode(paramValue,"UTF-8")); } // post back to PayPal system to validate URL url = new URL("http://www.paypal.com/cgi-bin/webscr"); URLConnection uc = url.openConnection(); uc.setDoOutput(true); uc.setRequestProperty("Content-Type","application/x-www-formurlencoded"); PrintWriter pw = new PrintWriter(uc.getOutputStream()); pw.println(replyBuf.toString()); pw.close();

Instant Payment Notification, in the JSP that PayPal calls

BufferedReader in = new BufferedReader( new InputStreamReader(uc.getInputStream())); String res = in.readLine(); in.close();
if (!"VERIFIED".equals(res)){ log("Payment failed. res not VERIFIED but "+ res); return; }

JSP Control Consoles
• Administrator • Customer

PROBLEM How can calls from Web App reach the objects in the mailserver?


Run Web App and Mailserver in the same JVM

Starting the Mailserver
It Is a

public class StartServlet extends HttpServlet {
public void init(){ ServerManager.start(); } public void destroy(){ ServerManager.myOneServerManager.startGracefulShutdown(); }


mailscreen web.xml file
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE web-app PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN" "http://java.sun.com/dtd/web-app_2_3.dtd"> <web-app> <servlet> <servlet-name>do it damn it</servlet-name> <servlet-class>common.StartServlet</servlet-class> <load-on-startup>3</load-on-startup> </servlet> <!-- security for regular customers --> <security-constraint> <web-resource-collection> <web-resource-name>customerResource</web-resource-name> <url-pattern>/customer/*</url-pattern> </web-resource-collection> <auth-constraint> <role-name>customer</role-name> </auth-constraint> <user-data-constraint> <transport-guarantee>CONFIDENTIAL</transport-guarantee> </user-data-constraint> </security-constraint> ...

Tomcat server.xml file
<Server port="8005" shutdown="SHUTDOWN"> <Service name="Catalina"> <Connector port="80" maxThreads="15" minSpareThreads="1" maxSpareThreads="2" enableLookups="false" redirectPort="443" acceptCount="6" connectionTimeout="20000" disableUploadTimeout="true" /> <Connector port="443" maxThreads="10" minSpareThreads="1" maxSpareThreads="2" enableLookups="false" disableUploadTimeout="true" acceptCount="5" scheme="https" secure="true" clientAuth="false" sslProtocol="TLS" /> <Engine name="Catalina" defaultHost="localhost"> <Host name="mailscreen.net" appBase="/usr/website/WebRoot" unpackWARs="true" autoDeploy="true"> <Alias>www.mailscreen.net</Alias> <Valve className="org.apache.catalina.valves.AccessLogValve" directory="msLogs" prefix="access." suffix=".log" pattern="common" resolveHosts="false"/> <Realm className="org.apache.catalina.realm.JDBCRealm" debug="99" driverName="org.postgresql.Driver" connectionURL="jdbc:postgresql://localhost:3246/pqlms" connectionName="mailData" connectionPassword="e3$fe4uii0p" userTable="uLogin" userNameCol="uName" userCredCol="cred" userRoleTable="uRole" roleNameCol="power" /> <Context path="" docBase="." /> </Host> </Engine> </Service> </Server>

Complexity of SMTP sending, Classification draft
M4 - a message as received in SMTP, with potentially: • many domains among the recipients • many delivery attempts (try or retry) necessary for each domain • many servers (IP addresses) to attempt during each delivery attempt • many recipient addresses to attempt when a connection to a server succeeds M3 - part of an M4, with recipients at only one domain, but with potentially: • many delivery attempts (try or retry) necessary for each domain • many servers (IP addresses) to attempt during each delivery attempt • many recipient addresses to attempt when a connection to a server succeeds M2 - one of attempts (try or retry) to deliver a M3, but with potentially: • many servers (IP addresses) to attempt during each delivery attempt • many recipient addresses to attempt when a connection to a server succeeds M1 - part of an M2, the attempt to deliver an M3 to one server (IP address), but with potentially: • many recipient addresses to attempt in this connection M0 - part of an M1, the attempt to deliver an M3 to one user address at one server (IP address) during one try or retry attempt


Key Classes

DistinctDataMessage DDM
String: ddmId, status boolean: disposable



List: envelopeTo MailAddress: envelopeFrom String: id File: dataFile

Remote Recipient RR
MailAddress: address String: status



MessageToOneRemoteDomain MORD
List: recipients String: domainName


MessageToMsRecipient MSR
void deliverTermsMet() void stopWaiting()


ForwardedMessage toCustomer

void writeTo(OutputStream) features



Customer: customer MailHeaders: headers

Delivery Status Notification
This notification pertains to a message you sent. that message by these headers taken from it:
To: susie@foggybottom.net Subject: coming through Date: Sun, 28 Mar 2004 20:22:21 -0500 This message could not be delivered to one or more recipients. A permanent failure was encountered while attempting to deliver to: susie@foggybottom.net No further action will be taken here on your message.

You may identify

Lessons about Spam
• Spam is not bad enough to drive people to the tactless solution offered by Mailscreen. • People do not want to risk the possibility that a cherished contact might receive a “payment required” message. • The problem is not just technical, because there are many technical solutions. • The problem is sociological.

• RFC Index http://www.rfc-editor.org/rfcindex.html • Delivery Status Notifications RFC 1891, 1894 • Java Network Programming, 2nd ed., Elliotte Rusty Harold, 2000 • Programming Internet Email, David Wood, 1999

Libraries Used
• org.xbill.DNS • javax.mail

Source Available to You at
• http://mailscreen.net/jug.zip • for one week, until Jan. 22, 2007

Thanks To
• • • • • Sun Microsystems, for Java, the Java API Apache James project Ralph Cook IETF email list, ietf-smtp@imc.org Triangle Java Users Group

To top