Curbing Spam Poses New Challenges

An Internet Café, sponsored by the 13th Corps Support Command in Balad, Iraq, has helped deployed warfighters keep in touch with family through e-mail since 2004. To keep this morale booster viable, however, the U.S. Army may have to make adjustments so that legitimate e-mail is not viewed as spam.

The Sender Policy Framework is an emerging Internet standard that could cause a large part of the U.S. Army’s legitimate e-mail to be categorized as spam and dropped. Large e-mail providers in the commercial world are in the initial phases of implementing the framework, and while deleterious effects on Army e-mail have been rare so far, they are almost guaranteed to grow as more providers and intermediaries adopt the standard. However, several courses of action can address the issue, and Army Knowledge Online already has taken steps to implement the framework while simultaneously protecting the future viability of the service’s e-mail system.

Symantec Corporation, Cupertino, California, estimates that as much as 60 percent of all e-mail is spam, which it defines as unsolicited bulk e-mail. At its most innocuous, spam is a major drain on resources in employee time and network assets and increases frustration. At its worst, spam represents a significant security threat in the form of phishing attempts and a vector for viruses and other malicious software.

Army Knowledge Online (AKO) spends more than $1.3 million a year to filter spam from Army e-mail, and while it is by far the largest Army provider, AKO is only one of thousands of Army e-mail providers worldwide. In fact, the exact number of Army e-mail providers is unknown because each Army installation supports its own infrastructure, and separate units on installations often will have local e-mail support as well.

Techniques to help decrease spam volume such as the Sender Policy Framework (SPF) will reduce expenditures and raise security. The SPF allows e-mail providers to list the specific e-mail servers that are authorized to send e-mail from their domain. The framework enables e-mail recipients to decide whether they will accept certain e-mail.

Although the SPF is a well-intentioned protocol that the commercial sector is implementing, one unintended consequence is that a large part of the Army’s legitimate       e-mail will be classified as spam and dropped as it flows through the Internet to its destination. This will occur because, by protocol, e-mail must contain certain fields, one of which is the “envelope-from” field. This identifies the purported sender’s domain and is the domain name that is exploited by the SPF.

Under the SPF protocol, mail providers such as AKO publish the Internet protocol (IP) addresses of all e-mail servers that are authorized to send e-mail from their domain. This is done via the domain name system (DNS) within the SPF record that acts as a reverse mail exchange, or MX, for the domain—a technique for determining the ownership of the DNS entry. The SPF record is stored as a text or an SPF attribute in the DNS. The receiving server executes a DNS lookup for the SPF record of the domain listed in the envelope-from field, and that information determines whether a message is spam.

In general, an e-mail provider publishes the IP addresses of all machines that are authorized to send mail from its domain by adding an SPF record to its DNS record. Mail recipients accept e-mail and before processing it note the IP address of the machine it came from and execute an SPF lookup of the domain in the envelope-from field. If the IP address gained from the packet headers matches an authorized e-mail-sending host, the mail is processed; otherwise, the message is marked for additional processing. In many cases, the message is quarantined or dropped as spam.

This process is easy to execute and adds very little load on an e-mail providers’ system, since the DNS lookups can be cached. That said, it is very powerful and, just as important, inexpensive to implement. Several SPF tools and plug-ins as well as open-source tools are available in the commercial market. They require no maintenance or pattern updates and can be managed by entry-level administrators.

The SPF is recognized as a protocol-in-draft by the Internet Engineering Task Force. Its adoption is purely voluntary, and by convention, SPF rules are applied to      e-mail only when the domain reflected in the envelope- from field has published an SPF record.

This is beginning to change, however, as more organizations are electing to implement the SPF unilaterally, regardless of whether the originating sender is participating in the SPF. Several large e-mail providers such as AOL, Yahoo! and Hotmail have published policies stating their intentions to handle e-mail originating from non-SPF organizations differently from those originating from an organization that supports the SPF. Many Internet service providers (ISPs) have indicated that by the end of 2006 they will not accept e-mail from domains that do not have published SPF records, and a few ISP e-mail administrators already have implemented these policies.

For most organizations, the SPF will not affect e-mail operations noticeably. They simply have to publish an SPF record and maintain it for the system to work. However, because of some of its policies and the way it implements some programs, the Army is radically different. The SPF will affect Army e-mail significantly, and if the Army takes no action, a large portion of Army e-mail will be dropped as spam by commercial providers.

By policy, the return address on each Army e-mail must be the us.army.mil or an AKO address. Although AKO has become the de facto e-mail hub for the Army, most of the Army’s e-mail originates at the local level. Much of the locally originating e-mail stays within the local domain. Mail originating in an Active Directory forest is routed by Microsoft Exchange and not by common IPs. E-mail sent from one user to another within hqda.army.mil never leaves the hqda.army.mil domain. The remainder is routed through the open network. In practice, this means that even though e-mail originates from a non-AKO domain and never passes through servers at AKO, the return address routes the mail back to AKO.

In the absence of intervention, an organization that is implementing the SPF and receives this e-mail will accept the e-mail, note the IP address of the sending machine and examine the envelope-from field. It will execute an SPF lookup on the domain contained in this field and compare the IP addresses found in the SPF record to the IP address of the sending machine. The organization then will note that the IP address in the field does not match any of those in the SPF record and will mistake the message as spam.

As a result, a significant portion of the Army’s e-mail that is external to the local domain and that leaves the nonsecure IP router network (NIPRNET) for an Internet-based destination could be dropped as spam as a direct and predictable result of existing Army policy and processes. This could amount to as much as 10 percent of all Army e-mail routed to the Internet or as much as 100,000 messages a day. Most of those message senders will not know that their messages were dropped.

Four reasonable courses of action can address this issue. First, the Army could do nothing; in the very near term, this course will have little detrimental effect. AKO has noted nine cases over the past year in which the SPF has resulted in valid Army e-mail bounced as spam. As more time passes and more organizations adopt the SPF, however, more Army e-mail will be dropped. Without a uniform way to track and address these incidents, the Army will have no way to know how much e-mail is lost and no way to solve the problem once it worsens. As a result, while inaction is a reasonable course in the very near term, it becomes nonviable quickly thereafter.

A second approach is to place an “all others” entry in AKO’s SPF record. One SPF protocol feature is the ability for e-mail originators to publish an SPF record that contains a final entry known as “all.” When preceded by a plus sign, this indicates that any IP address is allowed to send e-mail on behalf of that domain. A question mark indicates that the domain is unsure which IP addresses sent e-mail on behalf of the domain. A tilde indicates that addresses other than those listed can send e-mail for the domain, but it is discouraged. The dash or hyphen indicates that other IP addresses are prohibited from sending e-mail for the domain.

Adding +all to the SPF record would allow other Army domains to send e-mail through AKO and would be an easy fix, but it would have the highly undesirable effect of defeating the benefits of the SPF for the entire Army       e-mail domain. Spammers are a highly adaptable group; the Army’s status as wide open for spam would be exploited quickly.

The third course of action would be to gather the IP addresses for all of the Army’s authorized e-mail-originating hosts and to publish them in AKO’s SPF record. If all of the Army’s originating e-mail hosts were known, they could be published in AKO’s SPF record and the SPF would work exactly as advertised.

This approach includes both significant drawbacks and advantages. First, it would be difficult and time-consuming to gather and maintain this information. The process would never be complete because e-mail hosts are constantly in flux. One benefit, however, would be that it would force the creation of a definitive list of Army e-mail hosts—or at least those that will ever send e-mail outside the NIPRNET.

The crippling drawback to this alternative is the sheer size of the resulting SPF record, which would have to be requested and scanned for each and every e-mail, slowing down e-mail processing and the entire DNS. The SPF request for comment dictates that SPF entries should be less than 512 bytes, and the Army record would be far larger. Even if the Army decided to disregard this recommendation, DNS servers typically impose a practical size limit on the DNS record they will accept. For computers with Microsoft software, that size is typically 16 kilobytes, so this solution is probably not viable.

The fourth alternative would be to relay all Army e-mail that leaves the local domain through AKO. This is the most revolutionary alternative in terms of a change in the way the Army handles e-mail today, but it also is the only solution guaranteed to work in the long term.

Processing all out-of-domain Army e-mail through AKO means that the return domain of us.army.mil will match the list of authorized e-mail hosts in AKO’s SPF record. As a result, trickery or short-circuiting of the SPF system will not be required to keep Army e-mail flowing. At the same time, the SPF protocol and process are respected, and the Army accrues the anti-spam benefits of this process.

More importantly, this alternative would enable the Army to apply several centralized control and security enhancements to e-mail that are otherwise impossible. Specifically, if all e-mail is cycled through AKO, the Army can impose a single naming convention, making enterprise identity management possible. In addition, the Army could impose a uniform anti-spam and antivirus process on all e-mail, increasing security and usability overall. Finally, it could impose a unified set of security policies without having to convince each director of information management worldwide to toe the line.

This alternative shares one of the advantages of the third approach: It builds a definitive list of all the Army’s e-mail hosts without any of the disadvantages of the other approaches. In particular, AKO would require a list of all sending hosts so that properly authenticated relaying could be allowed. That list would be used to validate hosts connecting to AKO that do not have an SPF record.

Information on all relaying e-mail hosts could be gathered with a Web-based tool so e-mail administrators could enter their information and automatically push it onto a list for AKO’s use. The data provided would be added to  AKO e-mail gateways to enable transmission as well as to provide AKO with contact information so that the appropriate administrator could be contacted if problems were detected.

Using AKO to relay e-mail could be made strictly voluntary. Non-AKO administrators could choose not to take part but would have to face the problem of being unable to send to many—and over time an increasing number of—Internet domains.

To implement this last approach, AKO has published an SPF record that contains the IP addresses that AKO’s servers use and the “~all” entry that indicates that other senders are allowed but discouraged. This should not adversely affect delivery from Microsoft Exchange servers, but it does not provide the benefits of the SPF to stop spam and phishing attacks. In some cases, publishing the SPF record may actually facilitate delivery. This solution is similar to the second course of action and has all of its drawbacks.

As a result, the fourth course of action appears to be the only true choice. This approach would require all of the Army’s out-of-domain e-mail to cycle through AKO before it leaves the NIPRNET. It is technically feasible and can be implemented with existing technology.

The question is whether AKO can handle the traffic. Because all e-mail addressed to us.army.mil from the Internet already passes through AKO, incoming e-mail will not add to the load. Daily, approximately 400,000 messages of the Army’s outgoing e-mail come from AKO as well and represent no additional load. The only additional load would stem from outgoing e-mail from non-AKO sources, which the Army estimates at perhaps 600,000 messages daily. The total load on AKO’s system would jump from approximately 8 million messages daily to approximately 8.6 million daily—a 7.5 percent increase; however, AKO’s basic infrastructure could handle this additional load today and would be well positioned for expansion in the future. This solution also would allow the Army to implement other emerging standards such as domain keys identified mail/DomainKeys in a simple and uniform fashion.

Relaying all Army e-mail that leaves the local domain through AKO also would increase outbound Army e-mail security by subjecting it to rigorous, unified, common anti-spam and antivirus solutions. Additionally, it makes it possible to finally achieve what has been so far unachievable—complete knowledge of all Army e-mail providers in one place. The positive implications of this knowledge for information assurance and network security are impossible to dismiss.

At Army Knowledge Online, Col. Taylor Chasteen, USA, is the program director; Maj. Cheryl Hynes, USA, is the chief of data center operations; and Lt. Col. Ken Blakely, USA, is the chief of operations.

Web Resources
Army Knowledge Online: https://www.us.army.mil/suite/login/welcome.html
Sender Policy Framework: www.openspf.org
Internet Engineering Task Force: www.ietf.org