The software engineers helped create the spam problem. Can they solve it?
As politicians have been racing to find ways to ban the junk e-mail known as spam, Internet providers have been boasting about filtering technologies capable of identifying the sort of messages typically sent by spammers and disposing of them.
But the spammers have been keeping ahead of the law and the filters. The open nature of e-mail technology - designed decades ago by computer scientists who had little reason to anticipate spam - lets spammers hide their tracks and transform many of their messages to avoid detection.
As a result, many e-mail software experts now contend that the most powerful way to clean people's mailboxes is to focus not on catching the spam, but on identifying the legitimate mail.
"People have been spending all their time creating filters to find the bad guys," said Nico Popp, vice president for research and advanced products of VeriSign, the largest registrar of Internet sites and a seller of online identification systems. "We want to turn that on its head and find ways to identify the good guys and let them in."
Put simply, these efforts are trying to develop the Internet equivalent of caller ID, a technology that will let the receiver of an e-mail message verify the identity of the sender. As with caller ID for telephones, senders will be able to choose whether to remain anonymous. But also like caller ID, recipients may presume that those who do not identify themselves are sending junk.
The loudest calls for such a system are coming from the banks, travel companies and online stores that are finding that much of their e-mail is getting caught in spam filters. The advertisers gave the big Internet providers an earful recently at a forum sponsored by Doubleclick, an advertising technology company. The Internet providers responded that they were working on standards for a new system, and that their much-delayed proposal would be published this fall.
The technical challenges to creating such an identification system are daunting. The millions of computers that process e-mail for half a billion users may need to be retrofitted. Moreover, the Internet is not governed in any organized way. Rather, it is influenced by impromptu committees that nudge practices in certain directions.
These plans must compensate for e-mail technology designed two decades ago by a happy-go-lucky confederation of computer scientists. They created the protocols - the technical communication rules for e-mail - in such a trusting and open way that now anyone can send e-mail impersonating someone else with little prospect that the messages can ever be traced to the original sender. Little did these pioneers suspect that the systems, meant to exchange research papers, would lead to a global system that provides nearly instant communication for hundreds of millions of people, yet now is in danger of being overrun by anonymous purveyors of pills and pornography.
A lot of money is riding on how these problems are solved. And under the technical discussion of spam fighting systems is a power struggle between the companies that send a lot of e-mail and the large Internet service providers. The e-mailers are exasperated with the filters and want a system that defines their e-mail as legitimate and guarantees its delivery.
The Internet providers are concerned more about customer complaints and do not want to promise to deliver any particular mail. Some are also wary about creating an industry standard for spam fighting, because the big providers are finding that their proprietary spam filters attract new customers.
Circling all of these discussions are a group of e-mail handling companies and other organizations that expect to profit if their identification ideas are accepted as the standard. (Even the Postal Service wants to get into the act, with an electronic postmark.)
The upper hand is probably held by the four largest service providers - Microsoft, America Online, Earthlink and Yahoo - which have been meeting since April to try to define spam fighting standards. They say they must go slowly out of respect for the decentralized nature of the Internet (and on the advice of their lawyers).
"We're very self-conscious about being a big player in the e-mail business, and we don't want to be seen as laying out the law for everyone," said Brian Sullivan, senior director for mail operations at America Online. "We need to build a consensus around a framework."
There already does appear to be agreement that any system would be optional for senders. Most likely big commercial e-mailers would use it at first. Mail from others that did not adopt these new technologies - individuals, small business, those in foreign countries - might still be delivered but it would be subject to greater scrutiny.
The clearly identified mail "will be like the express line at the airport," said Kevin Doerr, a business manager for antispam products at Microsoft. "You will only be frisked once and not thrown in with the unwashed masses."
Eventually, however, Internet Service providers expect to develop easy ways to help individuals and small businesses identify their e-mail so it, too, would have the benefit of the express line through the spam filters.
Most likely, individual users would not have to do anything differently as they read their mail. But some of the more elaborate proposals envision requiring users to get updated software that they could set to determine the sort of e-mail they want to read or delete.
There is also a growing agreement that it is not enough for an e-mail sender to identify itself. The sender must also earn the trust of e-mail recipients, by promising to follow certain standards and having violations tallied and published. That would let people choose to discard mail from senders with high complaint rates.
"Just because we can verify your identity doesn't mean you send good email," said Miles Libbey, the manager for antispam products at Yahoo. "You absolutely need identity and you also need reputation."
Deciding whose mail will be delivered and whose will be bounced is a thankless task, and most proposals envision that several independent groups would publish e-mail standards. A sender would choose one of these to follow, and that group, in turn, would monitor its compliance. Truste, a group that monitors Web site privacy policies, wants to get into that business.
The biggest unanswered question is how to identify each sender in a way that cannot be counterfeited. As with so many other issues, there is a choice between simple and quick solutions that are limited and technically flawed and more comprehensive approaches that will take years to fully adopt.
The most robust, but most complicated approach would have every sender attach to each message a unique code, called a digital certificate, which the recipient's Internet provider or e-mail software could authenticate.
Others argue that a far simpler approach would be good enough: they propose the creation of a registry of e-mail servers - the computers that process the mail for big companies and internet providers - whose owners have been verified. Any mail sent from an unregistered server would be automatically suspect.
The Internet service provider group appears to prefer this approach, at least as a first step.
But the big e-mail senders, and some security experts, say that it would be flawed. They note that spammers appear to be skilled and shameless hackers. Many have found ways to secretly take over the computers of unsuspecting people to relay their messages and to attack computers used by groups that fight spam.
Only digital certificates, they say, can clearly distinguish legitimate e-mail senders from counterfeit ones.
"It's not easy to change something as successful and widely used as e-mail," said Richard Reichgut, a vice president at AuthentiDate, a company that has a contract to develop electronic postmarks based on digital certificates for the Postal Service. "But the only way to fix e-mail is to have a strong way to know who is sending you mail."
Moreover, using digital certificates can allow finer distinctions among mail from one sender. A bank's credit card statements might get one identifier and its offers of new services another.
"My concern is that the solution not be penny-wise and pound-foolish," said Hans Peter Brondmo, a senior vice president of Digital Impact, a company that sends mail on behalf of big companies. "We could take six more months and have something that is far more comprehensive."
Mr. Brondmo is one of the main developers of Project Lumos, a sender identification proposal by the E-mail Service Provider Coalition, a group of big e-mail senders.
Some Internet service providers acknowledge that ultimately some form of digital certificate may be useful. But they disagree that the simpler approach, based on an existing identification number assigned to each server called Internet Protocol address, is too vulnerable to attacks.
Internet Protocol "spoofing is hard to do and easy to detect," Mr. Doerr of Microsoft said.
And that position is endorsed by Paul Q. Judge, the chairman of the Anti-Spam Research Group of the Internet Emerging Issues Task Force, one of the main groups that publishes standards for the Internet.
"There is a huge gain that can happen today with I.P.-based systems that use existing technology," Mr. Judge said. "The benefit of having a certificate for each individual e-mail is not worth the hassle and the cost."