My great passion for text classification has lead to a new web service - uclassify.com. A text classifier is used to automatically categorize documents, exactly the way Cactus Spam Filter successfully has been doing for years now!

The only difference is that you don’t have to be a rocket scientist to write your own classifiers any longer. Anyone with basic programming skills can do it by using our xml api!

Ahh yes, it’s free and you are allowed to make money using the service!!

Sites that already are using uclassify.com

typealyzer.com - Analyzes the personality of a blog
genderanalyzer.com - Tries to figure out if a blog is written by a man and woman
ofaust.com - Matches texts agains classical authors such as Plato, Shakespeare, Tolstoy etc.

Web developers, start classifying and be a pioneer of Web 3.0!

—————————
Privacy Mantra Error
—————————
An unexpected Exception occured. Please help us to improve this product by e-mailing us this message.
You can press CTRL+C to copy it. E-mail:

Privacy Mantra v2.04
Could convert the wide string to a normal string.
Method: PrivacyApp::InitInstance
..\..\..\..\source\products\privacy\privacy.cpp (301)
—————————
ОК  
—————————

This bug has been bugging me for quite a while since I’ve not been able to recrate it locally.  Finally I was able to track it down with great testing help by Piotr Nawratowicz (Poland), thank you very much! I really appreciate it! So one less bug, yay. Lol, even the error message in the assert was wrong  “Could convert” should of course be “Could not convert”.

If you are a previous donator and wish to appear on this list, please let me know!

Problem

For example, these snippets to include and require the existing file Log.php in a page or post

<?php include(”http://www.codeode.com/scripts/Log.php”); ?> or <?php require_once(”/scripts/Log.php”); ?>

would generate the following warnings:

Warning: Unknown(): URL file-access is disabled in the server configuration in /mnt/local/home/codeode/codeode.com/wp-content/plugins/exec-php/includes/runtime.php(42) : eval()’d code on line 4

Warning: Unknown(http://www.codeode.com/scripts/Log.php): failed to open stream: no suitable wrapper could be found in /mnt/local/home/codeode/codeode.com/wp-content/plugins/exec-php/includes/runtime.php(42) : eval()’d code on line 4

Warning: Unknown(http://www.codeode.com/scripts/Log.php): failed to open stream: no suitable wrapper could be found in /mnt/local/home/codeode/codeode.com/wp-content/plugins/exec-php/includes/runtime.php(42) : eval()’d code on line 4

Warning: (null)() [function.include]: Failed opening ‘http://www.codeode.com/scripts/Log.php’ for inclusion (include_path=’.:/usr/local/lib/php’) in /mnt/local/home/codeode/codeode.com/wp-content/plugins/exec-php/includes/runtime.php(42) : eval()’d code on line 4

Ok, as a WordPress rookie I am not completly sure why this is happening in the first place, anyone?

I started to search for solutions to the problem, but couldn’t find any good ones, only cumbersome that indicated me into changing the php.ini.

Solution

I ended up looking into WordPress code and found that they internally are using a variable called ABSPATH. This worked for me with WordPress 2.5.1:

<?php require_once(ABSPATH . “/scripts/Log.php”); ?>

I hope this post helps someone out there

Hiding the e-mail address

The simplest approach to avoid spam is to keep the e-mail address hidden from spammers. The email address can be revealed only to trusted parties. For communication with less trusted parties a temporary e-mail account can be used. If the e-mail address is published on a web page it can be disguised for e-mail spiders by inserting a tag that is requested to be removed before replying. Robots will collect the e-mail address with the tag, while humans will understand that the tag has to be removed in order to retrieve the correct e-mail address. For most users this method is insufficient. Firstly, it is time consuming to implement techniques that will keep the e-mail address safe, and secondly, the disguised address could not only mislead robots, but also the inattentive human. Once the e-mail address is exposed, there is no further protection against spam.

Pattern matching, whitelists and blacklists

This is a content-based pattern matching approach where the incoming e-mail is matched against some patterns and classified as either spam or legitimate. Many e-mail programs have this feature which is often referred to as “message rules” or “message filters”. This technique mostly consists of a plain string matching. Whitelists and blacklists, which basically are lists of friends and foes, fall into this category. Whenever an incoming e-mail is matched against an entry in the whitelist, the rule is to allow that e-mail through. However whenever an e-mail has a match against the blacklist, it is classified as a spam. This method can reduce spam up to a certain level and requires constant updating as spam evolves. It is time consuming to determine what rules to use and it is hard to obtain good results with this technique. In Mertz D. 2002 some simple rules are presented. The author claims that he was capable of catching about 80% of all spam he received. However, he also stated that the rules used had, unfortunately, relatively high false positive rates. Basically, this technique is a simpler version of the more sophisticated “rule based filters” which are discussed below.

Rule based filters

This is a popular content-based method deployed by spam filtering software such as SpamAssassin. Rule-based filters apply a set of rules to every incoming email. If there is a match, the e-mail is assigned a score that indicates spaminess or non-spaminess. If the total score exceeds a threshold the e-mail is classified as spam. The rules are generally built up by regular expressions and they come with the software. The rule set must be updated regularly as spam changes, in order for the filtering of spam to be successful. Updates are retrieved via the Internet. The tests results from the comparison of anti-spam programs presented in Holden 2003 show that SpamAssasin finds about 80% of all spam, while statistical filters (discussed later) find close to 99% of all spam. The advantage of rule-based filters is that they require no training to perform reasonably well. Rules are implemented by humans and they can be very complex. Before a newly written rule is ready for use, it requires extensive testing to make sure it only classifies spam as spam and not legitimate messages as spam. Another disadvantage of this technique is the need for frequent updates of the rules. Once the spammer finds the way to deceive the filter, the spam messages will get through all filters with the same set of rules.

Statistical filters

In Sahami et al. 1998, it is shown that it is possible to achieve remarkable results by using a statistical spam classifier. Since then many statistical filters have appeared. The reason for this is simple; they are easy to implement, have a very good performance and require a little maintenance. Statistical filters require training on both spam and non-spam messages and will gradually become more efficient. They are trained personally on the legitimate and spam e-mails of the user. Hence it is very hard for a spammer to deceive the filter.

E-mail verification

E-mail verification is a challenge–response system that automatically sends out a one-time verification e-mail to the sender. The only way for an e-mail to pass through the filter is if the sender successfully responds to the challenge. The challenge in the verification e-mail is often a hyperlink for the sender to click. When this link is clicked, all e-mails from that sender are allowed through. Bluebottle and ChoiceMail are two such systems. The advantage of this method is able to filter almost 100% of the spam. However, there are two drawbacks associated with this method. The sender is required to respond to the challenge which necessitates extra care. If this challenge is not recognized the e-mail will be lost. Verifications can also be lost due to technical obstacles such as firewalls and other e-mail response systems. It can also cause problems for automated e-mail responses such as online orders and newsletters. The verification e-mail also generates more traffic.

Distributed blacklists of spam sources

These filters use a distributed blacklist to determine whether or not an incoming e-mail is spam. The distributed blacklist resides on the Internet and is frequently being updated by the users of the filter. If a spam passes through a filter, the user reports the e-mail to the blacklist. The blacklist is updated and will now protect other users from the sender of that specific e-mail. This class of blacklists keeps a record of known spam sources, such as IP numbers that allow SMTP relaying. The problem involved in using a filter entirely relying on these blacklists is that it will generally classify many legitimate e-mails as spam (false positive). Another downside is the time taken for the networked based lookup. These solutions may be useful for companies assuming that all their e- mail communications are with other serious non-listed businesses. Companies offering this service include Mail Abuse Prevention System LLC (MAPSSM) and Spamcop.

Distributed blacklist of spam signatures

These blacklists work in a same manner to that described in previous paragraph. The difference is that these blacklists consist of spam message signatures instead of spam sources. When a user receives a spam, that user can report the message signature (typically a hash code of the e-mail) to the blacklist. In this way, one user will be able to warn all other users that a certain message is spam. To avoid non-spam being added to a distributed blacklist, many different users must have reported the same signature. Spammers have found an easy way to fool these filters; they simply add a random string to every spam. This will prevent the e-mail from being detected in the blacklist. However spam fighters attempt to overcome this problem by adapting their signature algorithms to allow some random noise. The advantage being that these kinds of filters rarely classify legitimate messages as spam. The greatest disadvantage is they are not able to recall much of the spam. Vipul’s Razor uses such a blacklist and states that it catches 60%-90% of all incoming spam. Another disadvantage is the time taken for the network lookup.

Money e-mail stamps

The idea of e-mail stamps is not new, having been discussed since 1992, but it is not until recently that major companies have considered using it to combat spam. The sender would have to pay a small fee for the stamp. This fee could be minor for legitimate e-mail senders, while it could destroy business for spammers that send millions of e-mails daily. There are two stamp types; money stamps and proof-of-work stamps (discussed later). GoodmailSystems is developing a system for money stamps. The basic idea is to insert a unique encrypted id to the header of each sent e-mail. If the recipient ISP is also participating in the system, the id is sent to Goodmail where it is decrypted. Goodmail will now be able to identify and charge the sender of the e-mail. Today there are many issues requiring solutions before such a system can be deployed. Who receives the money? Where is tax paid? Who are allowed to sell stamps? Since this is a centralized solution, what about scalability? It would also be the end of many legitimate newsletters.

Proof-of-work e-mail stamps

At the beginning of 2004, Bill Gates, Microsoft’s chairman, suggested that the spam problem could be solved within two years by adding a proof-of-work stamp to each e-mail. Camram is a system that uses proof-of-work stamps. Instead of taking a micro fee from the sender, a cheat-proof mathematical puzzle is sent. The puzzle requires a certain amount of computational power to be solved (matter of seconds). When a solution is found, it is sent back to the receiver and the e-mail is allowed to pass to the receiver. The puzzle Camram is using is called Hashcash. Whether it is money or proof-of-work e-mail stamps, many oppose the idea, not only because emailing should be free, but also because it will not solve the spam problem. To make this approach effective, most ISP’s would have to join the stamp program. As long as there are ISP’s that are not integrated into the stamp system, spammers could use their servers for mass e-mailing. It could then still be possible for the legitimate e-mailers to pay to send e-mails, while spam is still flooding into the inboxes of users. Many non-profit legitimate mass e-mailers will probably have to abandon their newsletters due to the sending cost. Historically, spammers have been able to deceive most of the other anti spam filters and this could also be the case with the stamp system.

Legal measures

In recent years many nations have introduced anti-spam laws, in December 2003, president George W. Bush signed the CAN-SPAM act, the Controlling the Assault of Non-Solicited Pornography and Marketing Act. The law prohibits the use of forged header information in bulk commercial email. It also requires spam to include opt-out instructions. Violations can result in fines of $250 per e-mail, capped at $6 million. In April 2004 the first four spammers were charged under the CANSPAM law. The trial is still on, but if the court manages to send out a strong message, this could deter some spammers. The European Union introduced an anti-spam law on the 31st of October 2003 called “The Directive on Privacy and Electronic Communications”. This new law requires that companies gain consent before they send out commercial e-mails. Many argue that this law is toothless since most of the spam comes from the outside of EU. In the long-run legislation can be used to slowdown the spam flood to some extent, but it will require an international movement. Legislation will not be able to solve the spam problem by itself, at least not in the near future.

Conclusion

The most commonly used methods for eliminating spam were described in this chapter. Perhaps legislation is the best option in the long run. However, it requires a world wide effort and this process could be slow. Presently users need to protect themselves and for the moment statistical filters are the most promising method for this purpose. They have superior performance, can adapt automatically as spam changes and in many cases are computationally efficient.

References

• Mertz, D., (2002), Spam filtering techniques, 2004-02-03.
• Holden, S. (2003), Spam Filters, 2004-03-18.
• Sahami, M., Dumais S., Heckerman D., and Horvitz, E. (1998), A Bayesian approach to filtering junk e-mail. Learning for Text Categorization. Papers from the AAAI Workshop, Madison Wisconsin, pp. 55–62. AAAI Technical Report WS-98-05.
• Graham, P. (2002), A plan for spam, 2003-11-13

The idea is to have a more live and interactive site that is easy to update and get feedback on. Oh, I think the screaming is about a football game, Sweden VS Spain. It seems like we lost. Crap.

Feel free to comment on the new site, any feedback is appreciated (not spam though, lol, I may have to integrate Cactus Spam Filter into WordPress if Akismet isn’t doing it good enough.)

/Jon