Time and money are lost on reading of a spam by employees of the design company. In 2003 in many companies the share of the spam correspondence in incoming mail made already 60-70%! In a stream of uninvited advertising the important business letters are quite often lost that also leads to serious financial losses (still the big losses arise at use of incorrectly working means of a filtration of a spam with considerable percent of false operations). Статьи
Despite that today there is a considerable quantity of the convenient free post clients, some employees of the design organisation continue to use post programs to which interfaces of the user they have got used. One of such popular programs is paid program The Bat!
According to the developers, the guaranteed quality of a filtration of a spam provides a trained plug-in for The Bat! On the basis of spam-filter AGAVA Spamprotexx, created with the assistance of developer The Bat! Company Ritlabs.
We understand the advertising offers dispatched by e-mail as a spam.
Let's consider behaviour rules in the environment the Internet.
“Behaviour rules” have enough general character and their purpose is not hindrance to a spam, and prevention of its occurrence in your box.
First, it is desirable not “to distribute” the address to the right and on the left – not to leave it, for example, at various forums, conferences, guest books, bulletin boards, etc. not to publish it on the site, not to bring it in the various registration forms busily offered by many Internet resources. But, so far as without it it is not possible to manage, it is desirable to have some mail boxes – at least one – for business, and one (for example, on one of free post servers) – just for this purpose. At least, thus you can “to separate flies from cutlets” using the additional address for tasks of second importance.
At definition of the address it is desirable to use long and difficult designs (it is desirable with figures), not containing English words or Russian words a Latin, and also widespread names. This principle is caused by feature of work of many spam programs of search of names, and directed on difficulty of their work.
Specifying the address at forums, bulletin boards, etc., it is desirable to alter it – for example, “Vasya777Pupkin (dog) domain.ru” that spam programs-robots have not reacted to a symbol.
In case of need address placings on the (corporate) site, it is desirable to register it in the implicit (ciphered) kind – by means of a corresponding script or in the form of the image. It is less convenient for the one who will want to take advantage of it, but it is absolutely inconvenient for spam robots.
Do not answer and do not send anything to the spamer – a unique conclusion which it will make of your answer, there will be a confidence that your address really exists and mail arriving there is read by the owner. Accordingly, statements in some spam messages that you can exclude yourselves from the dispatch list, having sent to the certain address a command “remove” or having come on a certain site, as a rule, are lie. Neither to answer the spamer, nor to try “to send” some mbyte of dust is not necessary to it – at the best, it is possible to complain of too annoying spamer to the provider – to sense, most likely, will not be, but also superfluous such measure too does not become.
On use theme antispam programs we will stop slightly in more details.
Let's consider principles of work of antispam programs.
Work of the majority of antispam programs is based on use of three methods – in this or that combination.
The first method – use “black” and “white” lists. In the white list addresses contain, mail which has come with which, obviously a spam is not. Accordingly, black lists carry out opposite function – the messages received from addresses from these lists automatically are marked as a spam and can be automatically removed from a server. Black and white lists are formed by the user – accordingly, their efficiency increases in due course. Some programs are capable to use also “public” black lists, loadable from specialised Internet resources.
In the majority of programs at formation of lists probably use so-called wild cards – that is, incomplete addresses with which help you can blacklist all addresses from the certain domain or, for example, all addresses containing a combination of symbols vasya88.
Advantages. First, you can define in the white list of the address messages with which obviously will not get to dust without dependence from their maintenance. In the second, you can exclude completely reception of mail from certain domains. For example, if you basically do not assume to receive the letter from the Dutch domains, it is enough to register of something in black sheet like (according to syntax of the concrete program) *@*.nl. or nl.
Lacks. Recently spamers even more often resort to the mass dispatches made from free post servers (for example, hotmail). For this purpose it is got “disposable” a mail box which will not be used subsequently. Thus, entering of such address in the black list practically is not meaningful, and entering in this list of all domain can be unjustified – for example if someone from your acquaintances uses this post server. Besides, efficiency of such method increases gradually – in process of formation of lists, and first the program overall performance will be minimum. Also it is not necessary to speak about high degree of automation of process – it is necessary to tinker with lists “handles” as you should process anyhow letters from the senders who are not appearing in lists.
The second method – inquiry about acknowledgement. It is used, as a rule, in a combination to the first method. At reception of mail from the sender which addresses are not present in the white list, the program automatically generates the letter-answer. In it the inquiry about intention to enter into a correspondence will be sent the sender. It is supposed that if it is really necessary for the recipient that you have received his letter he will not be slow to answer, and its post address will be brought in the white list. If is not present, the program will remove the letter from a server through the established time interval. Besides it, the address of the sender will be declared “unreliable” also it is blacklisted. This method recognises that spamers, as a rule, do not answer letters of the “victims”. For example:
Hello! We inform you that your letter sent to the addressee%email of% is detained. That the addressee all the same has received your letter, it is necessary to answer this message simply. Will pay, please, attention that the given acknowledgement means that in case of the answer to the given message, the addressee will receive only one your letter and if you are the spamer, all your subsequent messages will leave without the prevention. If you do not answer the given message within 1 week, your letter will be removed.
Advantages. Strengthens an overall performance with the white list. Excludes possibility of reception of letters from unknown senders. From the point of view of elimination of the messages sent by senders unknown to you, the method is the extremely effective.
Lacks. The specified advantages bear in themselves and lacks. At all the fact that “diligent” (but yet known to the program) the correspondent can answer in time on inquiry – everyone happens. As a result you can lose the important message, or with lateness will read the urgent letter. Besides, the convenience offered by this method, is realised by creation of inconveniences for any sender who is not appearing in the white list. At volume correspondence you too hardly will like to send instead of one letter two. Besides, in the event that the spamer uses the real-life address, it also can automate sending of the answer to your inquiry.
The third method – the analysis of the office information and the message text. In particular, the way of passage of the letter and conformity of some office fields of the message – for example, conformity of the address from which the letter has been sent, to the address specified for the answer is analyzed. Besides, the heuristic (is intellectual-semantic) analysis of a field of a theme of the message and the text of the message about revealing of the typical linguistic designs used by spamers is carried out. The text analysis is usually made by comparison of the text of the message with the certain preliminary established templates, and also by search in a body and heading of the letter of keywords and word-combinations and their comparisons with the set set of samples.
Advantages. Possibility to eliminate the undesirable correspondence without dependence from, whether appears the address of the sender in the black list. Depending on quality of the analyzing program – high enough elimination of undesirable mail.
Lacks. Rather high possibility of elimination of the necessary messages. Actually, here dependence is observed: the above percent of the detained spam, the above and a dropout rate of the necessary correspondence. The author on a private experience was convinced of this lack: attempt to send to the office address the text of article connected with a theme of cash registers, has terminated in failure. The certain serious antispam program “has rejected” article in dust …
Let's consider some most popular antispam programs.
To compare among themselves antispam programs, probably, not so correctly. From the point of view of an overall performance it is impossible to spend any powerful enough comparative researches – one program will eliminate one messages, having passed others, and another – absolutely on the contrary. Thus in percentage expression the relation of the detained spam to its general volume at them can be identical.
Therefore, probably, it is necessary to pay attention only to those characteristics of programs which concern convenience of the user – without forgetting thus about used methods of work. It is necessary to carry program cost, russification, support of Cyrillics in looked through headings (texts) of messages, degree of automation of work, convenience of adjustment, volume of a place taken in memory and a number of other characteristics to such characteristics.
The program extends free of charge. Distribution kit volume – 62K (zip) or 98K (exe-installer). Official russification is not present. Can work with several accounting records (mail boxes), however to adjust them it is necessary manually.
Program K9 analyzes arriving electronic letters and allocates a potential spam. Working together with the post program which is carrying out connection under report POP3 (for example, Outlook Express), the program uses results of the heuristic analysis for a filtration. Presets-samples for the analysis are minimum, therefore the program it is necessary “to train”. After “training” such approach relieves the user of necessity of regular loading of updatings and manual input of rules. The initial result will be modest, but in process of training K9 remembers signs of messages which you carry to a spam, and marks all similar messages. Thus, the program itself adjusts filters, using your statistics. In noted as the spam messages can make entry in the field “the Theme” – for example, a word “SPAM”. White and black lists of addressees are provided also.
The program extends free of charge. Distribution kit volume – 606K (MailChecker 2003) or 457K (MailChecker) in rar-archive. The Russian interface, supports all basic Russian codings. Can work with several accounting records.
Works in a background mode. Check occurs without uploads of letters on the local car though it is possible download of letters, including in an automatic mode at will of the user. Presence of black and white lists. Viewing of post messages and the enclosed files is realised. Removal of messages directly from a post server without uploads on the car, automatic removal of messages from senders “the black list”. Mail sending under report SMTP without use of external post clients is possible. There is a possibility automatic uploads of all letter and installation of code page on which each new letter from the sender from the white list will be automatically recoded.
For citizens of the former USSR cost of the limitless version – $15.00. The Russian interface. Works with several accounting records (defines them automatically). The program is integrated with Outlook Express. The size of the distribution kit – approximately 1.5M.
Filtration rules are updated by developers of the program through the Internet on-line. Presence of white and black lists, with support wild cards. In case the program is switched off, the post client cannot accept incoming mail. The general rules (black lists) are formed by all community of users of the program – by departure “complaints” under concrete spam letters. Five levels of a filtration of entering messages: without a filtration, low (only personal black sheet), average (personal lists and safe rules), high (includes unsafe rules – risk of loss of the necessary correspondence) and extra (only the white list).
The program extends free of charge. Russifications are not present. Support of several accounting records which are defined automatically (except for input of passwords). The size of the distribution kit – approximately 2.7M.
Works independently from the post client – at first it is necessary to start Email Control, to make clearing and then to open the post program and to accept mail. Actually, is almost high-grade post client processing mail directly on a server. Filters are defined by the user. Support of black and white lists. Possibility of a filtration of the correspondence not only on the address of the sender, but also on a theme of the letter and its contents. The built in service WHOIS, allowing to learn the information on a post server with which help the letter has been sent.
The program extends free of charge. There is a support of Russian and a management in Russian. The size of the distribution kit – approximately 400K. Support of the several accounting records which are adjusted manually.
It is simple in circulation. Supports personal black and white lists. It is integrated with various post clients. In a case if the program is switched off, the post client cannot accept incoming mail. Besides, can use lists DNSBL – dynamically updated lists of IP-addresses of servers with which help mass dispatches of mail were made. The program adds office heading X-SpamPal: SPAM on which basis the post client makes a filtration. The filtered suspicious letters do not leave, and are located in preliminary created folder of the post client and can be seen later. The basic feature of the program – possibility to expand its functionality for the account docking of plug-ins which can be found on a site of manufacturers.
Registration cost: for private persons – $5, for the organisations – $10. The Russian interface. The size of the distribution kit – approximately 900K. Support of the several accounting records which are adjusted manually.
It is integrated with various post clients. In a case if the program is switched off, the post client cannot accept incoming mail. Supports personal black and white lists. A typical example of the program working on a method of inquiries about acknowledgement: at reception of mail from the sender which addresses are not present in the white list, the utility automatically generates the letter-answer. In it the inquiry about intention to enter into a correspondence will be sent the sender. If it is really necessary for the recipient, that you have received his letter he will not be slow to answer, and its post address will be brought in the white list. If is not present, WinAntiSPAM will remove the letter from a server through the established time interval. Besides it, the address of the sender will be blacklisted.
The program extends free of charge. Russian complete support. The size of the distribution kit – approximately 500K. Support of the several accounting records which are adjusted manually.
The program for management of letters in a mail box under report POP3. Application “intellectual” checks of new messages does the program of communication tolerant to ruptures with a server (repeatedly headings and messages are not loaded), and also allows to define and delete duplicates of messages and to inform that among addressees of the message there is no address of the given mail box. Use of rules for messages allows to appoint to the message a badge, colour of the text, to put marks “to remove from a server” and “to load from a server”. Application “The fast filter” allows to display quickly messages from the same sender, with the same theme, with the same badge as at the current allocated message. Possibility of search and sorting of messages according to contents of various fields.
Magic Mail Monitor
The program extends free of charge. Russian support is absent. The size of the distribution kit – 71K.
The program is very simple in use. Works independently from the post client. Actually, is the simple post client processing mail directly on a server. Support of black and white lists. The interface is divided into two windows: in the top themes of the letters which are on a server are displayed, and in bottom it is possible at desire to see their contents.
The program extends free of charge. There is an informal russification. Not all Cyrillic fonts are supported. The size of the distribution kit – 1.5M. Support of the several accounting records which are adjusted manually.
The program is very simple in use. Works independently from the post client – program start is necessary before mail reception. Support of the black and white lists formed by the user. Use possibility wild cards in lists. Possibility of viewing of the contained letter.
The program extends free of charge. Russian support is absent. The size of the distribution kit – approximately 1.2M. Support of work with several accounting records is absent.
It is integrated with various post clients. Defines a message accessory to a spam on the basis of the frequency analysis of symbolical sequences entering into the message.
Program cost – approximately $100. Russian Complete support. A complex of protection against a spam of corporate networks which can be established both on a server, and on a separate workstation.
Efficiency of the decision is provided at the expense of simultaneous use of system updated in a mode on-line “black” lists, use of samples of letters and application of system of the heuristic analysis of incoming documents. The program provides a filtration of post messages even before their hit in boxes of end users. Can be used as the filter together with any corporate post system – Sendmail, Qmail, Postfix, MS Exchange, etc. For receipt prevention in mail boxes of undesirable messages in the program some methods of a filtration of various attributes of the letter are realised, namely: addresses of the sender and the addressee, the size and a transit of the letter, its heading. The filtration of addresses provides check on presence of e-mail addresses and IP addresses in “black lists”. The program allows to carry out content processing of the enclosed files in formats Plain Text ASCII, HTML, MS Word 6.0, RTF. In the program support of Russian and English of languages is provided. Depending on preliminary set options, passed a filtration and the letter carried to this or that category can be delivered to destination (without any changes or with addition of the corresponding heading specifying in an accessory of the letter to this or that category, defined as a result content to a filtration), it is redirected on any certain address or it is removed.
Plug-in AGAVA Spamprotexx in “The Bat!”
To the user “The Bat!” it is more convenient to use plug-in AGAVA Spamprotexx for following reasons:
- Uses the built in mechanisms “TheBat!” for access reception to incoming mail. Works faster and is more convenient in use at the expense of close integration with the post client.
- Does not change headings of letters.
- It is adjusted from the interface of the post client.
- Training is made by means of points of menu TheBat! Specials – Mark As Junk, Mark As Not Junk.
- The plug-in is protected from “conversion training” since at first classifies letters given for training.
By results of the interrogation spent Softkey.info, more than half of users have given preferences at a choice of the post client to program The Bat!. Since the second version, this program began to support plug-ins of external developers. For post clients the first requirement is protection of mail of the user against a spam. There are some antispams-plug-ins, capable to work with program The Bat!. One of them – a plug-in from the company “Agave” created on the basis of antispam-filter Agava Spamprotexx with the assistance of developer The Bat! Companies RitLabs. This filter uses simple, but the effective programs based on Bayesian algorithm.
Let's consider Bayesian algorithm.
The Bayesian algorithm is based on statistical methods of the analysis of the words most often meeting in spams-messages. This method has the advantages, but also is not deprived lacks. The antispam programs working on the basis of this method, frequently incorrectly realise advantages of a statistical method, than and cause censures. According to founders of program Spamprotexx, they managed to solve the following problems inherent in programs, working on Bayesian algorithm:
- Sensitivity reduction to errors in the course of training. Spamprotexx uses some know-how which watch a condition of a database and exclude casual erroneous additions of correct letters in a spam. There is a possibility to rectify an error, having sent the same message for training as correct.
- The account of typical errors. Creation by spamers of a new way of detour AntiSpam protection leads to an avalanche of spam messages in post clients of users. Carrying over of such messages for training in programs of protection against a spam leads to that certain types of messages get the overestimated factors that leads to so-called excessive training of a database. Spamprotexx solves this problem, classifying each message before being trained on its example. If it manages to classify it it will not admit this message for training.
- Account HTML-tegov. The majority of spams-messages comes in the form of HTML, as a result of it some programs start to perceive standard HTML-tegi as spams-words. Spamprotexx uses parser HTML to exclude influence tags on classification. Instead of inclusion of all tags in spams-words Spamprotexx pays attention to their properties – fonts, paragraphs, a body, images etc. For example, Spamprotexx it is capable to find out and remember that spams-messages are often created with use of fonts of certain colours, the sizes and types.
- The analysis of headings. In case of very short text messages Spamprotexx uses headings of these messages (1-2 kilobytes of the information) for successful classification of the letter as spam. In headings a lot of information helping Spamprotexx to make the correct decision contains.
- The account of office parts of speech. There are many words (for example, pretexts) which are not characteristic for a spam or not spam. Without this feature there is a decrease in quality of a filtration because filters in the majority study on spams-examples, rather than on usual letters more likely. Spamprotexx has stop sheet for such words not to use them for the purpose of classification.
At last, we will consider, Bayesian algorithms in realisation from company Agava how much productively work.
Let's consider realisation of Bayesian algorithms – Agava Spamprotexx.
First of all, we will notice that Agava Spamprotexx works with all post clients and does not demand their adjustment. It is very convenient for any user, and especially for unprepared: it is not necessary to enter proxy-server parametres. However, the program nevertheless is more convenient for users of post clients from Microsoft (Microsoft Outlook and Outlook Express), than for other post clients. Panels of commands microsoft’s post clients Spamprotexx builds in special baskets for training. It is enough to transfer the spam which has made the way through protection to a red basket, and the message wrongly classified as a spam, – in dark blue, and Spamprotexx will apprehend these messages for training. For other post clients for training it is necessary to make slightly more actions, than simple dragging of the letter.
For training not microsoft’s post programs in options Spamprotexx it is necessary to enter special post addresses and on special algorithm (described in the user's guide) to send messages for training. In the same place it is possible to set and a spam-label by which field Subject of the letter will be marked at classification of this letter as a spam. Options are required to be carried out only once.
Spamprotexx creates in post clients a special folder in which puts the messages classified as a spam. The user can see at any moment this folder about the messages wrongly qualified as a spam. Normal mail arrives in a folder “Entering” or it is sorted by the sorting rules, the set user for the post client. If necessary it is possible to see a broad gull of work of the program always.
The program conducts the “white” the list of the entrusted correspondents. Feature is that at reception of letters from such entrusted correspondents there is a check on conformity of a name and the electronic address. If the divergence between them the letter is checked on a spam, as usual is revealed. At reception of a spam from the entrusted correspondent the address is excluded from base “white” the list.
In options Spamprotexx possibility of replacement of level of protection is provided.
Levels recommended by the developer are exposed by default. These values are recommended for a permanent job. But for an initial stage it is more preferable to establish higher level (an order of 80-90%). At such range lie-positive of results (the normal electronic letter classified as a spam) practically will not be, but you should send often enough broken spam on training.
Installation is very simple: simply to start the exe-file of the program downloaded from a site of the developer. However, right after the plug-in will not earn installation: it still needs to be connected.
Plug-in connection is made by standard means of program The Bat!. Through a command “Properties – Adjustment – Protection against a spam” the window of connection of external plug-ins is caused.
Input of passwords, logins, servers, choice of reports is not required. The plug-in is already ready to work, it is possible to swing mail. But for correct work it is desirable for adjusting.
First of all it is recommended to create “white” the list of your correspondents. It will be the biggest work of the user at program adjustment. Users can be added manually or to make import from a file in format TXT. The third and most simple variant – to send on training all normal letters. Then senders will automatically be added in the list of friends.
If at you the big base of the checked up users, entering in “white” the list will occupy a lot of time. It is possible not to make such mass addition of addressees in the list, and to add them as required. However, thus you lose possibility to facilitate to yourselves residing at an initial grade level of the program. In plug-in options there is a function “To be trained on letters from friends as on not a spam”. If to add in base of the addressees and to include this function the plug-in will begin automatic training on letters from your entrusted correspondents, than will facilitate to you a life.
Entering in “white” the list probably and at manual training on letters as on not a spam. For this purpose it is necessary to make active function “to Add addresses automatically at training on not a spam”. That is now, if you mark the received letter as not a spam, the plug-in automatically brings it in “white” the list.
Addition in “white” the list and inclusion of the above-named functions are made in a window of options of the plug-in, unlike installations for work with letters according to the rating appropriated to these letters.
Work adjustment on a rating is made in a window of options of program The Bat! On command “Properties – Adjustment – Protection against a spam”. Such mechanism of adjustment is realised because the plug-in actually only calculates a letter rating on the internal algorithms, and sorting of letters is made by already post client.
The rating can be understood as average, minimum and maximum. The direction of actions of functions of the program depends on a choice of these installations. If the understanding of a rating as minimum all made active functions are applied to letters with a rating above the established is chosen. If the understanding of a rating as maximum, on the contrary is chosen. For inexperienced or those whom laziness to potter with options, it is recommended to leave understanding of a rating exposed by default as an average.
Values of a rating are very easy for changing, being adjusted under personal concrete preferences. If you are a private user and number of your correspondents is limited, work of programs can be toughened, having raised values of a rating for removal of letters from a server and for mail estimation as spam. If you conduct active correspondence and at you a great number of casual correspondents ratings it is necessary to lower and reconcile to occasionally passing spam.
Program training is made by means of standard means of the post client. The letter which needs to be classified as a spam or not a spam, it is necessary to make active. Then by means of the contextual menu to choose a command “Special – to Mark as not a spam”. For spam letters of action are similar, the command only gets out “to Mark as a spam”. As you can see, training process is very simple now and is not burdensome.
By the way, developers from RitLabs have for some reason decided not to appropriate to these points shortcuts – a combination of keys for a fast call of corresponding functions. But it can be made manually of interface The Bat!: “The kind – Combinations of keys”.
In the first weeks there is a plug-in training, but also not trained it shows quite good results. Results higher, than at plug-in Bayes it, moreover – are not present effect “conversion training” peculiar to the last.
1. Basyrov R. Cricket projectile and armor [http://softkey.info/reviews/review1414.php]
2. We filter a spam in The Bat! – 2007 [http://www.klerk.ru/soft/articles/84435/]
3. Corporate decisions [http://www.kaspersky.ru/corporatesolutions]
4. The review of programs for struggle against a spam [http://www.klerk.ru/soft/articles/3853]
The author: Челябэнергопроект
Comments of experts of Челябэнергопроект: