64bit

2008-08-29T13:21:38+00:00

Current crm114 in mainline has (serious) issues on 64bit systems, but these have been (mostly?) addressed in GerH's version. So here's one users' working recipe, freshly from the ML. Keep in mind though that CSSfiles are not, in general, 32<->64 bits portable, and also that GerH's may be incompatible with mainline's. Check the docs and the ML before doing something dangerous.

bit_entropy

2007-04-09T21:23:49+00:00

Entropic Classification Basics This is an implementation of an entropic bitwise lattice Markov classifier. This is very roughly based on some ideas in the DMC (Dynamic Markov Compressor) data compression technique of Gordon Cormack et al and inspired by Andrej Bratko's use of it in spam filtering ( Gordon originally proposed DMC for compression and then Andrej used it for spam filtering. Matthew Young-Lai then proposed a technique for merging nodes which is limited to re-merging…

books

2007-04-10T22:40:12+00:00

* Jonathan Zdziarski (who is also the author of the excellent DSPAM spam filter ) has written an excellent book: ENDING SPAM. This is a paper book; but it's one of the better books on the details of spam and spam filtering I have seen. It even gets many of the CRM114 internals close to right. :-). Here's another review.: Ending Spam . You can get Jon's book via O'Reilly or No Starch Press

changelog

2007-04-19T14:03:07+00:00

CRM114 - the Controllable Regex Mutilator ================================================================= 20070406-BlameSpamConf Latest release in src/ and CVS - mostly bugfixes again - some details will be added later on ...20070217-BlameBaltar This is a bugfix version. Fixed: mail*.crm: :datadir: is now gone; mailreaver.crm: You can now use start/len on INPUT on stdin. Arithmetic now respects >=, <=, and !=. Hyperspace has been de-normalized Entropic classification is now somewhat smart…

coding_standards

2007-08-23T13:19:24+00:00

Coding Standards for CRM114 Please try to use something vaguely like these coding standards for patchse and for CRM114 programs. These aren't absolute rules, but there's a reason for them. Yes, some of the reasons are arbitrary but we at least attempt to rationalize things.

consultants

2007-09-26T11:16:57+00:00

CRM114 Consultants If you have a CRM114-related task that can't be solved easily on the mailing list, and you want to hire someone to “take care of it” for you, the following people have claimed to have appropriate knowledge and talents. (Note- this is not an endorsement by anybody of these consultants. Posting here is by request, not by judgement -- Bill Yerazunis)

contacts

2007-04-10T17:11:21+00:00

Contacts, Bugs and Comments? * First thing to do: read the documentation, including the FAQ and the HOWTO. If that solves your problem, great. * Second thing you should do with a bug is to check the mailing lists archive. If you have a question that the archive doesn't cover, then the best action is to first join the crm114-general mailing lists and (second) ask there. The mailing lists allow only subscribers to post, so you need to subscribe first. You don't need to wait for your sub…

coolthings

2007-10-03T07:21:55+00:00

Other people are using CRM114's language to create useful things. Check 'em out here! * X86-64 (all OSes) / MS-Windows Ger Hobbelt has been creating extremely good ports of CRM114 bleeding edge code for X64 platforms (both Linux and Windows), including some bugfixes that haven't made it back to mainline or even bleeding edge yet. You can pull down full kits or patch sets (for both bleeding edge and “release” versions) from Ger's site here:

daily_email_statistics

2008-07-02T18:59:39+00:00

Date: Sat, 3 May 2008 16:22:19 +0200 From: Uwe Scholz Subject: daily email statistics Message-ID: <20080503142219.GA4791@uwe-desktop> Hello Paolo, today I've made some changes to my daily mail summary script. Now it is possible to chose, if the summary of mails is sorted by date of incomming or by spam possibility. For this purpose, I had to include two new files (scan.spam_dat and scan.spam_pos) and delete an old (scan.spam). I'm still using 20041231.BlameSanAndreas (TRE 0…

daily_email_statistics_script

2008-02-15T02:57:08+00:00

This is a script you can use to monitor the performance of your mailreaver spam filter. It relies on the reavercache to count the different classes of email. To make it work you will need to change the CRMDIR to point to the reavercache directory. If you call it with no command line paramaters it will give you the stats for yesterday. If you call it with -i it will give you the stats for the last 30 days.

default_sidebar_left

2007-08-23T13:12:32+00:00

* About * News * Quick Look * Documents * Books * changelog * Download * Resources * CoolThings * Mailing Lists * Contacts * HOWTOs * Submitting Patches * Coding Standards * Licensing * Consultants

documents

2008-02-14T20:53:43+00:00

* Spam Conferences * CRM114 papers from the 2006 TREC SPAM conference * TREC 2006 papers (look esp. for spam-related) * Filter Taxonomy * Seven Hypothesis about Spam Filtering (local copy (DjVu) * OSBF-Lua - A Text Classification Module for Lua - The Importance of the Training Method (local copy, DjVu) * KNN/Hyperspace slides * KNN/Hyperspace Filters * results with Hyperspace * notes paper

download

2009-12-21T19:49:41+00:00

CRM114 Newest Slightly Unstable Mainline (online updates) * You can use wget to pull down the (possibly) slightly unstable latest mainline version by typing wget -m -np http://crm114.sourceforge.net/src/ in a directory where you want the unstable mainline to be downloaded to. Note that this is a smart download and will only pull down new or modified files. CAUTION - for this wget pulldown, you *must* download TRE (Ville Laurikari's enhanced regex library) and install it ( using the ./con…

how_to_train

2008-09-24T00:10:38+00:00

stub - TODO

how_to_upgrade

2007-08-20T06:14:07+00:00

These are the steps to take if you already have crm114 installed and wish to upgrade. * Backup your existing installation * Download the source for the version you wish to upgrade to * compile, build and install * copy *.crm files * check mailfilter.cf is up to date * test

howtos

2008-08-29T13:05:53+00:00

Here you will find a list of HOWTO articles relating to crm114. * first, did you check Quick Look? * (tentative) Thresholds-How(l)-To * How to upgrade an existing installation of crm114. * Daily email statistics (with links to scripts) * Daily email statistics script. * How To Train - i.e. (almost) all about TOE, TUNE, TONE, TONER, SSTTT ... * 64bit - How to make (current) crm114 (and yourself!) happy on 64bit systems

licensing

2007-08-23T12:38:48+00:00

Licensing CRM114 is licensed with the Free Software Foundation's GPL version 2 (and the license is included in the source kits). To summarize and slightly oversimplify: 1) Yes, you can use CRM114 internally in your organization. You don't need to ask permission from anybody. Just do it.

mailing_lists

2007-04-10T16:58:57+00:00

Mailing Lists? Sure! Choose from these three, er, four: Go to the signup/archive form for CRM114 Announcements (very low volume list). This is what you want if you just need to keep track of releases. Go to the signup/archive form for CRM114-General list. This is where to sign up to ask about CRM114 specific issues, (mis)features, bugs, get help, etc.

news

2008-07-02T19:38:30+00:00

these news are lazily (un)maintained, please check the README in the sources. check the current version here. See Download for how to get latest version. ---------- August 10, 2007 - 20070810-BlameTheSegfault (562532 bytes) - bugfixes. June 22, 2007 - 20070731-BlameTheInterns released - new classifiers: CLUMP and SVM - and bugfixes again (of course).

quick_look

2008-02-15T08:46:06+00:00

Grab A Quick Look at ... * the CRM114 mailfilter How-To for setting up anti-spam mailfiltering * the FAQ (Frequently Asked Questions) for the CRM114 controllable regex mutilator. * the quick reference card for the CRM114 language (a language specifically designed to write text filters in). * (tentative) Thresholds-How(l)-To * the Language Introduction for the CRM114 controllable regex mutilator. * how the sparse binary polynomial hash phrase-based matcher works internally. * …

resources

2007-04-10T21:55:03+00:00

The following extra packages may be required on your platform, either to satisfy dependencies or just to complete the filter chain used in some test setup described in papers or on mailing lists. Besides standard packages expected to be found on any Linux/Unix system, that is.

submitting_patches

2007-08-23T19:42:07+00:00

Submitting Patches The best way to submit a patch is to do it via the email list; the best way to submit that is (assuming the patch is short) is with a “unified” diff that shows +/- changes as well as context. You can generate this with the following command:

submitting_patches_and_licensing

2007-08-20T18:32:50+00:00

thresholds-how_l_-to

2007-10-30T06:39:11+00:00

[This how-to is a somewhat edited version of how to choose the (initial) thresholds? ] How do you set the thresholds for the usure gap between spam and good? Good point. A question asked more than once on the ML. By trial and error Good answer, eh ;)

wikihome

2007-08-23T13:23:13+00:00

CRM114 is a system to examine incoming e-mail, system log streams, data files or other data streams, and to sort, filter, or alter the incoming files or data streams according to the user's wildest desires. Criteria for categorization of data can be via a host of methods, including regexes, approximate regexes, a Hidden Markov Model, Bayesian Chain Rule Orthogonal Sparse Bigrams, Winnow, Correlation, KNN/Hyperspace, Bit Entropy, CLUMP, SVM, Neural Networks ( or by other means- it's all …