There are still a number of open issues with regard to accuracy. Those will be addressed in 1.0.2, scheduled for late August, early September. We’re working on improving CUDA, stream engine improvements and inline mode as well. Keep an eye on redmine for the open and fixed issues.

I’ll be taking some time off to recharge a bit, the last couple of months have been exhausting. Things are very exciting, so I can hardly wait to get back to improve our little Meerkat! Cheers!

At this point I really don’t care much. What the Suricata development by the OISF has shown in my opinion is that we’ve managed to create a very promising new Open Source project out here. In little over a year, funded for about $600k by the US government and with heavy (and growing) industry support, we’ve produced a new IDS/IPS engine mostly compatible with Snort but build on a all new code base an incorporating some very interesting fresh ideas. We’re already seeing a community form around our project with a lot of support from that new community.

So about this performance fuzz. Who to believe? Is Suricata faster than Snort? Yes, no, ehhh, depends on how you look at it. Is Suricata faster than Snort on a single core cycle for cycle, tick for tick? No. It’s pretty clear we aren’t, I didn’t expect us to be either. But we scale. We’ve had reports of running on a 32 core box and scaling to use all cores. There Suricata is much faster. Like Martin Roesch wrote on the VRT blog one can set up Snort on a box to one have instance of Snort per core (or multiple per core). This is in fact the way many appliance builders get to high speeds with it. While this may be feasible for appliance builders, admins we talked to that run their own IDS/IPS think it’s a management nightmare.

As we’re a new project with a fresh codebase, there is going to be a lot of low hanging fruit in performance optimizations. I’ll give an example here. On a test pcap, with a reduced ruleset (about 10k rules), Suricata took about 400s to inspect. Then with a bigger ruleset (about 14k rules), it suddenly took 1600s! After a little bit of cache profiling it turned out that the part of the engine where the address part of a signature was inspected was horribly cache inefficient. In less than an afternoon I rewrote it to be more efficient. Result, the same test now completes in under 600s. This code is in the current git master and will be in 1.0.1.

My point here being that there will be lots of room for optimizations, and not just minor stuff. So far we’ve mostly focused on being accurate (we still have work to do here) and having the algorithms be correct. Hardly any tuning has been done. In our last OISF meeting we’ve gotten a few very interesting help offers for serious performance testing and tuning on some really big boxes, state of the art CUDA hardware, 10GBit labs, etc. So I expect a lot of progress in the months to follow.

It’s clear that we have work to do. What I’m really excited about is how fast that work is progressing, how much help we’re getting both from our brand new community and the industry, and the openness of our development process.

On a final note, during the development of this project we’ve found a lot of bugs and issues in other tools. Will Metcalf, who runs our QA, has been reporting many issues in Snort and VRT sigs to Sourcefire, in Emerging Threats sigs to the ET community. We’ve found bugs in other tools as well, for example in a neat library called libcap-ng. So everyone benefits from our work!

I think it’s a good release too. Is it perfect? No, we have a list known issues that we will continue to work on. So expect a 1.0.1 and maybe more maintenance releases in the following weeks.

On July 16th we will be having a public meeting in San Francisco to discuss the next major development milestone. Everyone is welcome to join us there to bring in new ideas. If you can’t make it, no sweat, you can also send ideas to us privately or discuss them on our mailing lists.

The code analysis is pretty nice: it gives statistics on code base size, growth, comment ratio, languages used, etc. Per developer it tracks quite a few stats as well.

It also does a estimate of the cost of a project. For the Suricata project it currently estimates cost of 2.1 million USD. Actual cost are significantly less than that, less than half of that. So either we are severely underpaid or the calculation is off quite a bit

The per developer code statistics show that I’ve “touched” 131k lines of code out of 148k which confirms what I already knew: I need some vacation…

Anyway, check it out. Vuurmuur is on there, as are Snort and ModSecurity.

Oh by the way, Suricata 1.0 coming out tomorrow!

Starting with Suricata 0.9.0 the engine can run as an unprivileged user. For this create a new user called “suricata”.

useradd --no-create-home --shell /bin/false --user-group --comment “Suricata IDP account” suricata

This command will create a user and group called “suricata”. It will be unable to login as the shell is set to /bin/false.

The next thing to do is creating a configuration directory. Create /etc/suricata/ and copy the suricata.yaml example config into it. The example configuration can be found in the source archive you used to build Suricata:

mkdir /etc/suricata
cp /path/to/suricata-0.9.0/suricata.yaml /etc/suricata/
cp /path/to/suricata-0.9.0/classification.config /etc/suricata/

Next, create the log directory.

mkdir /var/log/suricata

The log directory needs to be writable for the user and group “suricata”, so change the ownership:

chown suricata:suricata /var/log/suricata

The last step I’ll be describing here is retrieving an initial ruleset. The 2 main rulesets you can use are Emerging Threats (ET) and Sourcefire’s VRT ruleset. Since putting VRT to use is a little bit more complicated I’ll be focussing on ET here.

First, download the emerging rules:

wget http://www.emergingthreats.net/rules/emerging.rules.tar.gz

Go to /etc/suricata/ and extract the rules archive:

cd /etc/suricata/
tar xzvf /path/to/emerging.rules.tar.gz

There is a lot more to rules, such as tuning and staying updated, but thats beyond the scope of this post.

Suricata is now ready to be started:

suricata -c /etc/suricata/suricata.yaml -i eth0 --user suricata --group suricata

If all is setup properly, Suricata will tell you it is now running:

[2087] 9/5/2010 — 18:17:47 – (tm-threads.c:1362) (TmThreadWaitOnThreadInit) — all 8 packet processing threads, 3 management threads initialized, engine started.

There are 3 log files in /var/log/suricata that will be interesting to monitor:

- stats.log: displays statistics on packets, tcp sessions etc.
- fast.log: a alerts log similar to Snort’s fast log.
- http.log: displays HTTP requests in a Apache style format.

This should get you going. There is a lot more to deploying Suricata that I plan to blog on later.

Here is how to compile Suricata 0.9.0 in inline mode on Ubuntu Lucid 10.04.

First, make sure you have the “universe” repository enabled. Go to the System menu, Administration, Software Sources. There enable “Community-maintained Open Source Software (universe)”. If you’re not running a gui, edit /etc/apt/sources.list and enable the universe repository there. Don’t forget doing an “apt-get update”.

Install the following packages needed to build Suricata: libpcre3-dev libpcap-dev libyaml-dev zlib1g-dev libnfnetlink-dev libnetfilter-queue-dev libnet1-dev libcap-ng-dev.

apt-get install libpcre3-dev libpcap-dev libyaml-dev zlib1g-dev libnfnetlink-dev libnetfilter-queue-dev libnet1-dev libcap-ng-dev

Download Suricata 0.9.0 here

Extract the suricata-0.9.0.tar.gz file as follows:

tar xzvf suricata-0.9.0.tar.gz

Enter the extracted directory suricata-0.9.0.

Run “./configure –enable-nfqueue”
If “./configure –enable-nfqueue” was succesful, run “make”
If “make” was succesful, run “sudo make install”
Except for Suricata itself, the build process installed “libhtp”. For that to work properly, run “ldconfig”.

Run “suricata -V” and it should report version 0.9.0.

To use Suricata in inline mode, pass -q <queue id> to the command line. Example:

suricata -c /etc/suricata/suricata.yaml -q 0

Most notable changes are the following new features:

Go get the release here: http://www.openinfosecfoundation.org/download/suricata-0.9.0.tar.gz

First, make sure you have the “universe” repository enabled. Go to the System menu, Administration, Software Sources. There enable “Community-maintained Open Source Software (universe)”. If you’re not running a gui, edit /etc/apt/sources.list and enable the universe repository there. Don’t forget doing an “apt-get update”.

Install the following packages needed to build Suricata: libpcre3-dev libpcap-dev libyaml-dev zlib1g-dev libnfnetlink-dev libnetfilter-queue-dev libnet1-dev.

apt-get install libpcre3-dev libpcap-dev libyaml-dev zlib1g-dev libnfnetlink-dev libnetfilter-queue-dev libnet1-dev

Download Suricata 0.8.2 here

Extract the suricata-0.8.2.tar.gz file as follows:

tar xzvf suricata-0.8.2.tar.gz

Enter the extracted directory suricata-0.8.2.

Run “./configure –enable-nfqueue”
If “./configure –enable-nfqueue” was succesful, run “make”
If “make” was succesful, run “sudo make install”
Except for Suricata itself, the build process installed “libhtp”. For that to work properly, run “ldconfig”.

Run “suricata -V” and it should report version 0.8.2.

To use Suricata in inline mode, pass -q <queue id> to the command line. Example:

suricata -c /etc/suricata/suricata.yaml -q 0

One of the tools we used to help improve the engine is a fuzzer created by Will Metcalf, our QA lead. In short, the script takes a pcap file, runs it through editcap (part of wireshark) altering a number of random bytes, then feeds the altered pcap file to Suricata. This resulted in many interesting corner cases. Naturally the script makes sure you don’t forget to enable “ulimit -c unlimited” and such More on that script can be found on Will’s blog node5.

For the next period we’ll be working on resolving a number of open issues. There are still a number of improvements we need to make to the relation between our app layer decoding modules and our detection engine. Next to this we’re still missing support for a number of rule keywords, such as asn1 and http_headers. We’re also working on getting our CUDA accelaration into a more usable shape. This release improved it slightly, by making it work on x86_64, but it’s still not useful in production environments.

So as usual, enough to do! Meanwhile, we’re looking for feedback on our release!

What we have so far is a implementation our 2 gram SBNDM pattern matcher algorithm in CUDA. The detection thread(s) currently send packets one by one to a central dispatcher thread that controls the GPU. This setup is far from ideal performance wise, but our first goal was to get it working at all. Currently on my desktop CUDA actually slows things down.

In the next weeks and months we plan to do some redesigning of the CUDA implementation and it’s integration into the engine. We plan to send the packets in batches to the dispatcher thread right after the decoders have determined what the payload portion of a packet is. The (separate) detection thread(s) can then process the results of the GPU when they get to a packet. By using the CUDA scanning async like this we hope that we can reduce the costs of the transfer of packets from and to the card.

Currently the code in the tree can be activated by passing the “–enable-cuda” option to ./configure. Next, in the configuration file enable the cuda pattern matcher by setting the “mpm-algo” option to “b2g_cuda”. As a first test, run the CUDA unittests (assuming you enabled the building of the unittests too) by using “suricata -uUCuda”. Please note that currently running all unittests will fail if CUDA is enabled.

The code is only tested on 32bit Linux at the moment. There are some issues with 64bit that we’re resolving right now. We’re expecting to be continuously updating this code, so be sure to work with the most current version of the git repo all the time!

Let us know your experiences!