To be able to work with un-ACK’d data, we need to make sure we deal with possible evasions properly. The problem, as extensively documented by Judy Novak and Steven Sturges, is that in TCP streams there can be overlapping packets. Those are being dealt with differently based on the receiving OS. If we would need to account for overlaps in the application layer, we would have to be able to tell the HTTP parser for example: “sorry, that last data is wrong, please revert and use the new packet instead”. A nightmare.

The solution I opted for was to not care about destination OS’ for overlaps and such. The approach is fairly simple: once we have accepted a segment, thats what it’s going to be. This means that if we receive a segment later that (partially) overlaps and has different data, it’s data portion will simply be overwritten to be the same as the original segment. This way, the IPS and not an obscure mix of the sender (attacker?) and destination OS, determines the data the destination will see.

Of course the approach comes with some drawbacks. First, we need to keep segments in memory for a longer period of time. This causes significantly higher memory usage. Secondly, if we rewrite a packet, it needs to be reinjected on the wire. As we modified the packet payload a checksum recalculation is required.

In Suricata’s design the application layer parsers, such as our HTTP parser, run on top of the reassembly engine. After the reassembly engine and the app layer parsers are updated, the packet with the associated stream and app layer state is passed on to the detection engine. In the case where we work with ACK’d data, an ACK packet in the opposite direction triggers the reassembly process. If we detect based on that, and decide we need to drop, all we can do is drop the ACK packet as the actual data segment(s) have already passed. This is not good enough in many cases.

In the new code the data segment itself triggers the reassembly process. In this case, if the detection engine decides a drop is required, the packet containing the data itself can be dropped, not just the ACK. The reason we’re not taking the same approach in IDS mode is that we wouldn’t be able to properly deal with the said evasion/overlap issues. The IPS can exactly control what packets pass Suricata. The IDS, being passive, can not.

You can try this code by checking out the current git master. In the suricata.yaml that lives in our git tree you’ll find a new option in the stream config, “stream.inline”. If you enable this, the code as explained above is activated.

Feedback is very welcome!

The main focus of this release was fixing the TCP stream engine. Judy Novak found a number of ways to evade detection. See her blog post describing the issues.

The biggest other change is the addition of a new application layer module. The SSH parser parses SSH sessions and stops detection/inspection of the stream after the encrypted part of the session has started. So this is mainly a module focused on reducing the number of packets that need inspection, just like the SSL and TLS modules.

As a bonus though, we introduced two rule keywords that match on the parsed SSH parameters:

ssh.protoversion will match against the ssh protocol version. I’ll give some examples.

ssh.protoversion:2.0

This will match on 2.0 exactly.

ssh.protoversion:2_compat

This will match on 2, but also 1.99 and other versions compatible to “2″.

ssh.protoversion:1.

The last example will match on all versions starting with “1.”, so 1.6, 1.7, etc.

ssh.softwareversion will match on the software version identifier. An example:

ssh.softwareversion:PuTTY

This will match only on session using the PuTTY SSH client.

Other changes include better HTTP accuracy, better IPS functionality.

For the next release we will focus on further improving overall detection accuracy, improving inline mode further, improving performance and specifically improving CUDA performance. As always, we welcome any feedback. Or if you are interested in helping out, please contact us!

Update: added a link to Judy Novak’s blog post on the TCP evasions.

Starting with Suricata 0.9.0 the engine can run as an unprivileged user. For this create a new user called “suricata”.

useradd --no-create-home --shell /bin/false --user-group --comment “Suricata IDP account” suricata

This command will create a user and group called “suricata”. It will be unable to login as the shell is set to /bin/false.

The next thing to do is creating a configuration directory. Create /etc/suricata/ and copy the suricata.yaml example config into it. The example configuration can be found in the source archive you used to build Suricata:

mkdir /etc/suricata
cp /path/to/suricata-0.9.0/suricata.yaml /etc/suricata/
cp /path/to/suricata-0.9.0/classification.config /etc/suricata/

Next, create the log directory.

mkdir /var/log/suricata

The log directory needs to be writable for the user and group “suricata”, so change the ownership:

chown suricata:suricata /var/log/suricata

The last step I’ll be describing here is retrieving an initial ruleset. The 2 main rulesets you can use are Emerging Threats (ET) and Sourcefire’s VRT ruleset. Since putting VRT to use is a little bit more complicated I’ll be focussing on ET here.

First, download the emerging rules:

wget http://www.emergingthreats.net/rules/emerging.rules.tar.gz

Go to /etc/suricata/ and extract the rules archive:

cd /etc/suricata/
tar xzvf /path/to/emerging.rules.tar.gz

There is a lot more to rules, such as tuning and staying updated, but thats beyond the scope of this post.

Suricata is now ready to be started:

suricata -c /etc/suricata/suricata.yaml -i eth0 --user suricata --group suricata

If all is setup properly, Suricata will tell you it is now running:

[2087] 9/5/2010 — 18:17:47 – (tm-threads.c:1362) (TmThreadWaitOnThreadInit) — all 8 packet processing threads, 3 management threads initialized, engine started.

There are 3 log files in /var/log/suricata that will be interesting to monitor:

- stats.log: displays statistics on packets, tcp sessions etc.
- fast.log: a alerts log similar to Snort’s fast log.
- http.log: displays HTTP requests in a Apache style format.

This should get you going. There is a lot more to deploying Suricata that I plan to blog on later.

Here is how to compile Suricata 0.9.0 in inline mode on Ubuntu Lucid 10.04.

First, make sure you have the “universe” repository enabled. Go to the System menu, Administration, Software Sources. There enable “Community-maintained Open Source Software (universe)”. If you’re not running a gui, edit /etc/apt/sources.list and enable the universe repository there. Don’t forget doing an “apt-get update”.

Install the following packages needed to build Suricata: libpcre3-dev libpcap-dev libyaml-dev zlib1g-dev libnfnetlink-dev libnetfilter-queue-dev libnet1-dev libcap-ng-dev.

apt-get install libpcre3-dev libpcap-dev libyaml-dev zlib1g-dev libnfnetlink-dev libnetfilter-queue-dev libnet1-dev libcap-ng-dev

Download Suricata 0.9.0 here

Extract the suricata-0.9.0.tar.gz file as follows:

tar xzvf suricata-0.9.0.tar.gz

Enter the extracted directory suricata-0.9.0.

Run “./configure –enable-nfqueue”
If “./configure –enable-nfqueue” was succesful, run “make”
If “make” was succesful, run “sudo make install”
Except for Suricata itself, the build process installed “libhtp”. For that to work properly, run “ldconfig”.

Run “suricata -V” and it should report version 0.9.0.

To use Suricata in inline mode, pass -q <queue id> to the command line. Example:

suricata -c /etc/suricata/suricata.yaml -q 0

Most notable changes are the following new features:

Go get the release here: http://www.openinfosecfoundation.org/download/suricata-0.9.0.tar.gz

First, make sure you have the “universe” repository enabled. Go to the System menu, Administration, Software Sources. There enable “Community-maintained Open Source Software (universe)”. If you’re not running a gui, edit /etc/apt/sources.list and enable the universe repository there. Don’t forget doing an “apt-get update”.

Install the following packages needed to build Suricata: libpcre3-dev libpcap-dev libyaml-dev zlib1g-dev libnfnetlink-dev libnetfilter-queue-dev libnet1-dev.

apt-get install libpcre3-dev libpcap-dev libyaml-dev zlib1g-dev libnfnetlink-dev libnetfilter-queue-dev libnet1-dev

Download Suricata 0.8.2 here

Extract the suricata-0.8.2.tar.gz file as follows:

tar xzvf suricata-0.8.2.tar.gz

Enter the extracted directory suricata-0.8.2.

Run “./configure –enable-nfqueue”
If “./configure –enable-nfqueue” was succesful, run “make”
If “make” was succesful, run “sudo make install”
Except for Suricata itself, the build process installed “libhtp”. For that to work properly, run “ldconfig”.

Run “suricata -V” and it should report version 0.8.2.

To use Suricata in inline mode, pass -q <queue id> to the command line. Example:

suricata -c /etc/suricata/suricata.yaml -q 0

One of the tools we used to help improve the engine is a fuzzer created by Will Metcalf, our QA lead. In short, the script takes a pcap file, runs it through editcap (part of wireshark) altering a number of random bytes, then feeds the altered pcap file to Suricata. This resulted in many interesting corner cases. Naturally the script makes sure you don’t forget to enable “ulimit -c unlimited” and such More on that script can be found on Will’s blog node5.

For the next period we’ll be working on resolving a number of open issues. There are still a number of improvements we need to make to the relation between our app layer decoding modules and our detection engine. Next to this we’re still missing support for a number of rule keywords, such as asn1 and http_headers. We’re also working on getting our CUDA accelaration into a more usable shape. This release improved it slightly, by making it work on x86_64, but it’s still not useful in production environments.

So as usual, enough to do! Meanwhile, we’re looking for feedback on our release!

As it’s lead developer I’m very much interested in getting feedback, bug reports and such. We run our ticket system in a redmine install at https://redmine.openinfosecfoundation.org/ If you have any feedback, please register an account and let us know what you think.

If you’re running into any issue, reconfigure and recompile the engine with –enable-unittests and –enable-debug and send us the output of “suricata -u” this will run all the unittests (1191 currently). If everything is set up properly, they should all pass. If not, please start bugging us!

Happy new year everyone!

For the last two years I’ve been doing work as a contractor in the (open source) security field. My experience is mostly in coding in C and Perl, primarily on Snort and Snort_inline. Recently I created the (Perl language) SidReporter program for Emerging Threats. Areas I worked in: IPv6 IDS/IPS coding, signature writing, Web Application Firewalls, threading, bandwidth accounting, and more…

Checkout my LinkedIn profile for more info. My resume is available on request.

If you have some work or know someone that does, please let me know!

The last year I’ve been working on a prototype of the engine we’re building as well. It’s private for now as the foundation licensing terms & conditions haven’t been determined yet. I’m writing it mostly to learn. While I’ve been working as a developer on the Snort_inline project for a number of years already and as a contractor on several Snort related projects, I never learned so much about IDS/IPS technology as I’m doing now. The prototype may or may not be used (partly) for the engine once we got our feature list complete. We’ll see about that when the time is there. I plan to blog more about this codebase in the new year.

In 2008 we had our first brainstorming session, and to us it was very successful. In 2009 we’re hoping to do a few more. Stay tuned for the dates and places. I hope we can continue our feature discussions in the new year and give the foundation further shape. And don’t forget to suggest us a name for the engine… “OISF engine” just doesn’t sound cool enough!