As you may still be interested in the errors they are now available through the rule language. In rules/http-events.rules and rules/smtp-events.rules rules for all possible events/errors can be found.

Example:
app-layer-event:http.missing_host_header;

This will match on HTTP/1.1 requests without a Host header.

Some of these rules might be noisy (they are not in my local network), but rather than disabling them I’d suggest suppressing then. The reason is that for each time they hit a flowint will be incremented:

flowint:http.anomaly.count,+,1;

This will allow you to get alerts on streams with high anomaly counts:

alert http any any -> any any (msg:"LOCAL really poor HTTP session"; flowint:http.anomaly.count,>,5; sid:123; rev:1;)

This will give you an alert if there have been more than 5 anomalies detected.

Blog spammers, malware and other unwanted HTTP users often use HTTP with all kinds of issues, so this may be a helpful tool in detecting those.

The full announcement for the 1.1.1 release is here.

Naturally, the issue has also been fixed in the 1.2 development branch.

So what is this all about? Simply put, we can now extract files from HTTP streams in Suricata. Both uploads and downloads. Fully controlled by the rule language. But thats not all. I’ve added a touch of magic. By utilizing libmagic (this powers the “file” command), we know the file type of files as well. Lots of interesting stuff that can be done there.

Rule keywords

Four new rule keywords were added: filename, fileext, filemagic and filestore.

Filename and fileext are pretty trivial: match on the full name or file extension of a file.

alert http any any -> any any (filename:”secret.xls”;)
alert http any any -> any any (fileext:”pdf”;)

More interesting is the filemagic keyword. It runs on the magic output of inspecting the (start of) a file. This value is for example:

GIF image data, version 89a, 1 x 1
PE32 executable for MS Windows (GUI) Intel 80386 32-bit
HTML document text
Macromedia Flash data (compressed), version 9
MS Windows icon resource – 2 icons, 16×16, 256-colors
PNG image data, 70 x 53, 8-bit/color RGBA, non-interlaced
JPEG image data, JFIF standard 1.01
PDF document, version 1.6

So how the filemagic keyword allows you to match on this is pretty simple:

alert http any any -> any any (filemagic:”PDF document”;)
alert http any any -> any any (filemagic:”PDF document, version 1.6″;)

Pretty cool, eh? You can match both very specifically and loosely. For example:

alert http any any -> any any (filemagic:”executable for MS Windows”;)

Will match on (among others) these types:

PE32 executable for MS Windows (DLL) (GUI) Intel 80386 32-bit
PE32 executable for MS Windows (GUI) Intel 80386 32-bit
PE32+ executable for MS Windows (GUI) Mono/.Net assembly

Finally there is the filestore keyword. It is the simplest of all: if the rule matches, the files will be written to disk.

Naturally you can combine the file keywords with the regular HTTP keywords, limiting to POST’s for example:

alert http $EXTERNAL_NET any -> $HOME_NET any (msg:”pdf upload claimed, but not pdf”; flow:established,to_server; content:”POST”; http_method; fileext:”pdf”; filemagic:!”PDF document”; filestore; sid:1; rev:1;)

This will alert on and store all files that are uploaded using a POST request that have a filename extension of pdf, but the actual file is not pdf.

Storage

The storage to disk is handled by a new output module called “file”. It’s config looks like this:

enabled: yes # set to yes to enable
log-dir: files # directory to store the files
force-magic: no # force logging magic on all stored files


It needs to be enabled for file storing to work.

The files are stored to disk as “file.1″, “file.2″, etc. For each of the files a meta file is created containing the flow information, file name, size, etc. Example:

TIME: 01/27/2010-17:41:11.579196
PCAP PKT NUM: 2847035
SRC IP: 68.142.93.214
DST IP: 10.7.185.57
PROTO: 6
SRC PORT: 80
DST PORT: 56207
FILENAME: /msdownload/update/software/defu/2010/01/mpas-fe_7af9217bac55e4a6f71c989231e424a9e3d9055b.exe
MAGIC: PE32+ executable for MS Windows (GUI) Mono/.Net assembly
STATE: CLOSED
SIZE: 5204

Configuration

The file extraction is for HTTP only currently, and works on top of our HTTP parser. As the HTTP parser runs on top of the stream reassembly engine, configuration parameters of both these parts of Suricata affect handling of files.

The stream engine option “stream.reassembly.depth” (default 1 Mb) controls the depth into a stream in which we look. Set to 0 for no limit.
The libhtp options request-body-limit and response-body-limit control how far into a HTTP request or response body we look. Again set to 0 for no limit. This can be controlled per HTTP server.

Performance

The file handling is fully streaming, so it’s very efficient. Nonetheless there will be an overhead for the extra parsing, book keeping, writing to disk, etc. Memory requirements appear to be limited as well. Suricata shouldn’t keep more than a few kb per flow in memory.

Limitations

Lack of limits is a limitation. For file storage no limits have been implemented yet. So it’s easy to clutter your disk up with files. Example: 118Gb enterprise pcap storing just JPG’s extracted 400.000 files. Better use a separate partition if you’re on a life link.

Future work

Apart from stabilizing this code and performance optimizing it, the next step will be SMTP file extraction. Possibly other protocols, although nothing is set in stone there yet.

The long development cycles should be something of the past. At our last brainstorm session, at RAID 2011, we decided to change our release policy. The aim of this policy is to do time based releases, roughly a “stable” every 2 months and a beta every other month. This way we’ll be making it much easier for users to stay current without have to run our “git master”.

Looking forward, we’ve started work on the 1.2 release, which should happen in about 2 months. Focus will be on performance. We’re planning to do a significant refactoring of our pattern matching engine, which should lead both to better performance and improved accuracy. Next to this, we’ll be finally adding the “file_data” keyword along with HTTP file carving — extracting files from HTTP requests. I am personally very excited about this.

We’re starting to see more and more community involvement. Not just on the user side, but also on the development side. As seen on the oisf-devel mailinglist, a large SSL/TLS patch set was contributed by Pierre Chifflier. This will make it’s way into the 1.2 release as well. Smaller contributions were accepted on PF_RING code and the HTTP code. I am very grateful for the contributions.

Eric Leblond and I will be doing a talk next week at DeepSec on Suricata. If you are able to, please come meet us!

In the Emerging Threats community the following if often heard: “PCRE is evil”. With this people refer to signatures that use “pure” PCRE matches, meaning without anchoring it to a content pattern match.

A while ago Will Metcalf initiated work to get Suricata to support a new PCRE feature by Herczeg Zoltán: SLJIT. Since then, support for this has found it’s way into the official PCRE release, currently at version 8.20-RC3.

I decided to run a quick benchmark to see how much difference there would be. The results are quite amazing. In my test I used an older Intel Core2 E6600 2.4Ghz on Ubuntu 10.10, a 416MB pcap full of badness (sandnet traffic) and a slightly older ruleset of 11.972 signatures.

The results:

suricata, OS default pcre (8.02)...................: 78s
suricata, pcre-8.20-RC3 (no jit), -O2..............: 80s
suricata, pcre-8.20-RC3 (no jit), -O3 -march=native: 72s
suricata, pcre-8.20-RC3 (jit), -O2.................: 53s

I played some more with GCC 4.6.1 and various optimization levels, but this was the best result so far. Quite surprising because in the past I saw some improvements from using the newer GCC over the OS default of 4.4.5.

Want to try the new PCRE without messing up your system?
./configure --prefix=/opt/pcre-8.20-RC3/ --enable-jit --enable-utf8 --enable-unicode-properties
make
sudo make install

Then recompile Suricata as well:
./configure --enable-pcre-jit --with-libpcre-libraries=/opt/pcre-8.20-RC3/lib/ --with-libpcre-includes=/opt/pcre-8.20-RC3/include/
make
sudo make install

You’ll need the Suricata code from git to take advantage of this.

Please give it a try, it’s free performance!

I’ve seen all talks and some were very interesting. However, being a Suricata IDS developer, I was not just interested in research for the hell of it, but I was actively scouting for ideas we could implement into Suricata. In this respect the conference was highly disappointing. Although with some of the talks I thought the idea was applicable in general security, like Erik Bosmans high speed memory tainting detection, I found nothing like that for NIDS.

Most inspiring part of the conference was spending an evening with Seth Hall, one of the Bro IDS engineers. Bro has a very different approach to inspecting the network than Suricata. Actually, I should say Suricata does it differently as Bro has been around much longer than Suricata. The conversation was all about sharing of ideas and experiences, and finding common grounds for actual cooperation.

A couple of notes from that conversation. First, Bro supports Unified2/Barnyard2 now, as input (so actually Barnyard2 can output to Bro). This means it can extend it’s analysis to include Suricata generated events. Second, we might try to have Suricata and Bro work together, where Suricata would be controlled by Brocolli. This way Bro could benefit from Suricata’s high speed signature matching engine, functionality Bro doesn’t have, and Suricata could benefit from Bro’s higher level understanding of the network. Finally, Bro’s binpack effort to define protocol parsers in a higher level language that can then be compiled into native code looks interesting as well. It would probably take quite a bit of changes to get this all going, but it might just be worth it.

Then there was the panel at the conference with Martin Roesch, Seth Hall and myself. A lot of people expected fireworks, but no such thing happened. Everyone was polite, respectful and friendly. It never really turned into a real discussion though, it was more a Q&A with the audience. Dominique Karg blogged about the panel here.

It was good to talk to Martin Roesch. The OISF – Sourcefire relation has definitely not started well, so it was good to have normal conversations and such. I offered Marty to work together, especially on SCADA detection. As was announced earlier, OISF will maintain the Digital Bond Quickdraw SCADA parsers and keywords, not only for Suricata, but also for Snort. Hopefully we can start a more constructive relationship on this topic, and elsewhere.

Some final thoughts on RAID. It was well organized and it was great to meet so many smart(er) people thinking about generally the same topics as I do. On the negative side I do feel disappointed over the apparent disconnect between the academic world and the more real world focused efforts like Suricata, Snort and tools like Streamdb, Sguil, Snortby, Squert, etc. But maybe I’m just lacking the vision to put the theory to practice.

The current tools out there may not be considered sufficient by everyone for every task. However, if RAID was a good benchmark, I fear we’ll have to settle for those for a while. Thats not necessarily a bad thing as fore-mentioned tools are under active development and continue to improve steadily.

On my Sguild server called “owl” — I’d like to think it’s very wise — I first went to my sguil directory, where the CVS checkout lives. There I did a “cvs up”. Next it was time to upgrade the database:

owl:/usr/local/sguil/server/sql_scripts# ./update_0.8.tcl
This script is used for upgrading from Sguil Version 0.7.x
to Sguil Version 0.8.x only.

Use these scripts at your own risk. Be sure to back
up your data before proceeding!!
Do you want to continue? (y/N) y
Database password: ***
Enter path to sguild.users file: /etc/sguild/sguild.users
Connecting to database…Success.
Trying to use database sguildb…Success.
Sguild DB Versions: 0.13
Migrating your password file (/etc/sguild/sguild.users) to the database:
Updating user name sanc
Updating user name victor
Success.

** Finished. The DB has been upgraded. **

owl:/usr/local/sguil/server/sql_scripts#

After this I started the sguild process and it came up just fine. Next I did “cvs up” on all sensors, restarted the agents. A final “cvs up” on my desktop and sguil.tk was updated as well.

The total upgrade took me only a few minutes and I’ve not encountered any regressions. Impressive work by Bamm and his team!

Meanwhile, people continued using Vuurmuur, it made it’s way into Debian, got removed from it again, made it’s way into Ubuntu. Lately, every few weeks someone would ask me if Vuurmuur was still being developed. My answer always was “yes, but very slowly”.

I plan to change that. The reason? IPv6. I’ve been using IPv6 on and off over the years, usually through the experimental tunnel service my ISP offered. But a while back my ISP started offering native IPv6 connectivity, which I’m using on a daily basis now. In the feature set Vuurmuur has, IPv6 is the only glaring omission. So, it’s time to address that.

Over the next months my idea is to slowly start adding IPv6 support to Vuurmuur. As I’m already using a simple script the idea is to start with logging support. Then move up from there.

Supporting all current features on IPv6 is going to require a lot of effort. In some cases I’m not even sure we can. But getting at least a basic IPv6 ruleset going should be fairly straightforward. If you’re interested in helping out, please let me know. Any help is greatly appreciated!

To be able to work with un-ACK’d data, we need to make sure we deal with possible evasions properly. The problem, as extensively documented by Judy Novak and Steven Sturges, is that in TCP streams there can be overlapping packets. Those are being dealt with differently based on the receiving OS. If we would need to account for overlaps in the application layer, we would have to be able to tell the HTTP parser for example: “sorry, that last data is wrong, please revert and use the new packet instead”. A nightmare.

The solution I opted for was to not care about destination OS’ for overlaps and such. The approach is fairly simple: once we have accepted a segment, thats what it’s going to be. This means that if we receive a segment later that (partially) overlaps and has different data, it’s data portion will simply be overwritten to be the same as the original segment. This way, the IPS and not an obscure mix of the sender (attacker?) and destination OS, determines the data the destination will see.

Of course the approach comes with some drawbacks. First, we need to keep segments in memory for a longer period of time. This causes significantly higher memory usage. Secondly, if we rewrite a packet, it needs to be reinjected on the wire. As we modified the packet payload a checksum recalculation is required.

In Suricata’s design the application layer parsers, such as our HTTP parser, run on top of the reassembly engine. After the reassembly engine and the app layer parsers are updated, the packet with the associated stream and app layer state is passed on to the detection engine. In the case where we work with ACK’d data, an ACK packet in the opposite direction triggers the reassembly process. If we detect based on that, and decide we need to drop, all we can do is drop the ACK packet as the actual data segment(s) have already passed. This is not good enough in many cases.

In the new code the data segment itself triggers the reassembly process. In this case, if the detection engine decides a drop is required, the packet containing the data itself can be dropped, not just the ACK. The reason we’re not taking the same approach in IDS mode is that we wouldn’t be able to properly deal with the said evasion/overlap issues. The IPS can exactly control what packets pass Suricata. The IDS, being passive, can not.

You can try this code by checking out the current git master. In the suricata.yaml that lives in our git tree you’ll find a new option in the stream config, “stream.inline”. If you enable this, the code as explained above is activated.

Feedback is very welcome!

It has been an very exciting year, with a lot of press and community interest for our project. Also, a lot of work has been done in the past year. I already wrote that our performance has increased a lot.

There have been over a thousand commits to our source code management system (git), more than all previous development together. Or as git shows it: “548 files changed, 193714 insertions(+), 39606 deletions(-)”. Quite impressive. Almost 20 developers have contributed code, some paid by OISF, some from our consortium members, some from the community.

Exactly one year ago, our codebase had a size of 120k lines of code. Today, we’re looking at 269k lines of code. Admittedly this includes our 2264 unittests (up from 1191), but still a large increase. In short in 2010 we’ve shown to the world we’re building a exciting and thriving project!

For 2011 we are in great shape. We have a stable 1.0.2 release and a promising upcoming 1.1 release. Emerging Threats (and also Emerging Threats Pro) has a tuned and optimized Suricata ruleset. Additionally, one of the 1.1 goals is to continue to fully support VRT’s ruleset.

Soon we’ll be starting work on Suricata 2.x, so there are exciting times ahead!

Happy New Year everybody!