Archive for the ‘ids’ Category

« Older Entries

HTTP parsing events in Suricata

Wednesday, January 11th, 2012

With the 1.2rc1 release you will notice no more HTTP errors on the screen. Or SMTP errors. This output has been disabled finally. This was a long time annoyance.

As you may still be interested in the errors they are now available through the rule language. In rules/http-events.rules and rules/smtp-events.rules rules for all possible events/errors can be found.

Example:
app-layer-event:http.missing_host_header;

This will match on HTTP/1.1 requests without a Host header.

Some of these rules might be noisy (they are not in my local network), but rather than disabling them I’d suggest suppressing then. The reason is that for each time they hit a flowint will be incremented:

flowint:http.anomaly.count,+,1;

This will allow you to get alerts on streams with high anomaly counts:

alert http any any -> any any (msg:"LOCAL really poor HTTP session"; flowint:http.anomaly.count,>,5; sid:123; rev:1;)

This will give you an alert if there have been more than 5 anomalies detected.

Blog spammers, malware and other unwanted HTTP users often use HTTP with all kinds of issues, so this may be a helpful tool in detecting those.

Tags: , , , ,
Posted in ids, oisf, Suricata | No Comments »

Suricata 1.1.1 released

Wednesday, December 7th, 2011

A maintenance update for the Suricata 1.1 series was just released. It fixed an important issue. In some cases Suricata could crash on SMTP traffic.

The full announcement for the 1.1.1 release is here.

Naturally, the issue has also been fixed in the 1.2 development branch.

Tags: , ,
Posted in ids, IPS, oisf, Suricata | No Comments »

File extraction in Suricata

Tuesday, November 29th, 2011

Today I pushed out a new feature in Suricata I’m very excited about. It has been long in the making and with over 6000 new lines of code it’s a significant effort. It’s available in the current git master. I’d consider it alpha quality, so handle with care.

So what is this all about? Simply put, we can now extract files from HTTP streams in Suricata. Both uploads and downloads. Fully controlled by the rule language. But thats not all. I’ve added a touch of magic. By utilizing libmagic (this powers the “file” command), we know the file type of files as well. Lots of interesting stuff that can be done there.

Rule keywords

Four new rule keywords were added: filename, fileext, filemagic and filestore.

Filename and fileext are pretty trivial: match on the full name or file extension of a file.

alert http any any -> any any (filename:”secret.xls”;)
alert http any any -> any any (fileext:”pdf”;)

More interesting is the filemagic keyword. It runs on the magic output of inspecting the (start of) a file. This value is for example:

GIF image data, version 89a, 1 x 1
PE32 executable for MS Windows (GUI) Intel 80386 32-bit
HTML document text
Macromedia Flash data (compressed), version 9
MS Windows icon resource – 2 icons, 16×16, 256-colors
PNG image data, 70 x 53, 8-bit/color RGBA, non-interlaced
JPEG image data, JFIF standard 1.01
PDF document, version 1.6

So how the filemagic keyword allows you to match on this is pretty simple:

alert http any any -> any any (filemagic:”PDF document”;)
alert http any any -> any any (filemagic:”PDF document, version 1.6″;)

Pretty cool, eh? You can match both very specifically and loosely. For example:

alert http any any -> any any (filemagic:”executable for MS Windows”;)

Will match on (among others) these types:

PE32 executable for MS Windows (DLL) (GUI) Intel 80386 32-bit
PE32 executable for MS Windows (GUI) Intel 80386 32-bit
PE32+ executable for MS Windows (GUI) Mono/.Net assembly

Finally there is the filestore keyword. It is the simplest of all: if the rule matches, the files will be written to disk.

Naturally you can combine the file keywords with the regular HTTP keywords, limiting to POST’s for example:

alert http $EXTERNAL_NET any -> $HOME_NET any (msg:”pdf upload claimed, but not pdf”; flow:established,to_server; content:”POST”; http_method; fileext:”pdf”; filemagic:!”PDF document”; filestore; sid:1; rev:1;)

This will alert on and store all files that are uploaded using a POST request that have a filename extension of pdf, but the actual file is not pdf.

Storage

The storage to disk is handled by a new output module called “file”. It’s config looks like this:

enabled: yes # set to yes to enable
log-dir: files # directory to store the files
force-magic: no # force logging magic on all stored files

It needs to be enabled for file storing to work.

The files are stored to disk as “file.1″, “file.2″, etc. For each of the files a meta file is created containing the flow information, file name, size, etc. Example:

TIME: 01/27/2010-17:41:11.579196
PCAP PKT NUM: 2847035
SRC IP: 68.142.93.214
DST IP: 10.7.185.57
PROTO: 6
SRC PORT: 80
DST PORT: 56207
FILENAME: /msdownload/update/software/defu/2010/01/mpas-fe_7af9217bac55e4a6f71c989231e424a9e3d9055b.exe
MAGIC: PE32+ executable for MS Windows (GUI) Mono/.Net assembly
STATE: CLOSED
SIZE: 5204

Configuration

The file extraction is for HTTP only currently, and works on top of our HTTP parser. As the HTTP parser runs on top of the stream reassembly engine, configuration parameters of both these parts of Suricata affect handling of files.

The stream engine option “stream.reassembly.depth” (default 1 Mb) controls the depth into a stream in which we look. Set to 0 for no limit.
The libhtp options request-body-limit and response-body-limit control how far into a HTTP request or response body we look. Again set to 0 for no limit. This can be controlled per HTTP server.

Performance

The file handling is fully streaming, so it’s very efficient. Nonetheless there will be an overhead for the extra parsing, book keeping, writing to disk, etc. Memory requirements appear to be limited as well. Suricata shouldn’t keep more than a few kb per flow in memory.

Limitations

Lack of limits is a limitation. For file storage no limits have been implemented yet. So it’s easy to clutter your disk up with files. Example: 118Gb enterprise pcap storing just JPG’s extracted 400.000 files. Better use a separate partition if you’re on a life link.

Future work

Apart from stabilizing this code and performance optimizing it, the next step will be SMTP file extraction. Possibly other protocols, although nothing is set in stone there yet.

Tags: , , ,
Posted in ids, IPS, oisf, Suricata | 7 Comments »

Suricata 1.1 released, 1.2 on the horizon

Thursday, November 10th, 2011

Today we released Suricata 1.1. This ends a rather long development cycle of more than a year. And it shows. Performance, accuracy and features were all greatly improved. I think it’s the best Suricata so far. If you’ve been looking at trying Suricata, now might be a good time to jump in.

The long development cycles should be something of the past. At our last brainstorm session, at RAID 2011, we decided to change our release policy. The aim of this policy is to do time based releases, roughly a “stable” every 2 months and a beta every other month. This way we’ll be making it much easier for users to stay current without have to run our “git master”.

Looking forward, we’ve started work on the 1.2 release, which should happen in about 2 months. Focus will be on performance. We’re planning to do a significant refactoring of our pattern matching engine, which should lead both to better performance and improved accuracy. Next to this, we’ll be finally adding the “file_data” keyword along with HTTP file carving — extracting files from HTTP requests. I am personally very excited about this.

We’re starting to see more and more community involvement. Not just on the user side, but also on the development side. As seen on the oisf-devel mailinglist, a large SSL/TLS patch set was contributed by Pierre Chifflier. This will make it’s way into the 1.2 release as well. Smaller contributions were accepted on PF_RING code and the HTTP code. I am very grateful for the contributions.

Eric Leblond and I will be doing a talk next week at DeepSec on Suricata. If you are able to, please come meet us!

Tags: ,
Posted in ids, IPS, oisf, Suricata | No Comments »

RAID 2011 Thoughts

Saturday, September 24th, 2011

The last few days I’ve been at the Recent Advances in Intrusion Detection (RAID) conference in California. Overall it has been a very pleasant and interesting experience. The nice California weather was certainly helping a lot!

I’ve seen all talks and some were very interesting. However, being a Suricata IDS developer, I was not just interested in research for the hell of it, but I was actively scouting for ideas we could implement into Suricata. In this respect the conference was highly disappointing. Although with some of the talks I thought the idea was applicable in general security, like Erik Bosmans high speed memory tainting detection, I found nothing like that for NIDS.

Most inspiring part of the conference was spending an evening with Seth Hall, one of the Bro IDS engineers. Bro has a very different approach to inspecting the network than Suricata. Actually, I should say Suricata does it differently as Bro has been around much longer than Suricata. :) The conversation was all about sharing of ideas and experiences, and finding common grounds for actual cooperation.

A couple of notes from that conversation. First, Bro supports Unified2/Barnyard2 now, as input (so actually Barnyard2 can output to Bro). This means it can extend it’s analysis to include Suricata generated events. Second, we might try to have Suricata and Bro work together, where Suricata would be controlled by Brocolli. This way Bro could benefit from Suricata’s high speed signature matching engine, functionality Bro doesn’t have, and Suricata could benefit from Bro’s higher level understanding of the network. Finally, Bro’s binpack effort to define protocol parsers in a higher level language that can then be compiled into native code looks interesting as well. It would probably take quite a bit of changes to get this all going, but it might just be worth it.

Then there was the panel at the conference with Martin Roesch, Seth Hall and myself. A lot of people expected fireworks, but no such thing happened. Everyone was polite, respectful and friendly. It never really turned into a real discussion though, it was more a Q&A with the audience. Dominique Karg blogged about the panel here.

It was good to talk to Martin Roesch. The OISF – Sourcefire relation has definitely not started well, so it was good to have normal conversations and such. I offered Marty to work together, especially on SCADA detection. As was announced earlier, OISF will maintain the Digital Bond Quickdraw SCADA parsers and keywords, not only for Suricata, but also for Snort. Hopefully we can start a more constructive relationship on this topic, and elsewhere.

Some final thoughts on RAID. It was well organized and it was great to meet so many smart(er) people thinking about generally the same topics as I do. On the negative side I do feel disappointed over the apparent disconnect between the academic world and the more real world focused efforts like Suricata, Snort and tools like Streamdb, Sguil, Snortby, Squert, etc. But maybe I’m just lacking the vision to put the theory to practice.

The current tools out there may not be considered sufficient by everyone for every task. However, if RAID was a good benchmark, I fear we’ll have to settle for those for a while. Thats not necessarily a bad thing as fore-mentioned tools are under active development and continue to improve steadily.

Tags: , , , , ,
Posted in ids, IPS, oisf, Snort, Suricata | No Comments »

Listening on multiple interfaces with Suricata

Friday, December 24th, 2010

A question I see quite often is, can I listen on multiple interfaces with a single Suricata instance? Until now the answer always was “no”. I’d suggest trying the “any”-pseudo interface (suricata -i any), with an bpf to limit the traffic or using multiple instances of Suricata. That last suggestion was especially painful, as one of the goals of Suricata is to allow a single process to process all packets using all available resources.

Last week I found some time to look at how hard adding support for acquiring packets from multiple interfaces would be. Turned out, not so hard! Due to Suricata’s highly modular threading design, it was actually quite easy. I decided to keep it simple, so if you want to add multiple interfaces to listen on, just add each separately on the command line, like so: suricata -i eth0 -i eth1 -i ppp0. This will create a so called “receive thread” for each of those interfaces.

I’ve added no internal limits, so in theory it should possible to add dozens. I just tested with 2 though, so be careful. Normally the thread name in logs and “top” for the pcap receive thread is “ReceivePcap”. This is still true if a single interface is passed to Suricata. In case more are passed to Suricata, thread names change to “RecvPcap-<int>”, e.g. RecvPcap-eth0 and RecvPcap-eth1. Untested, but it should work fine to monitor multiple interfaces from different types. Suricata sets the data link type in the interface-specific receive thread.

If you’re interested in trying out this new feature, there are a few limitations to consider. First, no Windows support yet. I hope this can be addressed later. Second, the case where two or more interfaces (partly) see the same traffic is untested. The problem here is that we’ll see identical packets going into the engine. This may (or may not, like I said, it’s untested) screw up the defrag, stream engines. Might cause duplicate alerts, etc. Addressing this is something that would probably require keeping a hash of packets so we can detect duplicates. This is probably quite computationally intensive, so it may not be worth it. I’m very much open to other solutions. Patches are even more welcome :)

So, for now use it only if interfaces see completely separate traffic. Unless you’re interested to see what happens if you ignore my warnings, in that case I’d like to know! The code is available right now in our current git master, and will be part of 1.1beta2.

Merry xmas everyone!

Tags: , , ,
Posted in ids, oisf, Suricata | 3 Comments »

Suricata 1.1 beta 1 released

Tuesday, December 21st, 2010

Today we’ve released Suricata 1.1 beta 1, the first beta of the upcoming Suricata 1.1 release. The official release announcement is here on the OISF website.

The main focus of the new release has been to improve performance and to add support to the features the new ET/ETpro ruleset needs. ET and ETpro have rulesets specially tuned and geared for Suricata. We’re still missing some new rule keywords that are used by VRT, so in the 1.1 beta 2 release we’ll address that.

Other than that, I got quite a few patches waiting. We’ll be improving stream reassembly, inline mode, prelude output, and numerous other things.

Like always, please give this a try and let us know how it works for you!

Tags: , , , , ,
Posted in ids, IPS, oisf, Suricata | 2 Comments »

Setting up Suricata 0.9.0 for initial use on Ubuntu Lucid 10.04

Monday, May 10th, 2010

The last few days I blogged about compiling Suricata in IDS and IPS mode. Today I’ll write about how to set it up for first use.

Starting with Suricata 0.9.0 the engine can run as an unprivileged user. For this create a new user called “suricata”.

useradd --no-create-home --shell /bin/false --user-group --comment “Suricata IDP account” suricata

This command will create a user and group called “suricata”. It will be unable to login as the shell is set to /bin/false.

The next thing to do is creating a configuration directory. Create /etc/suricata/ and copy the suricata.yaml example config into it. The example configuration can be found in the source archive you used to build Suricata:

mkdir /etc/suricata
cp /path/to/suricata-0.9.0/suricata.yaml /etc/suricata/
cp /path/to/suricata-0.9.0/classification.config /etc/suricata/

Next, create the log directory.

mkdir /var/log/suricata

The log directory needs to be writable for the user and group “suricata”, so change the ownership:

chown suricata:suricata /var/log/suricata

The last step I’ll be describing here is retrieving an initial ruleset. The 2 main rulesets you can use are Emerging Threats (ET) and Sourcefire’s VRT ruleset. Since putting VRT to use is a little bit more complicated I’ll be focussing on ET here.

First, download the emerging rules:

wget http://www.emergingthreats.net/rules/emerging.rules.tar.gz

Go to /etc/suricata/ and extract the rules archive:

cd /etc/suricata/
tar xzvf /path/to/emerging.rules.tar.gz

There is a lot more to rules, such as tuning and staying updated, but thats beyond the scope of this post.

Suricata is now ready to be started:

suricata -c /etc/suricata/suricata.yaml -i eth0 --user suricata --group suricata

If all is setup properly, Suricata will tell you it is now running:

[2087] 9/5/2010 — 18:17:47 – (tm-threads.c:1362) (TmThreadWaitOnThreadInit) — all 8 packet processing threads, 3 management threads initialized, engine started.

There are 3 log files in /var/log/suricata that will be interesting to monitor:

- stats.log: displays statistics on packets, tcp sessions etc.
- fast.log: a alerts log similar to Snort’s fast log.
- http.log: displays HTTP requests in a Apache style format.

This should get you going. There is a lot more to deploying Suricata that I plan to blog on later.

Tags: , , , ,
Posted in ids, IPS, oisf, Suricata, Ubuntu | 1 Comment »

Compiling Suricata 0.9.0 in Ubuntu Lucid 10.04 in IDS mode

Friday, May 7th, 2010

Note: the difference with the 0.8.2 post is that addition of libcap-ng-dev. This allows Suricata to run as an unprivileged user.

Ubuntu Lucid 10.04 LTS looks like a good platform for running an IDS on. It’s up to date and has long term support. Here is how to compile and install Suricata 0.9.0 on it.

Install the following packages needed to build Suricata: libpcre3-dev libpcap-dev libyaml-dev zlib1g-dev libcap-ng-dev.

apt-get install libpcre3-dev libpcap-dev libyaml-dev zlib1g-dev libcap-ng-dev

Download Suricata 0.9.0 here

Extract the suricata-0.9.0.tar.gz file as follows:

tar xzvf suricata-0.9.0.tar.gz

Enter the extracted directory suricata-0.9.0.

Run “./configure”
Note that you may get a warning about libnet 1.1 that is missing. You can ignore that, it’s only used in IPS/inline mode currently.
If “./configure” was succesful, run “make”
If “make” was succesful, run “sudo make install”
Except for Suricata itself, the build process installed “libhtp”. For that to work properly, run “ldconfig”.

Run “suricata -V” and it should report version 0.9.0.

To use Suricata in IDS mode, pass -i to the command line. Example

suricata -c /etc/suricata/suricata.yaml -i eth0

Tags: , ,
Posted in ids, Suricata, Ubuntu | 1 Comment »

Suricata 0.9.0 released

Friday, May 7th, 2010

Yesterday we released we first release candidate for our upcoming 1.0 release of Suricata. See the announcement on the OISF site here.

Most notable changes are the following new features:

- Support for the http_headers keyword was added
- libhtp was updated to version 0.2.3
- Privilege dropping using libcap-ng is now supported
- Proper support for “pass” rules was added
- Inline mode for Windows was added

Go get the release here: http://www.openinfosecfoundation.org/download/suricata-0.9.0.tar.gz

Tags: , , , , ,
Posted in ids, IPS, oisf, Suricata | No Comments »

« Older Entries