Emerging-Threats on Inliniac

Suricata profiling per keyword

Thu, 07 Nov 2013 14:37:04 +0000

Last week I’ve added some more profiling options to Suricata. It’s part of the current git master. It’s enabled only when --enable-profiling and then through the suricata.yaml:

profiling:
  # per keyword profiling
  keywords:
    enabled: yes
    filename: keyword_perf.log
    append: yes

This will output a table similar to below:

--------------------------------------------------------------------------
Date: 11/7/2013 -- 15:13:11
--------------------------------------------------------------------------
Stats for: total
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- -----------
threshold        355324491   190574   409      72276       1864.00     3625.00     1860.00
content          1274592063  534328   196738   312321      2385.00     2424.00     2362.00
pcre             56626031    11149    824      254562      5079.00     12234.00    4507.00
byte_test        153287955   128254   32109    67989       1195.00     1658.00     1040.00
byte_jump        3676404     2041     2041     15939       1801.00     1801.00     0.00
flow             38276182    22842    22842    63987       1675.00     1675.00     0.00
isdataat         580764      558      556      2427        1040.00     1040.00     1017.00
dsize            2212029     2062     2061     3711        1072.00     1072.00     789.00
flowbits         1677209     874      870      9873        1919.00     1923.00     884.00
itype            1653        2        1        1386        826.00      267.00      1386.00
icode            27383781    93827    2        25545       291.00      1021.00     291.00
flags            192751968   245519   189709   255639      785.00      753.00      892.00
urilen           6149297     6142     1099     28299       1001.00     1395.00     915.00
byte_extract     143091      78       78       7743        1834.00     1834.00     0.00
--------------------------------------------------------------------------
Stats for: packet
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- -----------
flow             38276182    22842    22842    63987       1675.00     1675.00     0.00
dsize            2212029     2062     2061     3711        1072.00     1072.00     789.00
flowbits         351171      294      290      5526        1194.00     1198.00     884.00
itype            1653        2        1        1386        826.00      267.00      1386.00
icode            27383781    93827    2        25545       291.00      1021.00     291.00
flags            192751968   245519   189709   255639      785.00      753.00      892.00
--------------------------------------------------------------------------
Stats for: packet/stream payload
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- -----------
content          1203990910  512902   183628   312321      2347.00     2365.00     2337.00
pcre             28087301    6598     54       254562      4256.00     12279.00    4190.00
byte_test        153287955   128254   32109    67989       1195.00     1658.00     1040.00
byte_jump        3676404     2041     2041     15939       1801.00     1801.00     0.00
isdataat         578172      556      554      2427        1039.00     1039.00     1017.00
byte_extract     143091      78       78       7743        1834.00     1834.00     0.00
--------------------------------------------------------------------------
Stats for: http uri
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- -----------
content          44775802    13102    8351     60993       3417.00     3257.00     3698.00
pcre             18284421    3646     97       61338       5014.00     8916.00     4908.00
isdataat         2592        2        2        1725        1296.00     1296.00     0.00
urilen           6149297     6142     1099     28299       1001.00     1395.00     915.00
--------------------------------------------------------------------------
Stats for: http raw uri
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- -----------
pcre             9534        2        0        4953        4767.00     0.00        4767.00
--------------------------------------------------------------------------
Stats for: http client body
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- -----------
content          1556904     441      181      58476       3530.00     2874.00     3986.00
pcre             63924       6        6        17358       10654.00    10654.00    0.00
--------------------------------------------------------------------------
Stats for: http headers
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- -----------
content          23688244    7631     4348     31098       3104.00     3311.00     2829.00
pcre             9998970     859      667      71904       11640.00    12727.00    7862.00
--------------------------------------------------------------------------
Stats for: http stat code
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- -----------
content          80052       39       20       3699        2052.00     2199.00     1898.00
--------------------------------------------------------------------------
Stats for: http method
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- -----------
content          476334      203      201      27240       2346.00     2351.00     1846.00
--------------------------------------------------------------------------
Stats for: http cookie
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- -----------
content          23817       10       9        2763        2381.00     2384.00     2358.00
pcre             181881      38       0        13095       4786.00     0.00        4786.00
--------------------------------------------------------------------------
Stats for: post-match
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- -----------
flowbits         1326038     580      580      9873        2286.00     2286.00     0.00
--------------------------------------------------------------------------
Stats for: threshold
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- -----------
threshold        355324491   190574   409      72276       1864.00     3625.00     1860.00

The first part has the totals for all keywords. After this the stats are broken down per buffer type.

More on Suricata lua flowints

Tue, 23 Apr 2013 10:17:52 +0000

This morning I added flowint lua functions for incrementing and decrementing flowints. From the commit:

Add flowint lua functions for incrementing and decrementing flowints.

First use creates the var and inits to 0. So a call:
    a = ScFlowintIncr(0)
Results in a == 1.

If the var reached UINT_MAX (2^32), it’s not further incremented. If the var reaches 0 it’s not decremented further.

Calling ScFlowintDecr on a uninitialized var will init it to 0.

Suricata Lua scripting flowint access

Mon, 22 Apr 2013 16:16:30 +0000

A few days ago I wrote about my Emerging Threats sponsored work to support flowvars from Lua scripts in Suricata.

Today, I updated that support. Flowvar ‘sets’ are now real time. This was needed to fix some issues where a script was invoked multiple times in single rule, which can happen with some buffers, like HTTP headers.

Also, I implemented flowint support. Flowints in Suricata are integers stored in the flow context.

Suricata Lua scripting flowvar access

Thu, 18 Apr 2013 16:36:56 +0000

Funded by Emerging Threats, I’ve been working on giving the lua scripts access to flowvars.

Currently only “flowvars” are done, “flowints” will be next. Please review the code at: https://github.com/inliniac/suricata/tree/dev-lua-flowvar

Pcre based flowvar capturing is done in a post-match fashion. If the rule containing the “capture” matches, the var is stored in the flow.

For lua scripting, this wasn’t what the rule writers wanted. In this case, the flowvars are stored in the flow regardless of a rule match.

Closing in on Suricata 1.4

Thu, 29 Nov 2012 16:50:15 +0000

I just made Suricata 1.4rc1 available with some pretty exciting features: unix socket mode and IP reputation.

Unix socket

First of all, Eric Leblond’s work on the Unix socket was merged. The unix socket work consists of two parts. The unix socket protocol implementation and a new runmode.

The protocol implementation is based on JSON messages over unix socket. Eric will be fully documenting it soon. Currently the commands are limited to shutting down and getting some basic stats. This part isn’t very exciting yet, but the groundwork for many future extensions has been laid.

IP Reputation in Suricata

Wed, 21 Nov 2012 19:22:01 +0000

Disclaimer: this work was sponsored by Emerging Threats Pro.

One thing we’ve been talking about for many years at OISF is IP Reputation. The basic idea is that many organizations have information about specific IP-addresses. This information may be that a host is infected, acts as a spam relay or many other things. We’ve always thought it might be useful to apply this info to the IDS directly.

In the last weeks I’ve developed code to load IP reputation information into Suricata. This code is now part of the Suricata git master, so it’s available to all.

Suricata http_user_agent vs http_header

Mon, 09 Jul 2012 18:43:12 +0000

One of the new features in Suricata 1.3 is a new content modifier called http_user_agent. This allows rule writers to match on the User-Agent header in HTTP requests more efficiently. The new keyword is documented in the OISF wiki. In this post, I’ll show it’s efficiency with two examples.

Example 1: rarely matching UA

Consider a signature where the match if on a part of the UA that is very rare, so not part of regular User Agents. In my example “abc”.

Suricata 1.3 released

Fri, 06 Jul 2012 16:06:52 +0000

Today, almost half a year after the last “stable” release, we released Suricata 1.3. I think this release is a big step forward with regard to maturity of Suricata. Performance and scalability have been much improved, just like accuracy and stability.

The official announcement can be found on the OISF site

In the last 6 months a lot of code has been changed:

384 files changed, 44332 insertions(+), 18478 deletions(-)

Suricata 1.1 beta 1 released

Tue, 21 Dec 2010 17:56:32 +0000

Today we’ve released Suricata 1.1 beta 1, the first beta of the upcoming Suricata 1.1 release. The official release announcement is here on the OISF website.

The main focus of the new release has been to improve performance and to add support to the features the new ET/ETpro ruleset needs. ET and ETpro have rulesets specially tuned and geared for Suricata. We’re still missing some new rule keywords that are used by VRT, so in the 1.1 beta 2 release we’ll address that.

Setting up Suricata 0.9.0 for initial use on Ubuntu Lucid 10.04

Mon, 10 May 2010 14:27:25 +0000

The last few days I blogged about compiling Suricata in IDS and IPS mode. Today I’ll write about how to set it up for first use.

Starting with Suricata 0.9.0 the engine can run as an unprivileged user. For this create a new user called “suricata”.

useradd –no-create-home –shell /bin/false –user-group –comment “Suricata IDP account” suricata

This command will create a user and group called “suricata”. It will be unable to login as the shell is set to /bin/false.

SidReporter beta2 released

Thu, 21 Aug 2008 15:08:42 +0000

A little over a week ago the second beta of the SidReporter from Emerging Threats was released (see http://www.emergingthreats.net/content/view/95/1/). I’ve been working with Matt Jonkman to setup this new project at Emerging Threats, mostly in writing the reporter scripts. I think it’s an exciting new project that could provide the community with great information. As Matt wrote on the initial announcement:

“As mentioned a few weeks ago, we’ve been working to bring out tool to anonymously report IDS/IPS hits. Similar to DShield’s firewall log reporting, we believe we can make some incredible data inferences with this information, as well as help improve the quality of our signatures while giving us all feedback to tune our rulesets.

New Snortsam patch for Snort 2.8.0.1

Tue, 08 Jan 2008 12:30:53 +0000

Matt Jonkman of Emerging Threats asked me to have a look at the existing Snortsam 2.8.0.1 patch as people were continuing to report problems with it. I updated it to compile without compiler warnings, build cleanly with debugging enabled, build cleanly with Snort’s IPv6 support enabled and added a check so it won’t act on alerts in IPv6 packets since the Snortsam framework does not support IPv6. Finally I removed the patch script so it’s provided as a ’normal’ diff. Here is the patch: http://www.inliniac.net/files/snortsam-2.8.0.1.diff