Posts Tagged ‘Sguil’

Using Modsec2sguil for HTTP transaction logging revisited

Wednesday, August 22nd, 2007

Recently I wrote about the idea to log all HTTP transactions into Sguil using my Modsec2sguil agent. I’ve implemented this in the current 0.8-dev5 release and it works very well. All events go into Sguil smoothly and I’ve not experienced slowdowns on the webserver. I’ve been running it for almost a week now, like to share the first experiences here.

I find it to be quite useful. When receiving an alert, it is perhaps more interesting to see what else was done from that ipaddress than to see what was blocked (unless you are suspecting a false positive of course). One area I find to be useful is when I’m creating rules against comment spam on this blog. By seeing all properties of a spam message I can create better rules. For example on broken user-agents or weird codes inserted into the comment field of WordPress.

It’s easy to search and filter on HTTP response codes because the code is a part of the RT message. For example, when searching for all HTTP 500 error codes, add the following ‘WHERE’ clause to a query:

WHERE event.signature like “%MSc 500%”

This works quite fast although you best limit the query on properties like date and port as well. To get all the HTTP code 500 alerts from the last days do something like:

WHERE event.timestamp > ’2007-08-18′ AND (event.dst_port = 80 OR event.dst_port = 443) AND event.signature like “%MSc 500%”

One thing that is disappointing is the inabillity to search in the event payloads stored in the database. Technically it’s possible to create mysql queries that search for certain strings, but this process is so slow that it’s hardly usable in practice. The problem here is that the database field containing the payload is not indexed. I’ll show the query I used here (ripped from David Bianco’s blog)

WHERE event.timestamp >= ’2007-08-18′ AND (event.dst_port = 80 OR event.dst_port = 443) AND data.data_payload like CONCAT(“%”, HEX(“Mozilla/5.0″), “%”)

If you know a more efficient query, please let me know!

Using Modsec2sguil for HTTP transaction logging

Wednesday, August 15th, 2007

Modsec2sguil is currently configured to send alerts to Sguil. ModSecurity can be configured to log any event or transaction, including 200 OK, 302 Redirect, etc. Modsec2sguil distinguishes between alerts and other events by only processing HTTP codes of 400 and higher. Since 0.8-dev2 there is a configuration directive to prevent certain codes, such as 404, from being treated as an alert.

Now I have the following idea. Since ModSecurity can log all events with details of request headers, response headers and POST message body, it may be interesting to just send all these events to Sguil. They should not be appearing as alerts, but having them in the database can perhaps be interesting. I know using flow data and full packet captures the same data can be accessed, but having it in the database makes querying it a lot easier and longer available.

Possible problems are mostly the performance hit the webserver may take for sending all these events to Sguil and the storage requirements in Sguil’s database. I estimate the events are about 1kb in size on average, so on a busy site this may cause the database to grow very rapidly. Of course this behavior would be optional so it can be disabled.

Any thoughts on this idea?

First Modsec2sguil release for Sguil 0.7-CVS

Tuesday, August 14th, 2007

I just uploaded a new version of Modsec2sguil. I’ve been working on it the last weeks to get it updated to Sguil 0.7. The scripts are changed all over the place. This is because in the 0.7 framework, my scripts would no longer be a replacement for Barnyard only talking to the sensor_agent on the localhost, instead now it would become a full agent talking to the Sguil server directly.

This brings some challenges. First the connection can be going over the internet, or another untrusted network, so the agent needs ssl support. Second, since the connection may be unreliable we need to be able to detect and deal with lost connections. Next to this I wanted to be able to run without superuser privileges.

The new version of modsec2sguil supports it all, and more:

  • Converted into a real agent for Sguil 0.7 (no more barnyard replacement)
  • Agent can drop privileges
  • Agent can daemonize
  • Pinging the server is supported
  • The agent reconnects to the server if the connection is lost
  • Agent supports SSL for the connection to the server
  • A sguil-compatible configuration file is now used
  • A debug mode was added

So if you run Sguil 0.7-CVS and ModSecurity, go check it out at http://www.inliniac.net/modsec2sguil/

Last but not least, the agent contains a SguilAgent.pm Perl library. I hope it enables developers to easily create Perl agents for Sguil. If you need help with that, please let me know!

Migrating a Sguil server from 0.6.1 to 0.7.0 (CVS)

Wednesday, July 4th, 2007

Today I finally restored my server that used to host my blog, mail server and sguil server. The sguil server was still at 0.6.1 so this was a good time to see how a migration procedure would work (the earlier 0.7.0 test were done with a newly setup server). I haven’t been able to find documentation about this procedure, but it looks very straightforward, so I think I did it all right.

Like everyone always does, I started by creating a backup (you do make backups before upgrades, don’t you? ;-) )

Next I got the latest version of Sguil from CVS. I made cvs place the directory in the /usr/local/ directory. In /usr/local/sguil/server/sql_scripts I saw a tcl script named ‘update_0.7.tcl’. As the name looked very promising, I decided to try it. I ran the script as a normal user, not as root.

$ ./update_0.7.tcl

This script is used for upgrading from Sguil Version 0.6.x
to Sguil Version 0.7.x only.

Use this script at your own risk. Be sure to back
up your data before proceeding!!
Do you want to continue? (y/N)

This looks good, let’s continue.

Database password:

First the script asks for the password of your database. Note that you need to enter the root password for your MySql database, not that of the sguil user.

Connecting to database…Success.
Trying to use database sguildb…Success.
Sguild DB Versions: 0.11
The DB schema needs to be updated. Do it now ([y]/n)?:

Yes please.

Path to update_sguildb_v11-v12.sql [./update_sguildb_v11-v12.sql]:

The default path is valid, so I pressed enter.

Updating database sguildb……………………..Success.
Getting a list of current Snort sensors…Success.

Okay, so far so good. This took only a few moments.

WARNING: The next step is important. Please make sure you understand the concept before continuing.

The functions of the old sensor agent have been split out into separate agents (snort_agent, pcap_agent, sancp_agent, and pads_agent). Each agent requires its own sensor id (sid). Older Sguil installs used the same sid for Snort alerts and SANCPflows. The separation of these agents also allows you to place agents on different pieces of hardware.

Net names are used to correlate data between these agents. For example, when an analyst requests the pcap associated with hosts from a specific alert, sguild will use the net name to determine which pcap agent to make the request too (each agent registers its net name when it connects).

This next process will create a new sid for any sensors and also prompt you for a net name to assign to them. Make sure you remember these net names as you will need them when you configure the agents. Net names should be simple descriptors like Corp_Ext_Net, DMZ, or web_farm.

** Press <Enter> when you are ready to continue **

I’m ready to go!

sid | hostname | net_name | agent_type
========================
1 | vuurmuur | |
2 | wiki-ids-01 | |
3 | eagle | |
4 | wikiweb01 | |

Please enter a net name for vuurmuur:

Nice sensor names huh? :) The sensor ‘vuurmuur’ is located at my home, so thats how I will call the net_name: ‘home’.

The net name for vuurmuur will be set to home.
Is this correct (y/n)?

Yes sir!

Updating net name and agent type for vuurmuur…Success.
Checking for SANCP data from this sensor…Success.
Found SANCP data for the sensor vuurmuur with an ID of 1.
Adding agent information to the sensor table…Success.
Updating SANCP data to reflect new sid (5). This could take a bit…Success.

This took about 7 minutes.

sid | hostname | net_name | agent_type
========================
1 | vuurmuur | home | snort
2 | wiki-ids-01 | |
3 | eagle | |
4 | wikiweb01 | |
5 | vuurmuur | home | sancp

Please enter a net name for wiki-ids-01:

This IDS sensor is located in Utrecht, so I’m naming the net_name ‘utrecht’. This time the process finished in about 2 seconds. Of the sensors ‘eagle’ and ‘wikiweb01′ the former went in ‘home’, the latter in ‘utrecht’. Both are ModSecurity sensors and thus have no associated sancp data. The update script properly detected this:

No SANCP data was found for the sensor eagle with an ID of 3.
No SANCP data was found for the sensor wikiweb01 with an ID of 4.

Okay, this script is done:

** Finished. The DB has been upgraded. **

So far it all went very smooth. Let’s have a look at the configuration. I’m using my 0.6.1 sguild.conf, which is fine although it needs a few adjustments.

Because my sguil installation is now at a different location I needed to change the library path setting:

# Path the sguild libs
set SGUILD_LIB_PATH /usr/local/sguil/server/lib

There is also a new directive in the configuration file:

# Where to store DB LOADable files until loaderd can put them in the DB
set TMP_LOAD_DIR /sguild_data/load

I also created this directory.

Next I couldn’t think of anything more to do, so I started sguild. And it worked! Client could connect, sensors could connect (ModSecurity sensors are not yet updated because Modsec2Sguil doesn’t support 0.7 yet) so as far as I can see, the upgrade process is complete and successful! Nice work Bamm! :)

Update on Sguil 0.7-CVS client on Ubuntu Feisty

Wednesday, July 4th, 2007

A short time ago I wrote about how the Sguil 0.7-CVS client can be installed on Ubuntu Feisty. Since then there has been a change to Sguil that changes the requirements a bit. Because of this the standard tcllib package in Feisty is no longer usable. It provides tcllib 1.8 while Sguil now needs 1.9. Luckily, we can use the tcllib package from the upcoming Ubuntu release called ‘Gutsy’. It can be found here: http://packages.ubuntu.com/gutsy/interpreters/tcllib

I installed it with this command:

$ sudo dpkg –install tcllib_1.9.dfsg1-1_all.deb

This made Sguil 0.7-CVS work again on my system.

Sguil 0.7-CVS client on Ubuntu Feisty

Tuesday, June 19th, 2007

I just got a new workstation that I’m setting up today with Ubuntu Feisty 7.04. When setting up the Sguil client from CVS I needed to install the following packages (including dependencies, but apt-get takes care of that):

tcl8.4
tclx8.4
tcllib
tk8.4
iwidgets4

After this it ran but looked horrible because of some ugly font that was used. I found that for my use the following fonts look good:

standard: Bitstream Vera Sans 12
fixed: Bitstream Vera Sans Mono 12

Both fonts are installed by default and can be selected in the Sguil client by opening the ‘File’ dropdown menu and choosing ‘Change Font’.

Sguil 0.7 CVS installation on Debian Etch

Tuesday, June 12th, 2007

Sguil 0.7 is getting shape quite nicely. One of the most interesting new features is the splitting up of different types of agents and the option to create ‘net groups’. This are groups of agents that Sguil considers part of the same network. You can use this to spread the agents over multiple servers, but still use it from Sguil as if it was one single sensor. For example, this way you can easily create a Snort sensor and a separate full content logging capture server. When you request the full content for a Snort event in Sguil, it will know that it needs to request the packet data from the capture server. This way you can also have multiple Snort agents without the need for capturing the same sancp and full content data over and over again.

David Bianco has written a very nice guide for installing Sguil 0.7 on Redhat Enterprise 4. I used this guide to install the server and sensor on a Debian Etch installation. The main difference is that I used Debian packages where ever possible. These packages could be used:

mysql-server
p0f
tcpflow
tcllib
mysqltcl
tcltls
tcl8.3
tclx8.3

Important: do not use the tcl8.4 package. It is not compatible with Sguil and will produce the following message:

ERROR: This version of tcl was compile with threading enabled. Sguil is NOT compatible with threading.
SGUILD: Exiting…

You can get Sguil 0.7 CVS by checking out the latest CVS version:

cvs -d:pserver:anonymous@sguil.cvs.sourceforge.net:/cvsroot/sguil login
cvs -d:pserver:anonymous@sguil.cvs.sourceforge.net:/cvsroot/sguil co sguil

I will update Modsec2sguil soon!

Modsec2sguil 0.7 released

Sunday, March 18th, 2007

I’ve just released version 0.7 of Modsec2sguil, the set of perl scripts to feed ModSecurity alerts to the Sguil NSM system. The main change of this release is that it adds support for alerts produced by ModSecurity 2.x, while 1.9.x remains to be supported. Next to this the conversion between ModSecurity’s severity and Snort’s priority was fixed, so alerts should show up in the right pane in Sguil again.

Please give this release a try and let me know how it works for you!

Download it here: http://www.inliniac.net/files/modsec2sguil-0.7.tar.gz

Experimenting with IPv6

Tuesday, March 13th, 2007

My ISP is one of the few here in the Netherlands that provides a IPv6 tunnel broker. I have played with it some during the last year or so, but now decided to get a little more serious with it. So I’ve decided to enable it for my blog. When opening up my site to IPv6 one thing that is important is security. I will describe the status of IPv6 support of my current setup:

Linux firewalling: IPtables supports IPv6 for quite some time, however it only very recently gained stateful packet filtering support. This hasn’t made it into Debian Sarge or even backports yet, so I’m just using stateless filtering now.

Vuurmuur: my own IPtables frontend has no support for IPv6 at all. I’ve been thinking about adding it for years, but decided to wait at least until stateful support would be available. Next to this my coding time is limited, and many other features are probably more interesting to Vuurmuur users.

Snort/Snort_inline: both Snort and Snort_inline lack support for IPv6. Sourcefire is working on it as far as I know, but no code is available from them. I did find a IPv6 patch for Snort 2.3.3, which can be found here. I ran it in sniffer mode and that works. I haven’t played with it much other than that, but I certainly will in the future.

ModSecurity: my Apache 2 installation has IPv6 enabled by default and ModSecurity 2.x just worked with it without any configuration change! I haven’t looked into how to create rules specific for IPv6 addresses however, so maybe surprises will come up here. I do know from looking at the source that the rbl functionality doesn’t support IPv6 addresses yet, but I haven’t even checked if realtime blacklists exist for IPv6.

Sguil/Modsec2sguil: my modsec2sguil script, that takes ModSecurity alerts and feeds them to Sguil, doesn’t act on the IPv6 alerts because it expects IPv4 addresses. This is not a problem however, since Sguil doesn’t support IPv6 addresses. This makes sense since Snort doesn’t support it either.

So compared to my IPv4 access, protection is somewhat limited. I’m only enabling HTTP for now, so ModSecurity should be able to handle that just fine.

Anyway, it seems to be working fine now, but consider the IPv6 support experimental, as I’m playing with how it all works. So don’t be surpised if it’s broken all of a sudden ;-)