Inliniac

I just uploaded a new version of Modsec2sguil. I’ve been working on it the last weeks to get it updated to Sguil 0.7. The scripts are changed all over the place. This is because in the 0.7 framework, my scripts would no longer be a replacement for Barnyard only talking to the sensor_agent on the localhost, instead now it would become a full agent talking to the Sguil server directly.

This brings some challenges. First the connection can be going over the internet, or another untrusted network, so the agent needs ssl support. Second, since the connection may be unreliable we need to be able to detect and deal with lost connections. Next to this I wanted to be able to run without superuser privileges.

The new version of modsec2sguil supports it all, and more:

• Converted into a real agent for Sguil 0.7 (no more barnyard replacement)
• Agent can drop privileges
• Agent can daemonize
• Pinging the server is supported
• The agent reconnects to the server if the connection is lost
• Agent supports SSL for the connection to the server
• A sguil-compatible configuration file is now used
• A debug mode was added

So if you run Sguil 0.7-CVS and ModSecurity, go check it out at http://www.inliniac.net/modsec2sguil/

Last but not least, the agent contains a SguilAgent.pm Perl library. I hope it enables developers to easily create Perl agents for Sguil. If you need help with that, please let me know!

Today I finally restored my server that used to host my blog, mail server and sguil server. The sguil server was still at 0.6.1 so this was a good time to see how a migration procedure would work (the earlier 0.7.0 test were done with a newly setup server). I haven’t been able to find documentation about this procedure, but it looks very straightforward, so I think I did it all right.

Like everyone always does, I started by creating a backup (you do make backups before upgrades, don’t you? )

Next I got the latest version of Sguil from CVS. I made cvs place the directory in the /usr/local/ directory. In /usr/local/sguil/server/sql_scripts I saw a tcl script named ‘update_0.7.tcl’. As the name looked very promising, I decided to try it. I ran the script as a normal user, not as root.

$ ./update_0.7.tcl

This script is used for upgrading from Sguil Version 0.6.x
to Sguil Version 0.7.x only.

Use this script at your own risk. Be sure to back
up your data before proceeding!!
Do you want to continue? (y/N)

This looks good, let’s continue.

Database password:

First the script asks for the password of your database. Note that you need to enter the root password for your MySql database, not that of the sguil user.

Connecting to database…Success.
Trying to use database sguildb…Success.
Sguild DB Versions: 0.11
The DB schema needs to be updated. Do it now ([y]/n)?:

Yes please.

Path to update_sguildb_v11-v12.sql [./update_sguildb_v11-v12.sql]:

The default path is valid, so I pressed enter.

Updating database sguildb……………………..Success.
Getting a list of current Snort sensors…Success.

Okay, so far so good. This took only a few moments.

WARNING: The next step is important. Please make sure you understand the concept before continuing.

The functions of the old sensor agent have been split out into separate agents (snort_agent, pcap_agent, sancp_agent, and pads_agent). Each agent requires its own sensor id (sid). Older Sguil installs used the same sid for Snort alerts and SANCPflows. The separation of these agents also allows you to place agents on different pieces of hardware.

Net names are used to correlate data between these agents. For example, when an analyst requests the pcap associated with hosts from a specific alert, sguild will use the net name to determine which pcap agent to make the request too (each agent registers its net name when it connects).

This next process will create a new sid for any sensors and also prompt you for a net name to assign to them. Make sure you remember these net names as you will need them when you configure the agents. Net names should be simple descriptors like Corp_Ext_Net, DMZ, or web_farm.

** Press <Enter> when you are ready to continue **

I’m ready to go!

sid | hostname | net_name | agent_type
========================
1 | vuurmuur | |
2 | wiki-ids-01 | |
3 | eagle | |
4 | wikiweb01 | |

Please enter a net name for vuurmuur:

Nice sensor names huh? The sensor ‘vuurmuur’ is located at my home, so thats how I will call the net_name: ‘home’.

The net name for vuurmuur will be set to home.
Is this correct (y/n)?

Yes sir!

Updating net name and agent type for vuurmuur…Success.
Checking for SANCP data from this sensor…Success.
Found SANCP data for the sensor vuurmuur with an ID of 1.
Adding agent information to the sensor table…Success.
Updating SANCP data to reflect new sid (5). This could take a bit…Success.

This took about 7 minutes.

sid | hostname | net_name | agent_type
========================
1 | vuurmuur | home | snort
2 | wiki-ids-01 | |
3 | eagle | |
4 | wikiweb01 | |
5 | vuurmuur | home | sancp

Please enter a net name for wiki-ids-01:

This IDS sensor is located in Utrecht, so I’m naming the net_name ‘utrecht’. This time the process finished in about 2 seconds. Of the sensors ‘eagle’ and ‘wikiweb01′ the former went in ‘home’, the latter in ‘utrecht’. Both are ModSecurity sensors and thus have no associated sancp data. The update script properly detected this:

No SANCP data was found for the sensor eagle with an ID of 3.
No SANCP data was found for the sensor wikiweb01 with an ID of 4.

Okay, this script is done:

** Finished. The DB has been upgraded. **

So far it all went very smooth. Let’s have a look at the configuration. I’m using my 0.6.1 sguild.conf, which is fine although it needs a few adjustments.

Because my sguil installation is now at a different location I needed to change the library path setting:

# Path the sguild libs
set SGUILD_LIB_PATH /usr/local/sguil/server/lib

There is also a new directive in the configuration file:

# Where to store DB LOADable files until loaderd can put them in the DB
set TMP_LOAD_DIR /sguild_data/load

I also created this directory.

Next I couldn’t think of anything more to do, so I started sguild. And it worked! Client could connect, sensors could connect (ModSecurity sensors are not yet updated because Modsec2Sguil doesn’t support 0.7 yet) so as far as I can see, the upgrade process is complete and successful! Nice work Bamm!

A short time ago I wrote about how the Sguil 0.7-CVS client can be installed on Ubuntu Feisty. Since then there has been a change to Sguil that changes the requirements a bit. Because of this the standard tcllib package in Feisty is no longer usable. It provides tcllib 1.8 while Sguil now needs 1.9. Luckily, we can use the tcllib package from the upcoming Ubuntu release called ‘Gutsy’. It can be found here: http://packages.ubuntu.com/gutsy/interpreters/tcllib

I installed it with this command:

$ sudo dpkg –install tcllib_1.9.dfsg1-1_all.deb

This made Sguil 0.7-CVS work again on my system.

I just got a new workstation that I’m setting up today with Ubuntu Feisty 7.04. When setting up the Sguil client from CVS I needed to install the following packages (including dependencies, but apt-get takes care of that):

tcl8.4
tclx8.4
tcllib
tk8.4
iwidgets4

After this it ran but looked horrible because of some ugly font that was used. I found that for my use the following fonts look good:

standard: Bitstream Vera Sans 12
fixed: Bitstream Vera Sans Mono 12

Both fonts are installed by default and can be selected in the Sguil client by opening the ‘File’ dropdown menu and choosing ‘Change Font’.

Sguil 0.7 is getting shape quite nicely. One of the most interesting new features is the splitting up of different types of agents and the option to create ‘net groups’. This are groups of agents that Sguil considers part of the same network. You can use this to spread the agents over multiple servers, but still use it from Sguil as if it was one single sensor. For example, this way you can easily create a Snort sensor and a separate full content logging capture server. When you request the full content for a Snort event in Sguil, it will know that it needs to request the packet data from the capture server. This way you can also have multiple Snort agents without the need for capturing the same sancp and full content data over and over again.

David Bianco has written a very nice guide for installing Sguil 0.7 on Redhat Enterprise 4. I used this guide to install the server and sensor on a Debian Etch installation. The main difference is that I used Debian packages where ever possible. These packages could be used:

mysql-server
p0f
tcpflow
tcllib
mysqltcl
tcltls
tcl8.3
tclx8.3

Important: do not use the tcl8.4 package. It is not compatible with Sguil and will produce the following message:

ERROR: This version of tcl was compile with threading enabled. Sguil is NOT compatible with threading.
SGUILD: Exiting…

You can get Sguil 0.7 CVS by checking out the latest CVS version:

cvs -d:pserver:anonymous@sguil.cvs.sourceforge.net:/cvsroot/sguil login
cvs -d:pserver:anonymous@sguil.cvs.sourceforge.net:/cvsroot/sguil co sguil

I will update Modsec2sguil soon!

I’ve just released version 0.7 of Modsec2sguil, the set of perl scripts to feed ModSecurity alerts to the Sguil NSM system. The main change of this release is that it adds support for alerts produced by ModSecurity 2.x, while 1.9.x remains to be supported. Next to this the conversion between ModSecurity’s severity and Snort’s priority was fixed, so alerts should show up in the right pane in Sguil again.

Please give this release a try and let me know how it works for you!

Download it here: http://www.inliniac.net/files/modsec2sguil-0.7.tar.gz

ModSecurity 2 has been out for a while now, and although I have played with it some, I never found some time to upgrade my own servers. The upgrading generally went quite smooth, even though ModSecurity 2 changed quite a bit.

First of all there are now 5 phases where you can filter. Actually, one of them only applies to the logging, so you can filter in 4 phases. The phases are headers and body for both request and response traffic. Filtering on specific URIs can be done in phase 1 (request headers), while inspecting a POST payload requires phase 2 (request body).

Next, some shortcuts where removed. In 1.9.4 there was a variable called POST_PAYLOAD, that enabled the user to match against payloads from POST requests easily. Now there is REQUEST_BODY, but since that can be part of non-POST requests as well, you have to use:

SecRule REQUEST_METHOD “POST” chain
SecRule REQUEST_BODY “evil”

instead of:

SecFilterSelective POST_PAYLOAD “evil”

One other change is visible above already. The keyword to create a rule has been changed from SecFilterSelective to SecRule. Many rules can be converted by just replacing the keyword, but certainly not all. A simple find/replace should not be done without a manual review!

I use a number of custom rules to protect certain parts of my server, so I needed to convert my rules. For most of them it was simply enough to replace SecFilterSelective with SecRule. For a few I had to replace the OUTPUT_STATUS variable with the RESPONSE_STATUS variable, as it is called now.

For one rule however, I had quite some problems to get it running correctly. This was the rule in 1.9.4 syntax:

# block wp-login.php
SecFilterSelective REMOTE_ADDR “!10\.1\.1\.1″ chain
SecFilterSelective REQUEST_URI “/wp-login.php” \
log,deny,redirect:http://www.inliniac.net/nologin.html

This rule makes sure only 10.1.1.1 can open the login page, everyone else is redirected to a simple html page containing a ‘Access Denied - Logins disabled’ message. I converted it to the following:

SecRule REMOTE_ADDR “!10\.1\.1\.1″ chain
SecRule REQUEST_URI “/wp-login.php” \
log,deny,redirect:http://www.inliniac.net/nologin.html

Guess what? It didn’t work. I’ve spend quite some time trying all kinds of variations of the rule, and finally I found out what the issue is. In 1.9.4 the rule actions, like deny, redirect etc could be in the final rule of a series of chained rules. With 2.0.4 this doesn’t work correctly. So when I changed the rules to the following, it worked:

# block wp-login.php
SecRule REMOTE_ADDR “!10\.1\.1\.1″ \ “chain,phase:1,log,deny,redirect:http://www.inliniac.net/nologin.html”
SecRule REQUEST_URI “/wp-login\.php$”

I haven’t looked into this further to find out whether this is a bug or a feature.

The last thing that was interesting is the modsec2sguil script. There have been some changes to the alert files. So expect a new version of the script soon!

Geek00L has created a OpenBSD 4.0 based QEMU image called OpenNSM which I decided to give a try. Getting the networking part to work with my Linux host was a bit of a puzzle, so i’m writing it down here. Most of the steps were taken from the unofficial qemu wiki page here, but not all of them were necesarry.

First of all, I had to use the QEMU version 0.8.2 from the QEMU website, because the 0.8.0 version from my Ubuntu Dapper workstation didn’t work. With the latter version the OpenBSD bootup would just hang at the message ‘clock: Unknown CMOS layout’.

For the networking to work, I needed to setup a bridge between the actual NIC and the tap interface that QEMU was going to talk to. First the tap:

Next the bridge setup on eth1. The tap was added to it later.

brctl addbr br0
ifconfig eth1 0.0.0.0 up
brctl addif br0 eth1

Added an IP Address to the bridge in the subnet connected to the eth1 NIC.

After this I setup the file /etc/qemu-ifup. This file is called by QEMU on startup to setup the tap networking.

Now I was ready to start the image:

Once the image was booted, I made the corrections described by Geek00L here. Finally, I ran /root/nsm-scripts/net-config.sh in the OpenBSD installation to setup the system with a unique IP Address. I chose 192.168.2.13. After this I could use networking from my OpenNSM installation

I’ve just release a new version of modsec2sguil, the set of Perl scripts that feeds ModSecurity alerts to Sguil. No new features, but many changes ‘under the hood’. I’ve created two modules, ModsecAlert and SguilBarnyardComms. These can be used in a Object Oriented way to parse ModSecurity events and communitcate a Sguil sensor agent.

It would be interesting to see if the SguilBarnyardComms module could be connected with the work of Jason Brevnik of SourceFire, who wrote a Barnyard replacement in Perl. If I have some spare time, I will have a look at this.

After this release, I want to look at bypassing the sensor_agent altogether and instead connect directly to the Sguil server. Bamm Visscher has plans to redesign parts of the agent - server communications. In the new ideas Sguil moves away from the one monolithic sensor_agent. Instead, different tasks will have their own agent which connects directly to the sguil server. For example a sancp sensor, a snort sensor, a pcap sensor, etc. Next, a number of the same sensors would be able to share a common id, called network id. This way the user can ask a transcript for an alert produced by a modsec sensor. It is my intention to create a Perl module or library that will make creating new agents for this setup easy.

Anyway, thats the future, modsec2sguil 0.6 is ready for your testing right now! Let me know how it works for you!

Download it here: http://www.inliniac.net/files/modsec2sguil-0.6.tar.gz

I have been writing about getting ModSecurity alerts into Sguil before. Today I can finally release a first public version. It’s pretty crude, but it WorksForMe(tm).

The release can be found here. If you are interested, please try it. There is some documentation in the archive.