Yesterday the much anticipated Sguil 0.7.0 final was released, as was announced here. I’ve updated Modsec2sguil to support it. Next to this Ryan Cummings sent me a patch for supporting ModSecurity 2.5. So that is included as well. I haven’t given it much testing yet, but works on my boxes.
Get the new release here: http://www.inliniac.net/modsec2sguil/
Thank you Ryan for your contribution!
Recently I wrote about the idea to log all HTTP transactions into Sguil using my Modsec2sguil agent. I’ve implemented this in the current 0.8-dev5 release and it works very well. All events go into Sguil smoothly and I’ve not experienced slowdowns on the webserver. I’ve been running it for almost a week now, like to share the first experiences here.
I find it to be quite useful. When receiving an alert, it is perhaps more interesting to see what else was done from that ipaddress than to see what was blocked (unless you are suspecting a false positive of course). One area I find to be useful is when I’m creating rules against comment spam on this blog. By seeing all properties of a spam message I can create better rules. For example on broken user-agents or weird codes inserted into the comment field of Wordpress.
It’s easy to search and filter on HTTP response codes because the code is a part of the RT message. For example, when searching for all HTTP 500 error codes, add the following ‘WHERE’ clause to a query:
WHERE event.signature like “%MSc 500%”
This works quite fast although you best limit the query on properties like date and port as well. To get all the HTTP code 500 alerts from the last days do something like:
WHERE event.timestamp > ‘2007-08-18′ AND (event.dst_port = 80 OR event.dst_port = 443) AND event.signature like “%MSc 500%”
One thing that is disappointing is the inabillity to search in the event payloads stored in the database. Technically it’s possible to create mysql queries that search for certain strings, but this process is so slow that it’s hardly usable in practice. The problem here is that the database field containing the payload is not indexed. I’ll show the query I used here (ripped from David Bianco’s blog)
WHERE event.timestamp >= ‘2007-08-18′ AND (event.dst_port = 80 OR event.dst_port = 443) AND data.data_payload like CONCAT(”%”, HEX(”Mozilla/5.0″), “%”)
If you know a more efficient query, please let me know!
Modsec2sguil is currently configured to send alerts to Sguil. ModSecurity can be configured to log any event or transaction, including 200 OK, 302 Redirect, etc. Modsec2sguil distinguishes between alerts and other events by only processing HTTP codes of 400 and higher. Since 0.8-dev2 there is a configuration directive to prevent certain codes, such as 404, from being treated as an alert.
Now I have the following idea. Since ModSecurity can log all events with details of request headers, response headers and POST message body, it may be interesting to just send all these events to Sguil. They should not be appearing as alerts, but having them in the database can perhaps be interesting. I know using flow data and full packet captures the same data can be accessed, but having it in the database makes querying it a lot easier and longer available.
Possible problems are mostly the performance hit the webserver may take for sending all these events to Sguil and the storage requirements in Sguil’s database. I estimate the events are about 1kb in size on average, so on a busy site this may cause the database to grow very rapidly. Of course this behavior would be optional so it can be disabled.
Any thoughts on this idea?
I just uploaded a new version of Modsec2sguil. I’ve been working on it the last weeks to get it updated to Sguil 0.7. The scripts are changed all over the place. This is because in the 0.7 framework, my scripts would no longer be a replacement for Barnyard only talking to the sensor_agent on the localhost, instead now it would become a full agent talking to the Sguil server directly.
This brings some challenges. First the connection can be going over the internet, or another untrusted network, so the agent needs ssl support. Second, since the connection may be unreliable we need to be able to detect and deal with lost connections. Next to this I wanted to be able to run without superuser privileges.
The new version of modsec2sguil supports it all, and more:
So if you run Sguil 0.7-CVS and ModSecurity, go check it out at http://www.inliniac.net/modsec2sguil/
Last but not least, the agent contains a SguilAgent.pm Perl library. I hope it enables developers to easily create Perl agents for Sguil. If you need help with that, please let me know!
I’ve just released version 0.7 of Modsec2sguil, the set of perl scripts to feed ModSecurity alerts to the Sguil NSM system. The main change of this release is that it adds support for alerts produced by ModSecurity 2.x, while 1.9.x remains to be supported. Next to this the conversion between ModSecurity’s severity and Snort’s priority was fixed, so alerts should show up in the right pane in Sguil again.
Please give this release a try and let me know how it works for you!
Download it here: http://www.inliniac.net/files/modsec2sguil-0.7.tar.gz
I’ve just release a new version of modsec2sguil, the set of Perl scripts that feeds ModSecurity alerts to Sguil. No new features, but many changes ‘under the hood’. I’ve created two modules, ModsecAlert and SguilBarnyardComms. These can be used in a Object Oriented way to parse ModSecurity events and communitcate a Sguil sensor agent.
It would be interesting to see if the SguilBarnyardComms module could be connected with the work of Jason Brevnik of SourceFire, who wrote a Barnyard replacement in Perl. If I have some spare time, I will have a look at this.
After this release, I want to look at bypassing the sensor_agent altogether and instead connect directly to the Sguil server. Bamm Visscher has plans to redesign parts of the agent - server communications. In the new ideas Sguil moves away from the one monolithic sensor_agent. Instead, different tasks will have their own agent which connects directly to the sguil server. For example a sancp sensor, a snort sensor, a pcap sensor, etc. Next, a number of the same sensors would be able to share a common id, called network id. This way the user can ask a transcript for an alert produced by a modsec sensor. It is my intention to create a Perl module or library that will make creating new agents for this setup easy.
Anyway, thats the future, modsec2sguil 0.6 is ready for your testing right now! Let me know how it works for you!
Download it here: http://www.inliniac.net/files/modsec2sguil-0.6.tar.gz
I have been writing about getting ModSecurity alerts into Sguil before. Today I can finally release a first public version. It’s pretty crude, but it WorksForMe(tm).
The release can be found here. If you are interested, please try it. There is some documentation in the archive.