Inliniac

After the successful test with the Perl script to add ModSecurity alerts to Sguil, I have been working on a more robust implementation, also in Perl. Let me first describe the basic setup. The setup works with two scripts. The first places links to event files into a special queue directory. The second reads the links from that directory, parses them and sends the alerts among these events to Sguil. After that, the links are removed.

In ModSecurity there is a logging method called concurrent. What it does is that it creates a directory structure based on date and time. For each event a file is created in the directory structure, with the details of the event. It also writes a logfile called index, that adds one line for every event, with location of the file in the date/time directory layout.

ModSecurity also supports using a external script to take care of this indexing task, which is what the script modsec-auditlog-collector in the ModSecurity source tarball does. I have modified this Perl script to create a link to the actual event file in a special queue directory. The links all have a name like: modsec.log.-timestamp-, like Snort uses with unified logging. I have chosen to use links because I think it might be faster than copying, but I have yet to benchmark that.

The second script takes care of processing the links and communicating with Sguil. It basically has the following workflow. It gets a list of all the files in the queue. Each file in the queue is parsed, then it is checked whether it is an alert. If so, the data is preprocessed to make sure it is what Sguil expects. Next it is send to Sguil. Finally, all the files are removed.

ModSecurity has a different rule syntax than Snort and sees a different type of data. Snort sees packets, and thus knows about TCP checksums, sequence numbers, etc. ModSecurity works on a higher level, so it has no access to that data. One thing that ModSecurity alerts do include is the action taken: blocked or redirected and with what HTTP code. If available the rule msg is used for the Event Message, otherwise the ModSecurity message with (a part of) the rule that matched. Below is a screenshot that shows both, and also the HTTP code.

The current implementation acts as a drop-in replacement of Barnyard. This way no changes to Sguil are necessary. Currently this means that Barnyard cannot run on the same sensor, although multiple instances of sensor_agent on a host could fix that. Sguil author Bamm Visscher suggested I can try to connect to the Sguil server directly instead of via the sensor_agent to fix this. I will probably have a look at that later.

Another issue is the way the alert data is presented in the Sguil interface. Currently I attach the entire alert file to the alert that is being send to Sguil, as the packet payload. However the Sguil packet payload view is not very usable for this kind of data, as can be seen on the screenshot below.

So here is room for much improvement. Ideally Sguil would be able to show the uri, the post payload if applicable, useragent, etc. I’ve never worked with TCL/TK, but I did use Perl/TK in the past, so that might help me a little if I decide to give it a shot.

I expect a public beta version to be ready some time next week, although I have not yet decided how to release it. I could create a project on SourceForge or just have a simple webpage. I just haven’t decided yet. Stay tuned…

A few days ago a new vulnerability was reported in Tikiwiki 1.9.x, the software I use for the Vuurmuur Wiki. Luckily, the Snort.org Community rules quickly had a rule for detecting the attack. Because I also run ModSecurity on the webserver, i wanted to have protection there as well. This rule should block the attack:

SecFilterSelective POST_PAYLOAD “jhot.php” “log,deny,status:403,msg:’LOCAL tikiwiki jhot.php attempt’”

Let’s see if I ever get a hit on it. An update for Tikiwiki as been released, so that should fix the issue completely.

I’m a huge fan of both Sguil and ModSecurity, but sadly the alerts generated by ModSecurity can’t show up in Sguil… or… can they? Well, if it all works out, soon they can!

Today I have hacked together a perl script that emulates barnyard for ModSecurity. It very much in a proof-of-concept phase, but it somewhat works already, at least good enough so i can show this screenshot.

Nice huh? It is not ready for release yet, but when it is i’ll announce it here. I plan to release it under the GPL. Sguil author Bamm Visscher told me that the next release of Sguil will have support for having barnyard and PADS on the same sensor. By then, i hope that ModSecurity can be added to that list!

Today, David Bianco showed me a way of creating SQL queries that I didn’t even know was possible. This is a technique with which it is possible to query the payload of Snort events in the Sguil database. These payloads are stored by Snort when it alerts and is the payload the actual rule triggered on. David showed a nice example of retrieving url’s for PHP url inclusion attacks.

I have written before about my usage of Mod_Security. I let Mod_Security respond with a 403 Forbidden message. Sadly, the alert generated by Mod_Security can not be displayed in Sguil. As somewhat of a replacement for that, I let Snort alert on the 403 Forbidden messages, so i can see in Sguil that Mod_Security blocked something. The disadvantage of this is that the 403 alert in itself does not contain much info. The sheer number of 403’s makes inspecting every single one with requesting a transcript a bit to much work.

This is where the new query comes in. The query creates a list of url’s that the server reported to be 403 Forbidden. Interesting is that we are not looking at an attack, but at an attack response. This means the the attackers IP is actually the destination IP.

The result is this:

As you can see, the first six results are from the comment spam I wrote about earlier. I left the source IP out because there is only one webserver. The query can easily be extended to show source IP’s as well.

Lately the wiki of my Vuurmuur project has been receiving quite a lot of comment spam. Although removing the spam manually is boring work, i still don’t really mind the spam, because it enables me to practice with ModSecurity rules to fight it off. So far, the spam seems to be following a pattern, in which the spam is posted by bots, and has the same general layout for longer periods of time. That makes it worthwhile to spend time on creating rules against it. Yesterday a new type of spam emerged on the wiki. The following audit_log is for one of them. I had to slightly edit it for layout reasons.

A general rule against the comment spam would probably not be that hard. Just blocking “http://” would probably to the trick. However, I’m not quite willing to do this, since people might actually post real and interesting links to the wiki. I noticed that the above comment and the 30 other already posted messages all contained the uppercase word URL, so i decided to block on that. Below are the rules, which as you can see are quite an improvement over these rules, because they now only match on actual comments posts

An Apache restart and just wait… but not for long:

This morning there were 59 attempts blocked already which i would have to remove manually without these rules. So taking the time to setup the rules really pays off.

Yesterday, Philippe Baumgart showed me that my obscurity setup is not yet perfect. In fact, he could very easily enter an URL that didn’t exist and caused the webserver behind my proxy to respond with a 404. In this 404 the name and the version of the webserver were exposed.

After some testing i found that adding the following to my config worked very well.

# enable output scanning in Mod Security.
SecFilterScanOutput On

# hide outgoing 404 by webserver behind proxy
SecFilterSelective OUTPUT_STATUS 404 deny,status:404

This catches outgoing 404 errors, and replaces them by the 404 from the proxy. For some reason, this still didn’t exaclty look like the 404 from the proxy itself, because it contained a message that an additional 404 was encountered. I solved this by changing the ErrorDocument in de Apache config:

ErrorDocument 404 “<html><head><title>404 Not Found</title></head><body><h1>Not Found</h1><p>The requested URL was not found on this server.</p></body></html>”

After this, there was no longer any difference between 404’s produced by the proxy and by the webserver behind it.

Next, Phil showed me that i also leaked my version number of PHP. By using Wordpress hiding the fact that I use PHP is impossible and pointless, but hiding the exact version still looks like a good idea. The version was leaked in the header from a server response: X-Powered-By: PHP/4.4.1build1. Solving this requires the mod_header module again:

# unset X-Powered-By to prevent leaking the PHP version
Header unset X-Powered-By

This hides this header. Thanks to Phil for doing some pen-testing

People can reach my webserver in three ways: by my domain inliniac.net, by the hostname connected to my dsl, and by my ipaddress. What i now wanted is setup the proxy in such a way, that only people visiting inliniac.net would be proxied to the webserver.

Blocking requests that are IP based instead of name based have an important advantage. IP based requests are mostly used by scantools and other forms of malicious traffic that just attempt connecting to port 80 on large IP-ranges. So this way one should be able to keep a lot of crap like worm traffic out.

Implementing this turned out to be a little more complicated than i thought. Essentially the only way i got it working was by creating three virtual hosts and put the proxy configuration in the virtual host for inliniac.net. The other two virtual hosts just deny access, so visiting them gives a 403.

Willam Metcalf told me that the server signature rewrite function from ModSecurity does not hide the original server signature in all cases. In normal 200 and 300 series responses, the original signature is still in the header. Enabling the ‘header’ module and adding this to your config, helps:

Header set Server “Apache”

This way people will know you run Apache, but won’t know which version.

Another nice feature of ModSecurity is rule based redirection. Lets say i want to block visitors of my website from opening the login page of wordpress, /blog/wp-login.php. I could of course just deny access to it, so the visitor gets a 403 error. This works fine, however sometimes you might want to use a more userfriendly message, for example: ‘Due to maintainance logins are currently disabled’.

To do this i first created a very simple html file called nologin.html, and placed it in the webroot of the server. Then i added the following rules to Mod_Security:

# block wp-login.php
SecFilterSelective REMOTE_ADDR “!192\.168\.1\.2″ chain
SecFilterSelective REQUEST_URI “/wp-login.php” log,deny,redirect:http://www.inliniac.net/nologin.html

These rules check first if the ipaddress is not 192.168.1.2, then look for /wp-login.php in the URI. If it is found, the request is logged and blocked. The server then sends a redirect 302 response to the client which will open the nologin.html page. Since the original request is blocked, the webserver behind the reverse proxy never sees the request.

Ok, that’s a bit misleading, because i’m not just hiding, but also blocking and logging. What i wanted is this: I’m running awstats on my reverse proxy, but i don’t want anyone to know. So i just made the entire ‘cgi-bin’ part forbidden for everyone, so that covers the script. The fact that my webserver has a cgi-bin directory is nothing special and won’t tell you i’m using awstats. However, awstats also uses icons, and these are by default in /awstats-icon/

Now, i could have made that restricted as well, but that still would give you the information that it exists! ModSecurity to the rescue. I want only access to awstats from my workstation, so i added these rules:

SecFilterSelective REMOTE_ADDR “!192\.168\.1\.2″ chain
SecFilterSelective REQUEST_URI “/awstats-icon/” log,deny,status:404

These say: if the ipaddress of the visitor is not 192.168.1.2, and his request contains /awstats-icon/, we log it, block it, and send back a 404 error. This makes the visitor think the directory doesn’t exist on the server.

I’m not sure how easy this can be extended to giving an entire subnet access though, maybe i’ll investigate this later.

A few weeks ago i wrote that i wanted to investigate setting up a reverse web proxy with mod_security. I have now finally found a little time to do so. What surprised me was how easy it actually is!

<IfModule mod_proxy.c>

#turning ProxyRequests on and allowing proxying from all may allow
#spammers to use your proxy to send email.

ProxyRequests Off

<Proxy *>
Order deny,allow
Allow from all
#Allow from .your_domain.com
</Proxy>

# Enable/disable the handling of HTTP/1.1 “Via:” headers.
# (”Full” adds the server version; “Block” removes all outgoing Via: headers)
# Set to one of: Off | On | Full | Block
ProxyVia Block

# use to disable proxy for a specific subdir
# ProxyPass /blog/wp-admin !
ProxyPass /blog http://internalserver/blog
ProxyPassReverse /blog http://internalserver/blog

# use to disable access to a specific subdir

<Location /blog/wp-admin>
Order deny,allow
Deny from all
Allow from 192.168.1.2
</Location>

# use to force errors by the internal host to look like to be coming from the proxy host.

# disabled because it causes problems with Wordpress.

#ProxyErrorOverride On

</IfModule>

This is the entire config for the proxy part. Next, enable mod_security. As far as i can see there is nothing special about ModSecurity and reverse proxies. So just enable it like you would on a webserver.
Some useful links for setting up reverse proxies using Apache:

http://httpd.apache.org/docs/2.0/mod/mod_proxy.html

http://www.apacheweek.com/features/reverseproxies

Thats all for now!