Introduction
Host firewalls, router firewalls, and Windows built-in firewall are great for stopping most exploits but the fact remains, if the bad guy wants in and they are proficient (and patient) enough, they will break through the locked door. Having a Intrusion Detection System in place can serve to alert you as soon as the black hat hackers first knock on the door.
An intrusion detection system (IDS) can perform log analysis, check file integrity, monitor policy changes, and keep an eye out for rootkit installations and alert you when a threat is first suspected. The IDS that I chose for my host was OSSES HIDS, an open source host-based Intrusion Detection System offered by Trend Micro. OSSEC is offered for most platforms including Linux, Windows, VMWare, FreeBSD, OpenBSD, NetBSD, Solaris, AIXS, HP-UX, and even MacOSX.
OSSEC is composed of multiple daemons (or services) that run silently in the background of your system. A “manager†stores the logs, events, and rules. Agents are installed on the remote systems that you wish to monitor and communicate back with the manager. OSSEC agents can monitor OS functions, firewalls, switches, routers, and much more and will silently send their findings to your centralized OSSEC manager where the accumulated results can be parsed and made available for viewing.
OSSEC uses XML based “rules†that define what exploits or conditions OSSEC should look for. The base OSSEC installation comes with a variety of rules including Unix, WordPress, and MySQL. Additional rules can be downloaded from other websites or custom written on your own. There are rules for just about any system you can think of.
The platform I installed OSSEC on was a 1and1 hosted Linux (CentOS) VPS. Although OSSEC is great at detecting intrusions, it’s not so great at reporting them in an easy to conceptualize manner. There is a OSSEC Web UI (OSSEC-WUI) that can be downloaded as an additional install and installed on the same server as OSSEC giving you a web based reporting system but it basically reads the OSSEC alert files and dumps them to you through the web browser. A solution to the poor reporting is to install a separate system to receive alerts and report on them. I installed a web front-end reporter, Splunk, on Windows and configured OSSEC to route alerts to the Splunk reporting system.
Splunk is one of the most intelligently, and beautifully designed systems I have come across. The interface is simple but provides many opportunities to drill down into the technical details of the attack vector. Splunk quite simply collects data generated by applications, servers, devices (even devices in the cloud) and provides a web based reporting system that aggregates, graphs, and reports on the cataloged events. Splunk offers an enterprise version and a free version. Most independent website owners find the free version of Splunk to be perfectly suitable for their needs.
In addition, the Splunk framework uses customizable “apps†to report on various systems. Splunk provides a database of customer-written apps for free download including a Splunk for OSSEC app that includes a variety of useful OSSEC reports.
Although the OSSEC installation instructions are pretty simple to follow, the installation of OSSEC on a VPS did not go smoothly and took quite a bit of troubleshooting to get it working properly. The installation of Splunk was much easier as was the OSSEC forwarding configuration. Of course, much firewall configuration was required to open the required ports to allow the OSSEC agents to communicate with Splunk.
Note: in many of the screenshots below, I whited out sensitive information. In screenshot with white backgrounds, fields that appear to be empty are likely filled but whited out.
Step by Step Installation
I used Putty to ssh to the server. My 1and1 hosted Linux environment did not have a compiler installed so the first step was to download and install a compiler from RHN over the Internet. Several dependencies were automatically resolved and gcc was installed with nary a problem. Use yum to install gcc. The OSSEC installation will verify the gcc installation.
yum install gcc gcc-c++ autoconf automake
Use wget to download and install OSSEC. Then untar it and start the installation.
wget http://www.ossec.net/files/ossec-hids-latest.tar.gz
tar -zxvf ossec-hids-*.tar.gz (or gunzip -d; tar -xvf)
./install.sh. Hit enter to accept [en] English as the language.
The installation will warn you and prompt for the start of the installation.
First you must select the type of installation you wish to initiate. Enter “server†and press Enter. Local could also be selected if you with to run the manager and agent on the same machine and do not intend to add other agents later.
The installation will continue to prompt you for selections. You can keep the default selection on most options. Note that if you wish to receive alerts via an email notification, you must have access to an SMTP relay. In my case, I chose not to email alerts and instead used Splunk to forward notifications. I did not have access to a SMTP host that allowed relaying and did not want to configure my virtual Linux host to relay off of a secured SMTP server. Note that this is probably not the best option because it requires Splunk to be running to send and receive alerts (hence you are relying on two systems for alert functionality vs. the single host system).
You will also be asked about the installation of various engines. I chose to install all of the engines which included integrity check (detects when files have changed), rootkit detection (detects if a black hat hacker has installed a rootkit on your system, active response (lets you execute a specific command based on the events received). I also chose to enable host-deny and firewall-drop responses which can be helpful in stopping SSHD brute force scans and portscans.
The installation will detect common log files (messages, secure, maillog, and apache error_log and access_log) and you can add others by editing the ossec.conf file located in the OSSEC/etc directory.
The installation will run for a while and finish with instructions on how to start and stop OSSEC HIDS manually. Note that the Init script is modified by default to start OSSEC HIDS during boot. Also note that even though we are installing on CentOS (which is basically Redhat), the system is reported as Redhat Linux. This is perfectly fine as all the Redhat scripts and configurations work perfectly on CentOS.
Use ifconfig to get the external ip address for your box. This will be needed later.
ifconfig -a
Start the installation again but this time choose “agent†for the installation type. You will also be prompted to enter the IP address of the OSSEC HIDS server that you just installed.
Next install an agent that will monitor system functions and communicate back to the OSSEC manager. This part was a bit confusing to me and I finally decided that the agent was not necessarily needed since the manager was running on the same machine. Your mileage may differ…
cd to the /var/ossec/bin directory and add the agent using the manage-agents app. Chose “A†to add an agent. Give the agent a friendly name and enter your server IP address.
./manage_agents
Next you will need to extract the key for the agent. Enter the ID of the agent (available OSSEC agents are listed) and a key will be generated (the large whited-out area in the screenshot below). Highlight this key and copy it to the clipboard.
Next you must import the key generated above into the sever. cd to /var/ossec/agents/bin (assuming /var/ossec/agents is the directory you chose to install the agent in – see above) and run manage_agents. Choose “I†to import and paste they key that you copied above. The agent information for the key will be displayed.
./manage_agents
Next we stop and start OSSEC using the ossec-control stop and ossec-control start commands.
ossec-control stop
ossec-control start
OSSEC is now installed and running. You can check various log files in the /var/ossec/logs directory (and sub-directories). ps –ef to see the processes (the screenshot below was taken after I installed OSSEC WUI).
ps –ef | grep ossec
Install OSSEC Web User Interface (OSSEC WUI)
First wget the ossec-wui file and untar it. Make a directory in your web root or as a subdirectory in your web site and copy the untarred files into the directory. In my configuration, to access the OSSEC web user interface, the url would be http://www.mydomain.com/ossec-wui.
wget http://www.ossec.net/files/ui/ossec-wui-0.3.tar.gz
tar zxvf ossec-wui-0.3.tar.gz
mkdir -p /var/www/vhosts/mydomain.com/website/ossec-wui
cp -rf ./ossec-wui-0.3/* /var/www/vhosts/mydomain.com/website/ossec-wui
Go to the directory you created and copied the ossec-wui files to and run setup.sh. The setup.sh script will install the ossec-wui application and secure the web directory so that a login is required to access the OSSEC web user interface.
cd /var/www/vhosts/mydomain.com/website/ossec-wui
./setup.sh
Add the ossec user to the apache group. You may need to change permissions on the ossec-wui directory to match the permissions of your other websites.
usermod -a -G ossec apache
You may receive an error stating that ossec could not find the /var/ossec/etc/shared/agent.conf file. You can manually create this file by copying the ossec.conf file to the /var/ossec/etc/shared directory. Edit the file deleting all the first lines leaving just the lines in the screenshot below. Also change the opening and closing tags from ossec.config to agent_config.
Go ahead and try to access the OSSEC WUI. You may need to restart apache and/or restart your virtual machine (or restart the Plesk container).
Depending on your server configuration and the ISP plan you have purchased, you may notice MaxClient errors in the log files. Toward the top of the http.conf file in /etc/httpd/conf, change the ServerLimit and MaxClients to 512 to correct this error. Note that this can impact server performance if you already consume a lot of resources on your virtual private server.
I also experienced a tricky problem that I tracked to a call from main.php to os_lib_handle.php where the opendir() php command was erroring. This turned out to be basedir configuration in PHP due to the fact that PHP cannot write to a lower directory level. On the 1and1 Linux VPS system, Plesk makes this tough to fix. You cannot update the http.include for a domain because Plesk will overwrite your changes each time the system is restarted. Instead, you will want to use a vhosts.conf configuration file. Create the vhosts.conf file in /var/www/vhosts/mydomain.com/conf. You can put Apache configuration options in this file just like you would http.include with the advantage being, Plesk will not overwrite changes in the vhosts.conf file. Add the lines below to your new vhosts.conf file.
<Directory /var/www/vhosts/mydomain.com/subsite-name>
<IfModule sapi_apache2.c>
php_admin_value open_basedir none
</IfModule>
<IfModule mod_php5.c>
php_admin_value open_basedir none
</IfModule>
</Directory>
Then run the following to reconfigure Plesk.
/usr/local/psa/admin/sbin/httpdmng –reconfigure-domains mydomain.com
OSSEC WUI should now be working but it may not start automatically when the system is rebooted. To fix this, copy the /var/ossec/bin/ossec-control file to /etc/init.d and rename it ossec_start. Chmod the file to 755 and then run chkconfig.
cp /var/ossec/bin/ossec-control /etc/init.d
cd /etc/init.d
mv ossec-control ossec-start
chmod 755 ossec-start
chkconfig –add ossec_start
You can run chkconfig –del to remove jobs (if you screw up) and chkconfig –list to list jobs (this might be a good time to remove any junk you don’t need running).
Install Splunk
Splunk gives you a much better front-end than the standard OSSEC WUI. Plus, with Splunk, you can monitor just about any event sink you can think of including Windows logs and events, other firewall logs, and other machines on your local network.
Go to the Splunk web site and register as a new user (if you do not already have a Splunk account). Registration will allow you to download Splunk for free. Make sure you select the correct version for your operating system (which in my case was the Windows version). The download screen will also display the correct wget command (look in the right-hand column of the page) to download a .rpm of the Splunk installation file to install on Nix. Note that Splunk is very resource intensive. I originally installed it on my Linux VPS but it consumed so many resources I had to remove it.
Installation of the Windows version of Splunk is a no-brainer. Once you have it installed, download and install a few apps and get to know the interface.
If you choose to use Splunk for OSSEC reporting, you’ll need to set up OSSEC to forward messages to your Splunk box. You’ll need to have your external IP address of your Windows machine in hand to complete the configuration (which means you’ll have to configure any routers to allow communication to the Windows box and configure Windows Firewall to let those communications come through).
In /var/ossec/etc, add the following to the bottom of your ossec.conf file. The <server> tag will contain the ip address of your Windows machine and port can be any number you want (e.g. 11005).
Restart OSSEC and give it a bit to start forwarding messages. Don’t forget, you’ll need to make sure the appropriate holes are poked in your firewall(s).
You’ll now need to add a “data input†in Splunk (choose UDP) to receive the messages. After a while you’ll see the OSSEC machines arriving in your Splunk application.
A great Splunk application for OSSEC is “Splunk for OSSECâ€. Click the apps link at the top of the page and choose to install a new application. Download and install Splunk for OSSEC (and any other apps you think will be useful). You can download the tarball to your windows machine and then open it in Splunk to install.
Monitor AT&T U-Verse Firewall through Splunk
Lastly, in addition to monitoring potential attacks against my Linux Virtual Private System, I set up Splunk to monitor connections to my AT&T U-Verse router, which includes a nice built-in firewall. I include this example for two reasons: (1) it’s pretty cool to be able to monitor your U-Verse firewall and (2) Splunk running on a machine with insufficient resources seems to be prone to hosing up. Â If your OSSEC UDP forwarding of events cannot keep up, you can configure your syslog daemon (i.e. Kiwi) to accept forwarded events from the remote OSSEC instance and then configure Splunk to monitor the Kiwi syslog directory.
To monitor the U-Verse firewall log, you must configure your 2Wire router to forward syslog events to a syslog daemon (we will use the free Kiwi Syslog Viewer as our syslog daemon) which will record the events to a log file that Splunk can access and include in its database.
Configure the 2Wire router
To configure the 2Wire router, navigate to Settings –> Diagnostics and check the Enable Syslog checkbox. Put in the internal IP address of the machine that is hosting your Kiwi Syslog Viewer which will probably be installed on the same machine you installed Splunk on. Add a port (514 is common) and if you’d like, enable throttling. Save the changes and you’re done with the 2Wire configuration.
Install Kiwi
Kiwi Syslog Viewer can be downloaded from the Kiwi web site. The download will be an enterprise evaluation version of the product that will shift to the free version once the free trial runs out. Since all we want to do is create a daemon to listen for 2Wire firewall events, the free version will work just fine.
Install Kiwi as a Windows service.
Use the local system account to run the Kiwi service.
Uncheck Install Kiwi Syslog Web Access. This feature is only available in the purchased licensed version of Kiwi or during the evaluation period. We do not need this so no use install the feature.
Click Next through the final dialog boxes and install Kiwi.
After installation completes, the Kiwi Syslog Service Manager application will automatically start. Click FIle –> Setup to access the configuration options.
Make sure the Log to file option is checked and note the location of the log file. The Kiwi log file format that it defaults to works fine in Splunk.
Set up Splunk
Navigate to the Data Inputs in Splunk.
Select Files and Directories.
Add a new Data Input file
Browse to the logs directory you noted in the Kiwi options, and select the SyslogCatchAll.txt log file. Chose to start a new source type and click continue to give a friendly name to the new source type. Save and Splunk will begin capturing AT&T U-Verse firewall log events.