Kalipot – Part 3: Monitoring The Data

So, we’ve set up a kali-lookalike Cowrie honeypot, and added some iptables rules to detect nmap scans. It would be nice, though, if we had some way to see what was going on other than opening a couple of shells and running tail -f /var/log/syslog | grep “<IPT>”.

So, let’s get our data somewhere useful.

Splunk

I’m using Splunk in my home lab because it’s pretty powerful, and a single-host instance is relatively easy to set up compared to something like Graylog or an ELK stack.

Splunk offers both a limited-functionality free version (which among other things, doesn’t have authentication, so please don’t put it anywhere Internet-facing) and a 30-day trial of their enterprise product. Picking a version and installing Splunk is beyond the scope of this series, and is left as an exercise for the reader.

We have two data sources we need to get into our Splunk system. The Cowrie logs, and the log data from our iptables firewall rules.

Cowrie natively supports sending JSON-formatted data to Splunk via http(s), and iptables logs its rules with the Raspbian-default rsyslog, which can also send data directly to Splunk.

We’ll start with Cowrie.

Login to your kalipot via the real SSH connection, change to your cowrie user, and open ~/cowrie/etc/cowrie.cfg with your preferred text editor.

$ sudo su - cowrie

$ cd ~/cowrie/etc

$ vi cowrie.cfg

Then do a search for ‘output_splunk’. You should find something like the following:

 

# Splunk HTTP Event Collector (HEC) output module
# sends JSON directly to Splunk over HTTP or HTTPS
# Use 'https' if your HEC is encrypted, else 'http'
# mandatory fields: url, token
# optional fields: index, source, sourcetype, host
#
#[output_splunk]
#enabled = false
#url = https://localhost:8088/services/collector/event
#token = 6A0EA6C6-8006-4E39-FC44-C35FF6E561A8
#index = cowrie
#sourcetype = cowrie
#source = cowrie

We see that the url and token fields are mandatory, but we need to get those from Splunk itself, once we’ve configured it to receive our data.

So, go to your Splunk instance, and in the upper right hand corner, go to “Settings>Data Inputs”.

First, select “HTTP Event Collector”

The HEC is disabled by default, but it can be enabled in the Global Settings, accessible through the button at the top right of the page. On the control panel that comes up, click “Enable”, then “Save”.

Go back to the Data Inputs page, and this time, select ‘Add New” on the HTTP Event Collector line. On the next page, set a name for your kalipot. I’ve already got one kalipot on my network, so this will be kalipot 2. Then hit next.

 

Now we have the input settings screen. Where you have options for “Automatic”, “Select” and “New”, choose “Select”. A new drop-down will appear under that. Type json into the search box, and select “_json”.

In the next dropdown, select your app context as “Search & Reporting”, and select an index. I created a “security_sensors” index for my kalipots, but depending on your needs, you could just put it in the main repository.

Next hit “Review” at the top of the screen, make sure everything is set the way you want it, then hit “Submit” You will see, in sort of faded-out text, your token, in the “Token Value” field.

You can also find it on the HTTP Event Collector page, off the Data Inputs page. Make sure it’s enabled, copy down the token value, then go back to your kalipot.

In the cowrie.cfg file, go to the Splunk section and

• Uncomment the [output_splunk] line
• Uncomment the enabled line and change ‘false’ to ‘true’
• Uncomment the url line and change ‘localhost’ to the name or IP of you Splunk instance
• Uncomment the token line and set the value to your token

You should end up with something like this:

[output_splunk]
enabled = true
url = https://splunk.mydomain.com:8088/services/collector/event
token = XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
#index = cowrie
#sourcetype = cowrie
#source = cowrie

If you like, you can also uncomment and set the index, sourcetype and source lines.

Log in to your kalipot via the cowrie interface, and execute a few commands. Go back into splunk and do a search for ‘cowrie’. If everything is working, you should see nicely-formatted log entries from your cowrie instance.

Syslog

The other data we want to get off of the kalipot is the logs from our iptables rules. By default, Raspbian ships with rsyslog installed, and the iptables logs will be stored in /var/log/syslog. To make it easier to review them locally let’s save them to a different file, where they won’t be mixed in with all the other syslog traffic.

Using sudo, make a backup copy of /etc/rsyslog.conf, then open /etc/rsyslog.conf in your favorite text editor.

$ sudo cp /etc/rsyslog.conf /etc/rsyslog.conf.bak
$ sudo vi /etc/rsyslog.conf

Down near the bottom of the file, add the following line

kern.warning /var/log/iptables.log

Close the file, then restart rsyslog

$ sudo service rsyslog restart

From another machine, nmap scan your kalipott

$ nmap -v -n -Pn -sX <<your ip>> -p 5555

Back on your kalipot, cat /etc/log/iptables.log, and you should something like this:

Now it’s easier to review your iptables logs on your kalipot, but we want to get them into Splunk. The first step toward doing that is to get splunk listening for them. Go to your Splunk console, and up in the right-hand corner, select “Settings>Data Input”

On the UDP line, select “Add new”

On the next page, choose a port to listen on, and if you want, you can add the IP address or hostname of your kalipot so that only data from that host will be accepted on that port. Then hit “next” at the top of the screen.

On the next page, select “linux_messages_syslog” as your data type, leave the App Context as “Search and Reporting (search)” and Method as “IP”. If you created a custom index for your security sensors, go ahead and select that from the index dropdown.

Once everything is good, choose “Review” at the top of the screen. On the next screen, if everything is good, hit “Submit”, and now splunk is listening on the UDP port you selected.

Back on your kalipot open up your /etc/rsyslog.conf file again, and go back to the line you just added. Right under it, add the following line, with the IP or hostname of your Splunk instance, and the number of the UDP port you just opened.

kern.warning <<splunk IP>>:<<udp port>>

Save the file, and restart rsyslog.

$ sudo service rsyslog restart

On another host, scan your kalipot again:

$ nmap -v -n -Pn -sX <<your ip>> -p 5555

On your Splunk box, search for “<IPT>” and you should see something like:

If you don’t see anything, you can check and see whether your syslog data is being sent to your splunk server by using tcpdump:

$ sudo tcpdump -i <ethernet interface> port <udp port for splunk> -s0 -vv

Scan your kalipot again and see whether tcpdump registers any traffic coming to your splunk server on the designated port. If not, check your rsyslog config and restart the service.

That’s it. You now have a working cowrie honeypot, customized to look like a fresh Kali Linux install, and configured to detect nmap scans and send all the data to a Splunk instance for analysis and alerting!