I wanted to ship my suricata alerts to my splunk instance. You could probably use syslog but the json won’t show up nicely in splunk: JSON from Syslog using Splunk, Is it possible to parse an extracted field as json if the whole log line isn’t json?, and Sending rsyslog JSON format. There are a couple of work arounds but they are kind of painful. I heard of the Splunk forwarder and I wanted to try it out, so I decided to go that route.

Download the SplunkForwader

After logging into the splunk site you can get the wget command for the download:

<> wget -O splunkforwarder-6.4.2-00f5bb3fa822-freebsd-10.1-amd64.txz 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=freebsd&version=6.4.2&product=universalforwarder&filename=splunkforwarder-6.4.2-00f5bb3fa822-freebsd-10.1-amd64.txz&wget=true'

Then we can just scp it to the pfSense machine:

┌─[elatov@kerch] - [/home/elatov] - [2016-08-02 10:57:12]
└─[0] <> scp splunkforwarder-6.4.2-00f5bb3fa822-freebsd-10.1-amd64.txz pfl:
root@pfl.kar.int's password:
splunkforwarder-6.4.2-00f5bb3fa822-freebsd-10 100%   12MB  12.2MB/s   00:01

Install the SplunkForwarder

Now let’s go ahead and extract the software. First ssh to the pfSense machine and then run the following:

[2.3.2-RELEASE][root@pf.kar.int]/root: tar xJfP splunkforwarder-6.4.2-00f5bb3fa822-freebsd-10.1-amd64.txz

The install instructions are covered here.

Configure SplunkForwarder

Next let’s create a configuration to define where to forward the data to, which will be our splunk server. This is done by creating the outputs.conf file. Here are the basic configuration settings that go into that file. I ended up creating the following:

[2.3.2-RELEASE][root@pf.kar.int]/root: cat /opt/splunkforwarder/etc/system/local/outputs.conf


Next let’s configure what we are going to send. I was just planning on sending the suricata alerts, which are in a json file here:

[2.3.2-RELEASE][root@pf.kar.int]/root: ls -l /var/log/suricata/suricata_re034499/eve.json
-rw-r-----  1 root  wheel  27974 Aug  2 11:43 /var/log/suricata/suricata_re034499/eve.json

So here is the inputs.conf file I ended up with:

[2.3.2-RELEASE][root@pf.kar.int]/root: cat /opt/splunkforwarder/etc/system/local/inputs.conf

Examples for the inputs.conf are here.

Configure Splunk Server/Indexer to Accept Data From Splunk Forwarder

Then on the indexer, I created a sourcetype (This is covered in here and here):

root@kerch:~# tail -6 /opt/splunk/etc/system/local/inputs.conf
disabled = false
sourcetype = suricata
connection_host = none
compressed = true

And then I marked it as json with props.conf:

root@kerch:~# cat /opt/splunk/etc/system/local/props.conf
KV_MODE = json

On the indexer, I restarted the splunk service:

root@kerch:~# systemctl restart splunk

And confirmed I was now listening on port 9997 tcp:

┌─[elatov@kerch] - [/home/elatov] - [2016-08-03 04:19:26]
└─[0] <> sudo /opt/splunk/bin/splunk list inputstatus -input 9997
Cooked:tcp :
                time opened = 2016-08-03T11:10:28-0600

tcp_cooked:listenerports :

Don’t forget your firewall:

root@kerch:~# sudo iptables -L -n -v | grep 9997
    0     0 ACCEPT     tcp  --  *      *            state NEW tcp dpt:9997

Last test you can do is a telnet from the pfSense to the splunk indexer:

[2.3.2-RELEASE][root@pf.kar.int]/root: telnet 9997
Connected to kerch.kar.int.
Escape character is '^]'.

Confirm Data is Sent From Forwarder

Now let’s start the forwarder:

[2.3.2-RELEASE][root@pf.kar.int]/root: /opt/splunkforwarder/bin/splunk start

Do you agree with this license? [y/n]: y

This appears to be your first time running this version of Splunk.

Splunk> Finding your faults, just like mom.

Checking prerequisites...
        Checking mgmt port [8089]: open
                Creating: /opt/splunkforwarder/var/lib/splunk
                Creating: /opt/splunkforwarder/var/run/splunk
                Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n
                Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css
                Creating: /opt/splunkforwarder/var/run/splunk/upload
                Creating: /opt/splunkforwarder/var/spool/splunk
                Creating: /opt/splunkforwarder/var/spool/dirmoncache
                Creating: /opt/splunkforwarder/var/lib/splunk/authDb
                Creating: /opt/splunkforwarder/var/lib/splunk/hashDb
New certs have been generated in '/opt/splunkforwarder/etc/auth'.
        Checking conf files for problems...
        Checking default conf files for edits...
        Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-6.4.2-00f5bb3fa822-FreeBSD9-amd64-manifest'
        All installed files intact.
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Generating a 1024 bit RSA private key
writing new private key to 'privKeySecure.pem'
Signature ok
Getting CA Private Key
writing RSA key

Let’s confirm it’s running:

[2.3.2-RELEASE][root@pf.kar.int]/root: /opt/splunkforwarder/bin/splunk status
splunkd is running (PID: 14737).
splunk helpers are running (PIDs: 14995).

If all is well you should see the following in the logs:

[2.3.2-RELEASE][root@pf.kar.int]/root: tail -f /opt/splunkforwarder/var/log/splunk/splunkd.log
08-02-2016 12:47:52.226 -0600 WARN  TcpOutputProc - Forwarding to indexer group my_indexers blocked for 1200 seconds.
08-02-2016 12:48:07.049 -0600 INFO  TcpOutputProc - Connected to idx=
08-02-2016 12:48:07.129 -0600 INFO  TailReader -   ...continuing.

And of course if you check out the splunk UI you will see the json alerts in there:


Enable Auto Start of Splunk Forwarder

if you were on another FreeBSD system you can enable it to start on boot with the following command:

┌─[elatov@moxz] - [/home/elatov] - [2016-08-03 10:35:40]
└─[0] <> sudo /opt/splunkforwarder/bin/splunk enable boot-start

Init script installed at /etc/rc.d/splunk.
Init script is configured to run at boot.

But with pfSense we can add the start command as a shellcmd, the process is covered here, here is what I added:


Published by Karim Elatov

24 February 2017