Pretext

During the summer of 2016, I spent some time at a Swiss Bank for an Internship. It was one of the best work experiences I had, where I learnt a lot technically and got to meet so many interesting people.

Throughout the internship, I got in touch with several new technologies, mainly Splunk and Tableau.

Tableau is an amazing visualization tool that takes in structured data and provides drag and drop features that allow you to build your reports with a lot of flexibility.

On the other hand, Splunk is able to manage both structured/unstructured data equally well, but does not allow the report building to be as flexible when you use its default app. (The default app allows you to customize the dashboard view through XML,HTML/JS/CSS as well). Also, more flexibility comes in through custom apps/add-ons.

Introduction

In this tutorial, I will show you how to use Splunk. The example takes in event logs from a windows machine, feed it into Splunk, and do a simple search to review the event logs. Simple, isn’t it? Let’s begin!

Step 1 – Create Splunk Account and Download Splunk Enterprise

Creating an account is fairly simple, I faced some issues with my account in the beginning but the Splunk team was efficient in solving it. After your account is created, download Splunk enterprise as shown below.

Step 2 – Extracting Event Log Data

Search Event Viewer on your Windows Machine. In the image below, you will notice that I am choosing Application Logs on the left panel. Then, you may proceed to use the right panel to save all events as a TXT file. We will use this TXT file in Splunk.

Save as AppLogs.txt

Step 3 – Log In and Load Data into Splunk

This step assumes you have Splunk installed, and has been automatically hosted on your machine.

You will have a default admin account with the following credentials:
UserName: admin
Password: changeme

The Splunk WebApp login page should look like this

Upon logging in, Proceed with the following steps as shown in the slide show:

• Add Data
• Upload File
• Select source

This slideshow requires JavaScript.

Next, Splunk gives you a preview of how you can expect the data to be indexed.

Here, we note a few points

• Splunk does not recognize the sourcetype, but is able to seperate the seemingly unstructured log file into seperate events
• Splunk is not able to recognize the headers (first event logged) as not all event logs look the same
• Splunk is able to extract the time of the event
• Splunk allows you the select sourcetype (csv, apache log etc.)
• Splunk allows you to set eventbreaks in regex
• Splunk allows a lot of other settings for you to configure how you want Splunk to index your data

This slideshow requires JavaScript.

Here, we save it as a Windows Application Log file sourcetype, and proceed on. We will extract more fields through another Splunk function later on.

Next, you save the data to an index. (Create an index first, where you only need to indicate the index name. All other fields can be left blank.) Host does not matter in our example.

After selecting an index, review and submit. You should reach the page as shown below.

Step 4 – Extract Fields

We want to make sense of the data, and search by certain parameters or fields. In a normal CSV with column headers, this is very simple. However, in our example, there is no key value pair, or any other sense of what values lie in the logs. To make sense of it, we need to extract fields.

In the screenshot above, select “Extract fields”

In this page, select a sample event. This sample event will be used as your template to identify fields

In the next step (Select Method), choose regular expression

From the sample event, select and assign names to fields

This slideshow requires JavaScript.

After scrolling down, you will be able to observe that Splunk can now recognize the fields. You may then add more sample events to further strengthen your extractions. If extraction errors appear, do not worry as you can also use regex to extract fields during search.

Proceed to complete the field Extractions

Step 5 – Search and review logs

There are so many functions in search. Here I will just show a simple where clause on a field I extracted.

Just to show the visualisation tools, we do a count by

All these searches can then be saved as a report, dashboard, or even alerts based on certain rulesets.

Conclusion

There are so many ways of using Splunk. It is also widely used in the scope of security and monitoring, due to its ability and ease of manipulating unstructured data, and providing good visualisation tools.

We can also automate a lot through free Splunk Apps such as

• DBConnect (Connecting to DB interfaces)
• *nix (for Unix Servers)
• Splunk forwarder (Forward files from directories)
• Lookup Editor (Create and edit lookup tables)

and many more!

Thanks for reading, if you have any questions or need any help, feel free to reach out! (This is a very basic tutorial, I have much more to offer 😉 )