Splunk Enterprise and Machine Data – How to Build your own instance to add and crunch data

Pretext

During the summer of 2016, I spent some time at a Swiss Bank for an Internship. It was one of the best work experiences I had, where I learnt a lot technically and got to meet so many interesting people.

Throughout the internship, I got in touch with several new technologies, mainly Splunk and Tableau.

Tableau is an amazing visualization tool that takes in structured data and provides drag and drop features that allow you to build your reports with a lot of flexibility.

On the other hand, Splunk is able to manage both structured/unstructured data equally well, but does not allow the report building to be as flexible when you use its default app. (The default app allows you to customize the dashboard view through XML,HTML/JS/CSS as well). Also, more flexibility comes in through custom apps/add-ons.

Introduction

In this tutorial, I will show you how to use Splunk. The example takes in event logs from a windows machine, feed it into Splunk, and do a simple search to review the event logs. Simple, isn’t it? Let’s begin!

Step 1 – Create Splunk Account and Download Splunk Enterprise

Creating an account is fairly simple, I faced some issues with my account in the beginning but the Splunk team was efficient in solving it. After your account is created, download Splunk enterprise as shown below.

Splunk enterprise.png

Step 2 – Extracting Event Log Data

Search Event Viewer on your Windows Machine. In the image below, you will notice that I am choosing Application Logs on the left panel. Then, you may proceed to use the right panel to save all events as a TXT file. We will use this TXT file in Splunk.

Event Viewer

Save as AppLogs.txt

saveas

Step 3 – Log In and Load Data into Splunk

This step assumes you have Splunk installed, and has been automatically hosted on your machine.

You will have a default admin account with the following credentials:
UserName: admin
Password: changeme

The Splunk WebApp login page should look like this

Login

Upon logging in, Proceed with the following steps as shown in the slide show:

  • Add Data
  • Upload File
  • Select source

This slideshow requires JavaScript.

Next, Splunk gives you a preview of how you can expect the data to be indexed.

Here, we note a few points

  • Splunk does not recognize the sourcetype, but is able to seperate the seemingly unstructured log file into seperate events
  • Splunk is not able to recognize the headers (first event logged) as not all event logs look the same
  • Splunk is able to extract the time of the event
  • Splunk allows you the select sourcetype (csv, apache log etc.)
  • Splunk allows you to set eventbreaks in regex
  • Splunk allows a lot of other settings for you to configure how you want Splunk to index your data

This slideshow requires JavaScript.

Here, we save it as a Windows Application Log file sourcetype, and proceed on. We will extract more fields through another Splunk function later on.

Next, you save the data to an index. (Create an index first, where you only need to indicate the index name. All other fields can be left blank.) Host does not matter in our example.

set index and source.png

After selecting an index, review and submit. You should reach the page as shown below.

uploadsuccess.png

Step 4 – Extract Fields

We want to make sense of the data, and search by certain parameters or fields. In a normal CSV with column headers, this is very simple. However, in our example, there is no key value pair, or any other sense of what values lie in the logs. To make sense of it, we need to extract fields.

In the screenshot above, select “Extract fields”

In this page, select a sample event. This sample event will be used as your template to identify fields

extractfield.png

In the next step (Select Method), choose regular expression

regex.png

From the sample event, select and assign names to fields

This slideshow requires JavaScript.

After scrolling down, you will be able to observe that Splunk can now recognize the fields. You may then add more sample events to further strengthen your extractions. If extraction errors appear, do not worry as you can also use regex to extract fields during search.

Proceed to complete the field Extractions

Step 5 – Search and review logs

There are so many functions in search. Here I will just show a simple where clause on a field I extracted.

search

Just to show the visualisation tools, we do a count by

countby.png

All these searches can then be saved as a report, dashboard, or even alerts based on certain rulesets.

Conclusion

There are so many ways of using Splunk. It is also widely used in the scope of security and monitoring, due to its ability and ease of manipulating unstructured data, and providing good visualisation tools.

We can also automate a lot through free Splunk Apps such as

and many more!

Thanks for reading, if you have any questions or need any help, feel free to reach out! (This is a very basic tutorial, I have much more to offer 😉 )

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s