Incident Handling with Splunk
Learn to use Splunk for incident handling through interactive scenarios. Practice detecting and responding to real-world threats using Splunk’s search and analysis capabilities.
Incident Handling Scenario
In this exercise, we will investigate a cyber attack in which the attacker defaced an organization’s website. This organization has Splunk as a SIEM solution setup. Our task as a Security Analysis would be to investigate this cyber attack and map the attacker’s activities into all 7 of the Cyber Kill Chain Phases. It is important to note that we don’t need to follow the sequence of the cyber kill chain during the Investigation. One finding in one phase will lead to another finding that may have mapped into some other phase.
Cyber Kill Chain
We will follow the Cyber kill Chain Model and map the attacker’s activity in each phase during this Investigation. When required, we will also utilize Open Source Intelligence (OSINT) and other findings to fill the gaps in the kill chain. It is not necessary to follow this sequence of the phases while investigating.
- Reconnaissance
- Weaponization
- Delivery
- Exploitation
- Installation
- Command & Control
- Actions on Objectives
Scenario
A Big corporate organization Wayne Enterprises has recently faced a cyber-attack where the attackers broke into their network, found their way to their web server, and have successfully defaced their website http://www.imreallynotbatman.com. Their website is now showing the trademark of the attackers with the message YOUR SITE HAS BEEN DEFACED as shown below.
They have requested “ US “ to join them as a Security Analyst and help them investigate this cyber attack and find the root cause and all the attackers’ activities within their network.
The good thing is, that they have Splunk already in place, so we have got all the event logs related to the attacker’s activities captured. We need to explore the records and find how the attack got into their network and what actions they performed.
This Investigation comes under the Detection and Analysis phase.
Splunk
During our investigation, we will be using Splunk as our SIEM solution. Logs are being ingested from webserver/firewall/Suricata/Sysmon etc. In the data summary tab, we can explore the log sources showing visibility into both network-centric and host-centric activities.To get the complete picture of the hosts and log sources being monitored in Wayne Enterprise, please click on the Data summary and navigate the available tabs to get the information.
Interesting log Sources
Some of the interesting log sources that will help us in our Investigation are:
| Log Source | Details |
|---|---|
wineventlog | Contains Windows Event logs |
winRegistry | Logs related to registry creation, modification, or deletion |
XmlWinEventLog | Sysmon event logs – important from an investigation perspective |
fortigate_utm | Fortinet Firewall logs |
iis | IIS web server logs |
Nessus:scan | Results from the Nessus vulnerability scanner |
Suricata | Alerts from Suricata IDS, showing what was triggered and why – critical for investigations |
stream:http | Network flow logs related to HTTP traffic |
stream:DNS | Network flow logs related to DNS traffic |
stream:icmp | Network flow logs related to ICMP traffic |
Note: All the event logs that we are going to investigate are present in index=botsv1
Now that we know what hosts we have to investigate, what sources and the source types are, let’s connect to the lab and start Investigating .
Room Machine
Before moving forward, deploy the machine. When you deploy the machine, it will be assigned an IP Machine IP : MACHINE_IP . The machine will take up to 3-5 minutes to start.
Disclaimer
As this is a public dataset released by Splunk, which depicts a realistic scenario, it is advised that this dataset may contain profanity, slang, vulgar expressions, and/or generally offensive terminology. Please use with discretion.
Reconnaissance Phase
Reconnaissance is an attempt to discover and collect information about a target. It could be knowledge about the system in use, the web application, employees or location, etc.
We will start our analysis by examining any reconnaissance attempt against the webserver imreallynotbatman.com. From an analyst perspective, where do we first need to look? If we look at the available log sources, we will find some log sources covering the network traffic, which means all the inbound communication towards our web server will be logged into the log source that contains the web traffic. Let’s start by searching for the domain in the search head and see which log source includes the traces of our domain.
Search Query: index=botsv1 imreallynotbatman.com
Search Query explanation: We are going to look for the event logs in the index “botsv1” which contains the term imreallynotbatman.com
Here we have searched for the term imreallynotbatman.com in the index botsv1.
In the sourcetype field, we observed that the following log sources contain traces of this search term:
Suricatastream:httpfortigate_utmiis
From the names of these log sources, it’s clear what each may contain. Every analyst may take a different approach to investigating a scenario.
Our first task is to identify the IP address attempting reconnaissance activity on our web server.
A logical starting point is to examine web traffic coming into the network.
So, we begin by investigating the log source stream:http, which contains HTTP traffic logs.
Look specifically at the src_ip field in the left panel — this field contains the source IP address found in the logs.
🔍 Search Query
index=botsv1 imreallynotbatman.com sourcetype=stream:http
Search Query Explanation: This query will only look for the term imreallynotbatman.comin the stream:http log source.
Note: If you don’t find the field of interest, keep scrolling in the left panel. When you click on a field, it will display all the values it finds in the logs.
So far, we have found two IPs in the src_ip field:
40.80.148.4223.22.63.114
The first IP (40.80.148.42) appears to be responsible for a higher percentage of the logs, which suggests it could be the suspicious actor.
To confirm:
- Click on each IP one by one — it will be added to the search query.
- Review the logs for each IP.
- Determine which one is performing reconnaissance.
🔍 Deep Dive into 40.80.148.42
To further confirm suspicion about 40.80.148.42:
- Click on the IP to filter logs.
- Examine important fields like:
- User-Agent
- POST requests
- URIs
These fields can reveal the nature of the traffic coming from that IP and help confirm whether it’s part of a reconnaissance attempt.
Validate the IP that is scanning
So what do we need to do to validate the scanning attempt? Simple, dig further into the weblogs. Let us narrow down the result, look into the suricata logs, and see if any rule is triggered on this communication.
Search Query: index=botsv1 imreallynotbatman.com src=40.80.148.42 sourcetype=suricata
Search Query Explanation: This query will show the logs from the suricata log source that are detected/generated from the source IP 40.80.248.42
We have narrowed our search on the src IP and looked at the source type suricata to see what Suricata triggered alerts. In the right panel, we could not find the field of our interest, so we clicked on more fields and searched for the fields that contained the signature alerts information, which is an important point to note.
Challenge
One Suricata alert highlighted the CVE value associated with the attack attempt. What is the CVE value?
What is the CMS our web server is using?
What is the web scanner the attacker used to perform the scanning attempts?
What is the IP address of the server imreallynotbatman.com?
Exploitation Phase
The attacker needs to exploit the vulnerability to gain access to the system/server.
In this task, we will look at the potential exploitation attempt from the attacker against our web server and see if the attacker got successful in exploiting or not.
To begin our investigation, let’s note the information we have so far:
- We found two IP addresses from the reconnaissance phase with sending requests to our server.
- One of the IPs
40.80.148.42was seen attempting to scan the server with IP 192.168.250.70. - The attacker was using the web scanner Acunetix for the scanning attempt.
Count
Let’s use the following search query to see the number of counts by each source IP against the webserver.
Search Query:index=botsv1 imreallynotbatman.com sourcetype=stream* | stats count(src_ip) as Requests by src_ip | sort - Requests
Query Explanation: This query uses the stats function to display the count of the IP addresses in the field src_ip.
Additionally, we can also create different visualization to show the result. Click on Visualization → Select Visualization as shown below.
Now we will narrow down the result to show requests sent to our web server, which has the IP 192.168.250.70
Search Query: index=botsv1 sourcetype=stream:http dest_ip="192.168.250.70"
Query Explanation: This query will look for all the inbound traffic towards IP 192.168.250.70.
The result in the src_ip field shows three IP addresses (1 local IP and two remote IPs) that originated the HTTP traffic towards our webserver.
Another interesting field, http_method will give us information about the HTTP Methods observed during these HTTP communications.
We observed most of the requests coming to our server through the POST request, as shown below.
To see what kind of traffic is coming through the POST requests, we will narrow down on the field http_method=POST as shown below:
Search Query: index=botsv1 sourcetype=stream:http dest_ip="192.168.250.70" http_method=POST
The result in the src_ip field shows two IP addresses sending all the POST requests to our server.
Interesting fields: In the left panel, we can find some interesting fields containing valuable information. Some of the fields are:
- src_ip
- form_data
- http_user_agent
- uri
The term Joomla is associated with the webserver found in a couple of fields like uri, uri_path, http_referrer, etc. This means our webserver is using Joomla CMS (Content Management Service) in the backend.
A little search on the internet for the admin login page of the Joomla CMS will show as -> /joomla/administrator/index.php
It is important because this uri contains the login page to access the web portal therefore we will be examining the traffic coming into this admin panel for a potential brute-force attack.
Reference: https://www.joomla.org/administrator/index.php
We can narrow down our search to see the requests sent to the login portal using this information.
Search query: index=botsv1 imreallynotbatman.com sourcetype=stream:http dest_ip="192.168.250.70" uri="/joomla/administrator/index.php"
Query Explanation: We are going to add uri="/joomla/administrator/index.php" in the search query to show the traffic coming into this URI.
form_data The field contains the requests sent through the form on the admin panel page, which has a login page. We suspect the attacker may have tried multiple credentials in an attempt to gain access to the admin panel. To confirm, we will dig deep into the values contained within the form_data field, as shown below:
Search Query: index=botsv1 sourcetype=stream:http dest_ip="192.168.250.70" http_method=POST uri="/joomla/administrator/index.php" | table _time uri src_ip dest_ip form_data
Query Explanation: We will add this -> | table _time uri src_ip dest_ip form_data to create a table containing important fields as shown below:
If we keep looking at the results, we will find two interesting fields username that includes the single username admin in all the events and another field passwd that contains multiple passwords in it, which shows the attacker from the IP 23.22.63.114 Was trying to guess the password by brute-forcing and attempting numerous passwords.
The time elapsed between multiple events also suggests that the attacker was using an automated tool as various attempts were observed in a short time.
Extracting Username and Passwd Fields using Regex
Looking into the logs, we see that these fields are not parsed properly. Let us use Regex in the search to extract only these two fields and their values from the logs and display them.
We can display only the logs that contain the username and passwd values in the form_data field by adding form_data=*username*passwd* in the above search.
Search Query: index=botsv1 sourcetype=stream:http dest_ip="192.168.250.70" http_method=POST uri="/joomla/administrator/index.php" form_data=*username*passwd* | table _time uri src_ip dest_ip form_data
It’s time to use Regex (regular expressions) to extract all the password values found against the field passwd in the logs. To do so, Splunk has a function called rex. If we type it in the search head, it will show detail and an example of how to use it to extract the values.
Now, let’s use Regex. rex field=form_data "passwd=(?<creds>\w+)" To extract the passwd values only. This will pick the form_data field and extract all the values found with the field. creds.
Search Query:index=botsv1 sourcetype=stream:http dest_ip="192.168.250.70" http_method=POST form_data=*username*passwd* | rex field=form_data "passwd=(?<creds>\w+)" | table src_ip creds
We have extracted the passwords being used against the username admin on the admin panel of the webserver. If we examine the fields in the logs, we will find two values against the field http_user_agent as shown below:
The first value clearly shows attacker used a python script to automate the brute force attack against our server. But one request came from a Mozilla browser. WHY? To find the answer to this query, let’s slightly change to the about search query and add http_user_agent a field in the search head.
Let’s create a table to display key fields and values by appending -> | table _time src_ip uri http_user_agent creds in the search query as shown below.
Search Query: index=botsv1 sourcetype=stream:http dest_ip="192.168.250.70" http_method=POST form_data=*username*passwd* | rex field=form_data "passwd=(?<creds>\w+)" |table _time src_ip uri http_user_agent creds
This result clearly shows a continuous brute-force attack attempt from an IP 23.22.63.114 and 1 password attempt batman from IP 40.80.148.42 using the Mozilla browser.
Challenge
What was the URI which got multiple brute force attempts?
Against which username was the brute force attempt made?
admin
What was the correct password for admin access to the content management system running imreallynotbatman.com?
batman
How many unique passwords were attempted in the brute force attempt?
412
What IP address is likely attempting a brute force password attack against imreallynotbatman.com?
23.22.63.114
After finding the correct password, which IP did the attacker use to log in to the admin panel?
40.80.148.42
Installation Phase
Once the attacker has successfully exploited the security of a system, he will try to install a backdoor or an application for persistence or to gain more control of the system. This activity comes under the installation phase.
In the previous Exploitation phase, we found evidence of the webserver iamreallynotbatman.com getting compromised via brute-force attack by the attacker using the python script to automate getting the correct password. The attacker used the IP” for the attack and the IP to log in to the server. This phase will investigate any payload / malicious program uploaded to the server from any attacker’s IPs and installed into the compromised server.
To begin an investigation, we first would narrow down any http traffic coming into our server 192.168.250.70 containing the term “.exe.” This query may not lead to the findings, but it’s good to start from 1 extension and move ahead.
Search Query: index=botsv1 sourcetype=stream:http dest_ip="192.168.250.70" *.exe
With the search query in place, we are looking for the fields that could have some values of our interest. As we could not find the file name field, we looked at the missing fields and saw a field. part_filename{}.
Observing the interesting fields and values, we can see the field part_filename{} contains the two file names. an executable file 3791.exe and a PHP file agent.php
Next, we need to find if any of these files came from the IP addresses that were found to be associated with the attack earlier.
Click on the file name; it will be added to the search query, then look for the field c_ip, which seems to represent the client IP.
Search Query:index=botsv1 sourcetype=stream:http dest_ip="192.168.250.70" "part_filename{}"="3791.exe"
Was this file executed on the server after being uploaded
We have found that file 3791.exe was uploaded on the server. The question that may come to our mind would be, was this file executed on the server? We need to narrow down our search query to show the logs from the host-centric log sources to answer this question.
Search Query: index=botsv1 "3791.exe"
Following the host-centric log sources, traces of the executable 3791.exe were found in:
SysmonWinEventlogfortigate_utm
For evidence of execution, we can leverage Sysmon and look at:
EventCode=1→ Program execution
📚 Reference:
Sysmon Documentation – Microsoft Docs
Search Query: index=botsv1 "3791.exe" sourcetype="XmlWinEventLog" EventCode=1
Query Explanation: This query will look for the process Creation logs containing the term “3791.exe” in the logs.
