Splunk for Beginners: Search Logs and Build a SIEM Dashboard

Splunk helps security teams collect machine data, search events and present useful findings in dashboards. In this beginner lab, you will ingest Linux authentication logs, write Splunk Processing Language searches and build a dashboard for monitoring failed SSH logins.

What Is Splunk and How Is It Used in Security?

Splunk is a data platform that indexes logs and makes them searchable. Security teams use it to investigate incidents, monitor authentication activity, build alerts and create dashboards that summarise events from servers, network devices, cloud services and security tools.

A simple diagram in words looks like this:

Log source → Forwarder or API → Splunk indexer → Search head → Dashboard or alert

Each component has a specific role:

Splunk Enterprise provides log indexing, searching and dashboards. Splunk Enterprise Security is a separate security application that adds SIEM-focused features such as correlation searches, risk-based alerting, notable events and investigation workflows. A dashboard built in Splunk Enterprise is useful for security monitoring, but it is not by itself a complete SIEM implementation.

If you are preparing for monitoring and incident-response roles, this lab supports the practical log-analysis skills covered in a Cybersecurity & SOC course. You can also review what a SOC analyst does with tools, alerts and escalation to understand where Splunk searches fit into daily operations.

What Should a Beginner Know About Splunk Data?

Splunk stores data as events inside indexes. Every event normally has a timestamp, raw message and metadata fields such as host, source, sourcetype and index.

The four important metadata fields are:

The sourcetype is especially important. It influences timestamp recognition, event breaking and field extraction. Assigning an incorrect sourcetype can produce broken timestamps or combine several log lines into one event.

Splunk searches use Splunk Processing Language, commonly called SPL. An SPL pipeline starts with a data search and passes results through commands separated by the pipe character:

index=security sourcetype=linux_secure
| stats count by host
| sort - count

Read this from top to bottom: find Linux security events, count them by host and sort the largest count first.

How Can You Create a Small Splunk Log Lab?

The quickest beginner lab is to upload a sample Linux authentication log through Splunk Web. In a longer-running environment, install the Splunk Universal Forwarder on the Linux host and send events continuously to an indexer.

Create a text file named auth-sample.log with these events:

Aug 28 09:12:10 web01 sshd[2311]: Failed password for invalid user admin from 203.0.113.25 port 50120 ssh2
Aug 28 09:12:18 web01 sshd[2312]: Failed password for root from 203.0.113.25 port 50121 ssh2
Aug 28 09:13:04 web01 sshd[2320]: Accepted password for student from 192.0.2.40 port 50310 ssh2
Aug 28 09:15:41 web01 sshd[2344]: Failed password for invalid user test from 198.51.100.17 port 51110 ssh2
Aug 28 09:16:02 web01 sshd[2345]: Failed password for invalid user oracle from 198.51.100.17 port 51111 ssh2
Aug 28 09:16:26 web01 sshd[2346]: Failed password for root from 198.51.100.17 port 51112 ssh2

In Splunk Web, complete these steps:

1. Open Settings → Indexes and create an index named security if it does not exist.
2. Select Settings → Add Data → Upload.
3. Upload auth-sample.log.
4. Preview the timestamps and individual event boundaries.
5. Select the linux_secure sourcetype, or create a suitable custom sourcetype for your lab.
6. Choose the security index and submit the input.

Menu names can differ slightly between Splunk versions and managed environments. Creating indexes and adding data also requires the relevant permissions.

For continuous collection with a Universal Forwarder, first configure the indexer to listen on TCP port 9997 under Settings → Forwarding and Receiving → Configure Receiving → New Receiving Port. Then run commands similar to these from the forwarder's bin directory:

sudo ./splunk add forward-server 192.0.2.10:9997
sudo ./splunk add monitor /var/log/auth.log -index security -sourcetype linux_secure
sudo ./splunk restart
sudo ./splunk list forward-server

On some Linux distributions, SSH authentication events are stored in /var/log/secure instead of /var/log/auth.log. The Splunk service account must have permission to read the file, and network firewalls must allow the forwarder to reach the receiving port.

How Do You Run Your First Splunk Searches?

Start with a narrow index, sourcetype and time range. This reduces unnecessary processing and helps you confirm that the expected data is present before adding complex commands.

Open Search & Reporting, select an appropriate time range and run:

index=security sourcetype=linux_secure

The event list should show the raw SSH messages. Check that _time, host, source and sourcetype contain sensible values.

To find failed logins, add the phrase from the raw event:

index=security sourcetype=linux_secure "Failed password"

To compare successful and failed logins:

index=security sourcetype=linux_secure
| eval outcome=if(searchmatch("Failed password"), "failure", if(searchmatch("Accepted password"), "success", "other"))
| stats count by outcome

Common SPL search controls include:

Use indexed metadata early in the search where possible. A search such as index=security host=web01 "Failed password" is normally more focused than searching all indexes for the words Failed password.

How Do You Extract Fields from SSH Logs?

Field extraction turns parts of a raw event into searchable values. For the sample log, you can use the rex command to extract the attempted username and source IP address during search time.

Run this search:

index=security sourcetype=linux_secure "Failed password"
| rex "Failed password for (?:invalid user )?(?<user>\S+) from (?<src_ip>\d{1,3}(?:\.\d{1,3}){3})"
| table _time host user src_ip
| sort - _time

The regular expression has two named capture groups:

• (?<user>...) creates a field named user.
• (?<src_ip>...) creates a field named src_ip.
• (?:invalid user )? optionally matches the words used for an unknown account.

Now count failed attempts by source address:

index=security sourcetype=linux_secure "Failed password"
| rex "Failed password for (?:invalid user )?(?<user>\S+) from (?<src_ip>\d{1,3}(?:\.\d{1,3}){3})"
| stats count AS failed_attempts values(user) AS attempted_users by src_ip
| sort - failed_attempts

For the sample data, 198.51.100.17 should have the highest count. This is a finding for investigation, not automatic proof of an attack. A SOC analyst should check whether the address belongs to a trusted scanner, administration network, test system or expected automation before escalating it.

Search-time rex is convenient for a lab. In a production deployment, frequently used fields should be extracted consistently through sourcetype configuration, Splunk knowledge objects or a supported add-on.

How Do You Build Your First SIEM Dashboard?

A useful first dashboard should answer a small set of operational questions: how many failures occurred, when they occurred, which sources generated them and which usernames were targeted. Keep the dashboard focused rather than adding charts that do not support investigation.

In Splunk Web, open Dashboards → Create Dashboard and select Dashboard Studio if it is available. Add a global time-range input, then create the following panels.

Panel 1: Total Failed SSH Logins

Use a single-value visualisation:

index=security sourcetype=linux_secure "Failed password"
| stats count AS "Failed SSH logins"

This panel provides immediate volume, but it needs a time range for context. A value of 30 means something different over ten minutes than over thirty days.

Panel 2: Failures Over Time

Use a line or column chart:

index=security sourcetype=linux_secure "Failed password"
| timechart span=15m count AS failures

A time chart helps identify spikes and repeated bursts. Adjust the span to suit the dashboard period; a 15-minute span is usually too detailed for a multi-month view.

Panel 3: Top Source IP Addresses

Use a bar chart or table:

index=security sourcetype=linux_secure "Failed password"
| rex "from (?<src_ip>\d{1,3}(?:\.\d{1,3}){3})"
| top limit=10 src_ip

The top command returns the count and percentage for each value. Verify that src_ip is populated before trusting the chart.

Panel 4: Targeted Usernames

Use a table:

index=security sourcetype=linux_secure "Failed password"
| rex "Failed password for (?:invalid user )?(?<user>\S+) from (?<src_ip>\d{1,3}(?:\.\d{1,3}){3})"
| stats count AS attempts dc(src_ip) AS unique_sources by user
| sort - attempts

The dc function calculates a distinct count. A username targeted from many sources may deserve different investigation from repeated failures caused by one user's old saved password.

Give every panel a clear title, connect it to the global time input and save the dashboard with restricted permissions until its searches have been reviewed. Broader cybersecurity labs, including validation of suspicious behaviour, can be developed alongside an Ethical Hacking course, but only in systems where you have explicit permission to test.

How Do You Troubleshoot Missing or Incorrect Splunk Results?

Check the time range, index, source and permissions before changing the SPL. Most beginner problems come from searching the wrong period, ingesting data into another index or using a field that was not extracted.

Use this troubleshooting sequence:

1. Search all accessible data for a unique phrase from the event.
2. Inspect the event's index, host, source and sourcetype fields.
3. Remove pipes one at a time to find which command removes the results.
4. Confirm that the timestamp matches the actual event time.
5. Check forwarder connectivity and file permissions for live inputs.

Useful diagnostic searches include:

index=* "Failed password"

index=_internal source=*splunkd.log (ERROR OR WARN)
| table _time host component log_level message
| sort - _time

Common problems and fixes are:

Avoid using index=* as a permanent dashboard search. It is useful for short troubleshooting checks but can search unnecessary data and consume more resources.

What Security Practices Should Beginners Follow?

Splunk contains sensitive operational and identity data, so access should follow least privilege. Protect the platform itself as carefully as the systems whose logs it stores.

Use role-based access, restrict sensitive indexes, synchronise system clocks and use encrypted transport between components. Do not ingest passwords, private keys, access tokens or unnecessary personal data. Production dashboards and alerts should also document their owner, purpose, time range and expected response.

Summary

Splunk receives logs, stores them in indexes and uses SPL to transform events into investigation results. A reliable beginner workflow is to verify the raw data, constrain the search, extract useful fields, aggregate the results and only then build dashboard panels.

The SSH lab demonstrates the core process used for many other sources, including firewall logs, Windows authentication events and cloud audit records. The same searches can later be extended with baselines, alerts, enrichment and documented incident-response procedures.

Frequently Asked Questions

Is Splunk difficult for beginners?

Splunk is approachable when you begin with one log source and a few SPL commands. Learn filtering, stats, table, sort and timechart before attempting complex correlations.

Is Splunk the same as a SIEM?

Splunk Enterprise provides searching, indexing and dashboards that support security monitoring. Splunk Enterprise Security adds dedicated SIEM capabilities such as correlation searches, notable events and risk-based alerting.

What is SPL in Splunk?

SPL stands for Splunk Processing Language. It is used to search events, extract fields, calculate statistics and prepare results for dashboards and alerts.

Which logs should a beginner ingest first?

Linux authentication logs are a practical starting point because they contain recognisable login successes and failures. After that, try Windows security events, firewall logs or cloud audit logs in an authorised lab.

Why does my Splunk search return no results?

The common causes are an incorrect time range, wrong index, missing permissions or data that was never ingested. Start with a unique raw phrase, then verify the event metadata and forwarder status.

Can a Splunk dashboard detect an attack automatically?

A dashboard displays patterns and investigation data, but a pattern is not automatically proof of an attack. Detection requires tested logic, context, thresholds and a response process that accounts for legitimate activity.

To practise log ingestion, SPL, dashboard creation and SOC investigation workflows in a guided lab, enquire about upcoming batch details for the Cybersecurity & SOC course.

Reviewed by Network Rhinos cybersecurity trainers.

Related reading: Ethical Hacking Phases: From Reconnaissance to Reporting