SPL for Beginners: Searching and Investigation in Splunk

The Initial Search

A simple search in Splunk can start like this:

index=lab sourcetype=auth earliest=-24h

The Time Range Picker can define the time, but explicitly writing earliest and latest is useful for reproducibility and documentation. index restricts to a specific repository and sourcetype describes the source structure. You can add a field condition already in the initial search:

index=lab sourcetype=auth action=failure earliest=-24h

When a field exists at search time, specifying `field=value` is more accurate than a free text search. Quotation marks are required for values with spaces. Wildcards can be useful, but a leading Wildcard or a broad search harms performance and can create unexpected results.

Pipe and Display Commands

The | symbol passes the results to the next command. `fields` keeps or removes fields, and `table` displays a table in the selected order. At the beginning of an investigation, table helps readability, but don't use it too early if a later command needs fields you removed.

index=lab sourcetype=auth earliest=-24h
| fields _time user src_ip action host
| table _time user src_ip action host

`rename` changes names for display. `sort` sorts, and `head` limits results. Events are usually displayed from newest to oldest, but when building a Timeline, it's recommended to sort explicitly:

index=lab sourcetype=auth user="student" earliest=-4h
| table _time user src_ip action host
| sort 0 _time

The number 0 in sort cancels a certain default limit on the number of results for sorting, but on large volumes, a full sort can be expensive. In an investigation, use a narrow range.

eval — Creating Fields

`eval` calculates or creates a field. You can use if, case, lower, coalesce, tonumber and other functions. For example, creating a uniform Outcome:

index=lab sourcetype=auth earliest=-24h
| eval outcome=case(action="success", "Success", action="failure", "Failure", true(), "Other")
| table _time user src_ip action outcome

`where` filters using an expression after events have been collected or after fields have been created. The initial search `action=failure` is usually preferred for basic filtering. `where` is useful for comparing fields, numerical conditions, or fields created with eval.

index=lab sourcetype=auth earliest=-24h
| stats count as attempts by user src_ip
| where attempts >= 5
| sort - attempts

stats — Turning Events into a Question

`stats` calculates Aggregations on the results. Without BY, one row is obtained; with BY, one row is obtained for each combination of values. Common commands include count, sum, avg, min, max, values, and dc — distinct count.

index=lab sourcetype=auth action=failure earliest=-24h
| stats count as failures,
        earliest(_time) as first_seen,
        latest(_time) as last_seen,
        values(src_ip) as src_ips
  by user
| convert ctime(first_seen) ctime(last_seen)
| sort - failures

`values` returns unique values without a guaranteed order and can grow large. Use it carefully and prefer a limited list or Drill-down when there are many IPs. `list` preserves values in order but can consume more memory.

stats vs. eventstats and streamstats

Beginners should master stats before using transaction. `transaction` can be convenient for grouping, but on large volumes, it's expensive and sometimes obscures logic. Often, stats, streamstats, or eventstats provide a more efficient and transparent solution.

Time Windows and timechart

`timechart` creates a time series and groups by `_time`. It's good for identifying Spike, trend, and change in volume. For example:

index=lab sourcetype=auth action=failure earliest=-24h
| timechart span=30m count by src_ip limit=10

Choose span according to the question. A too-small window will create noise, and a too-large window will hide Burst. timechart is a Transforming command: the result is a summarized table, not the raw events. For Evidence, Drill-down to the relevant range and IP.

Exercise: Failures and Successes in the Same Window

The goal is to find windows where the same user and IP generated several failures and at least one success. The following Query uses eval, bin, and stats:

index=lab sourcetype=auth earliest=-24h
| eval failed=if(action="failure", 1, 0), success=if(action="success", 1, 0)
| bin _time span=30m
| stats sum(failed) as failures,
        sum(success) as successes,
        earliest(_time) as window_start,
        latest(_time) as window_end
  by _time user src_ip
| where failures >= 5 AND successes >= 1
| sort - failures

The Query indicates a window that requires investigation. Since the data is summarized, it does not prove that the success occurred after the failures. Use the result to Drill-down on user, src_ip, and time:

index=lab sourcetype=auth user="student" src_ip="203.0.113.25" earliest="08/01/2026:09:00:00" latest="08/01/2026:09:30:00"
| table _time user src_ip action host reason
| sort 0 _time

Check: Does the success belong to the same Host or App? Is the source address a VPN? Were the failures caused by an old password in the service? Is there follow-up activity? SPL provides the data; Classification requires context.

Fields, Extraction, and CIM

Splunk extracts fields at Indexing or Search time. A missing field may require `rex`, `spath` for JSON, or defining Field extraction. Ad-hoc Extraction can help in the lab, but production Detection needs a maintained Parser and Data model.

CIM normalizes concepts across sources, for example Authentication.user or Network_Traffic.src. When the organization uses CIM, it's possible to build Detections and Dashboards that can be used across several products. It must be checked that the data actually conforms to the model and not just that the App is installed.

Improving Performance and Readability

• Set the Time range as narrow as possible.
• Start with index, sourcetype, and mapped fields.
• Filter early before stats or sort.
• Avoid leading Wildcard and broad text search when a field exists.
• Use fields to reduce Payload, but not before a command that needs the field.
• Limit values/list and memory-intensive operations.
• Give clear names to fields using `as`.
• Save Query with description, Owner, time, and version.
• Test on a small Sample before a long range.

Common Mistakes

• Searching all indexes unnecessarily.
• Assuming an action or user field exists in every sourcetype.
• Using table early and removing a field needed later.
• Interpreting stats as an accurate Timeline.
• Using transaction as a default.
• Activating an Alert on a Query not tested against a Baseline.
• Copying SPL from a different version or Data model without adaptation.
• Ignoring time zone and Ingestion delay.

Practice Checklist

1. Find the index and sourcetype of the lab data.
2. Display five Events and check the fields.
3. Filter one user or IP.
4. Display a Timeline using table and sort.
5. Summarize failures using stats.
6. Create a field using eval and filter it with where.
7. Display volume over time using timechart.
8. Write what each Query proves and what it does not prove.

Summary and CTA

Set up a small lab index or use authorized data, and run the same scenario in three views: Raw events, stats, and timechart. Write next to each view what question it answers and what Context is missing. The next step is to turn a stable search into a Detection with Owner, Threshold, and Playbook — not just save an impressive Query.

FAQ