All of those fancy security monitoring tools and the attackers still got in undetected… How could that be possible!?! It happens! Sometimes it’s luck, sometimes it’s skill but no matter what it is, they still got in undetected. You thought you could just deploy those tools and sit back and wait to be alerted when an attacker tried to bypass your defenses, but that doesn’t always work.
What if your tools fail you? Software is software and it’s not always perfect. You need a way to identify threats that get past your defenses undetected. You need to be another layer of detection. That’s where threat hunting comes into play.
So, What Is Threat Hunting?
Threat hunting is all about actively searching for threats in your environment that your security tools may have missed. It’s not reactive. It’s not based on alerts. It’s you, digging through logs of network traffic, endpoint data, application data, behavioral analytics, etc., looking for signs that something just isn’t right.
You’re not waiting for someone to tell you there’s a problem. You’re assuming there might be one and setting out to prove, or disprove, that theory.
In threat hunting, you take on the role of the adversary. You think like them. You trace their steps. You question the normal. And most importantly, you apply intuition and think about things in a way that most automated tools don’t. In reality, it’s not glamorous, but it’s necessary.
The Threat Hunter’s Mindset
I love talking about this part because you need to think like an attacker and it’s fun.
Threat hunting isn’t just about technical skills. It’s a way of thinking. You have to be endlessly curious, willing to dig deep, and patient enough to sift through a mountain of data without jumping to conclusions.
Many threat hunters will learn and practice offensive security to know what the attacks look like and understand common tactics and patterns used by adversaries. You can take it a step further by setting up a lab with monitoring tools and simulating attacks in a safe environment. This gives you first hand experience with connecting attacker behavior to detection signals, which is an incredible way to build your instincts as a threat hunter.
Example: Simulating Credential Dumping in a Lab
Let’s say you want to better understand how credential dumping looks in your environment. In a lab, you simulate a mimikatz attack on a test machine while collecting logs with Sysmon and forwarding them through Cribl Stream. You could look directly at the logs on the host but I think it's best to look at them in a log search platform like you would during a normal threat hunt.
During the simulation, record the time when you run the actual attack, and start looking at the logs during that time period. Try to look for:
What processes were spawned? Did mimikatz.exe launch directly? Did it masquerade under another name? What was the parent process?
What files did it create? Were any DLLs loaded that don’t normally appear? Was LSASS touched or dumped to a file?
What network activity occurred?
By analyzing this activity in a controlled setting, you start to recognize the signals such as uncommon Event IDs, strange process hierarchies, or odd command-line arguments. You start to train your brain to recognize abnormal patterns.
Later, when you’re combing through logs during a real hunt and you spot a similar process tree, it will stand out. That’s the power of thinking like an attacker. You don’t just know what to look for, you build intuition to know when something is wrong.
What to Hunt For
This is my favorite part of threat hunting because the real answer to this is, “I don’t know.” And you probably don’t either. If we knew exactly what to look for, we’d find every attacker, every time. But we don’t. That’s the whole point.
Start with a Baseline
Before you start doing any threat hunting, you need to establish a baseline. The key is knowing what normal looks like in your environment and then watching for things that don’t fit. Start simple. Build that muscle for spotting weirdness. You’re not going for perfection, you’re just simply looking for patterns that don’t seem right.
Ask yourself:
What should be happening here?
What if someone was compromised, how would they hide?
What does “normal” look like in this environment? And what doesn’t?
It all starts with knowing your environment. When you understand what normal looks like, the abnormal starts to stand out. And that’s when things get interesting.
How to Build a Baseline
Here’s how to approach building your baseline:
1. Pick a Scope
Start small. Don’t try to baseline your entire environment at once. Choose a specific set of data. Some examples to consider include:
Authentication logs.
Application specific logs (e.g., custom web apps)
WAF and network firewall logs
Network flow data (NetFlow, VPC Flow Logs)
DNS and proxy logs
EDR events (like process execution or script usage)
Windows Event Logs
By narrowing your focus, you can build a more manageable baseline and identify anomalies easier.
2. Gather Data Over Time and Analyze the Patterns
Collect and review logs from relevant sources over different time periods. Look for consistency in:
What processes typically run?
What accounts do you normally see?
What domains and IP addresses do you normally see?
Which scheduled tasks or scripts are expected?
Is there a service account that runs processes at a specific time? What processes does it normally run?
What parent-child process relationships are common?
The goal is to build a clear mental model of how the system behaves when nothing is wrong.
3. Start documenting
Document the patterns and observations from the last step. Things such as:
Common parent-child process relationships. For example, svchost.exe spawning rundll32.exe might be normal in one system but unusual in another.
Known internal-only domains and IP ranges. Document the ones that are part of expected operations, so anything outside of those can raise a red flag.
Normal remote desktop or VPN usage. Include typical users, connection times, and originating IPs. That way, an unexpected pattern jumps out.
Things that seem strange. Don’t ignore oddities just because they’re hard to explain. If something makes you tilt your head, write it down.
Don’t get caught-up trying to build the perfect system for documentation. The important thing is to start. The system will evolve over time. If you’re working on a team, it’s best to have this in a shared document that the team can update and reference.
4. Flag the Outliers
Once you know what “normal” looks like, anything that deviates from that should be investigated further:
An interactive login from a service account.
A service account login from a different country.
A script executing from a temp folder.
A process that didn’t exist last week.
A new outbound connection to a country you don’t do business with.
Not every outlier is malicious, but they’re worth investigating. Flag them, investigate them, and, most importantly, learn from them. Curiosity is a super power when threat hunting. The more you chase the oddities, the better your instincts become.
5. Revisit Regularly
Baselines aren’t static. Environments evolve as new code is developed, applications get deployed, business hours change, people shift roles, etc. Make it a habit to revisit and refine your baselines regularly.
How to Investigate an Anomaly
So you found something weird. Now what?
This is where a lot of new hunters panic, but don’t worry. Investigating an anomaly doesn’t mean you need to instantly declare it malicious or benign. Your goal is to understand it by gathering facts, asking questions, and following the trail.
Here’s a simple framework to guide your investigation:
Ask questions
What account is associated with this event? Identify the user or service account involved to understand its role in the network.
When did it happen? Identify the exact time or time window of the event to correlate with other activities. This is important when correlating other events.
Where did it happen? Determine which system, application, or network component was involved.
What else was going on around that time? Look for concurrent events or unusual activity in the same timeframe to build a broader picture.
Is this normal for this system or user? Assess whether this event fits within the expected behavior for the account or system.
Even if you don’t have all the answers right away, these questions will help shape your hunt and help you know what logs or context to pull next.
7. Gather Context and Expand Your Search
Try to look at other sources with similar information to get a clearer picture.
What else occurred around the same time?
Is this the same IP address accessing other hosts or accounts?
What other activity do we see from this account?
Same commands used across different machines
One breadcrumb often leads to another and you can follow them to gather more information.
8. Decide on Action
Not every anomaly needs a full-blown incident response. Based on what you find, your response might include:
Documenting it as a false positive
Notifying the responsible team
Adding a detection rule for future occurrences
Sounding the alarm and bringing in additional resources if it’s truly malicious
You’ll likely encounter a lot of “interesting” activity that turns out to be benign. That’s okay and is the best outcome of any threat hunt. Each anomaly is a learning opportunity and a chance to understand your environment better and refine your baseline.
Over time, you’ll get faster at distinguishing the weird-but-normal from the actually bad. And that’s how you level up your threat hunting game.
How Cribl Helps with Threat Hunting
The actual first step of threat hunting, even before establishing a baseline, is to ensure you have the logs in a place where they can be searched. Threat hunting relies on collecting and being able to search logs that cover a wide range of systems.
There is often a challenge of gathering the data efficiently and ensuring you’re not overwhelmed by logs that are just noise, but Cribl Stream can help overcome those challenges by helping you filter out the junk, enriching the useful things, and routing it to where it needs to go.
Now you have a way to gather the logs but where do you send them? They need to be stored somewhere that is affordable and easy to access. Cribl Lake is a great option for storage because it integrates easily with Cribl Search to allow you to quickly search through the data. While Cribl Lake is optimized for Cribl Search, it’s not a requirement, because Cribl Search can query data wherever it lives.
If you want a real world example of how we used Cribl Search for threat hunting, check out this blog post.
Final Thoughts
Threat hunting isn’t a one time thing, it’s an on-going exercise that you should include in your schedule. It helps improve your defensive posture, understand your environment, and improve incident response strategies.
It’s okay if your first few hunts don’t turn up anything. That’s still progress. Each hunt builds familiarity, improves your data hygiene, and gets your team more comfortable with the tools and techniques involved. Over time, you’ll move faster, ask better questions, and uncover more meaningful signals.
Make threat hunting a routine and treat it like exercise for your security muscles. You’re not just looking for attackers. You’re building a smarter, more proactive, more resilient security program.
So get curious. Ask questions. And keep hunting.
