Sigma Regression Testing Pipeline
Detection Rules That Work.
Proven — Before Attackers Find Out They Don't.
140 Sigma rules for Windows, Linux, and M365/Azure — with automated CI/CD validation, Splunk conversion, and Atomic Red Team regression testing built in. MITRE ATT&CK mapped. Free and open source.
140
Sigma detection rules
3
Platforms: Windows, Linux, M365/Azure
CI/CD
Automated validation on every commit
100%
Free & open source
The Problem
Most detection rules are written once and never tested again.
A SIEM update. A new log source. A parser change. Any one of them can silently break a detection rule that was passing yesterday. Without automated regression testing, detection engineers have no way to know, until an adversary finds the gap first. The Sigma Regression Testing Pipeline closes that loop.
No Tests
Most Sigma rules are validated once in staging against sample data, never against real production logs, field mappings, or actual adversary technique execution.
Silent Drift
Platform updates, parser changes, and log source additions break existing rules with no alert. Detection coverage degrades silently while teams assume their rules are still firing.
No Proof
Without an automated test harness, there's no evidence a rule fires against a real technique execution, no CI gate, no pass/fail record, nothing to show leadership that coverage is real.
What Is the Sigma Regression Testing Pipeline
Detection-as-Code with automated proof that rules actually fire.
The Sigma Regression Testing Pipeline is SCYTHE's open-source framework for building, converting, and continuously validating detection rules at scale. It pairs 140 vendor-neutral Sigma rules with an automated CI/CD pipeline that converts to Splunk SPL, executes corresponding Atomic Red Team tests, and gates deployment on proven coverage, so every rule in production has passed a real regression test.
Vendor-Neutral Sigma Rules
140 rules written in the Sigma standard, portable to any SIEM. The pipeline includes Splunk conversion out of the box; the rules themselves work anywhere pySigma runs.
Automated CI/CD Validation
Every commit triggers automatic rule conversion and test execution. CI gates prevent broken or untested rules from reaching production, detection coverage is enforced, not assumed.
Atomic Red Team Integration
Sigma rules are paired with Atomic Red Team tests that execute the corresponding technique. Pass/fail is determined by whether the converted SIEM query fires against the actual technique execution, not simulated data.
ATT&CK Mapped
Every rule is tagged to MITRE ATT&CK technique IDs. Coverage gaps surface automatically, giving detection engineering teams an accurate, tested heatmap of what they actually detect.
What's Included
The complete detection-as-code stack. Ready to deploy.
Everything from rule authoring through production deployment, including the CI/CD configuration, the conversion pipeline, the Atomic test mappings, and 140 rules that have passed regression testing before you even fork the repo.
Detection Rules
Windows Rules
Process creation, registry modifications, lateral movement, credential access, defense evasion, covering the most targeted Windows attack surfaces with rules mapped to ATT&CK technique IDs and paired with Atomic tests.
Linux Rules
Server-side detection coverage for Linux environments, covering privilege escalation, persistence mechanisms, suspicious command execution, and cloud workload attack patterns.
M365 & Azure Rules
Identity-focused detection for the cloud attack surface, mailbox forwarding rules, OAuth abuse, conditional access bypasses, Azure AD privilege escalation, and M365 admin activity monitoring.
Pipeline Components
CI/CD Configuration
Ready-to-use GitHub Actions workflow that runs on every pull request, converting rules, executing tests, and gating merges on regression pass. Bring your own environment, drop in the workflow.
Splunk Conversion Pipeline
pySigma-based conversion pipeline with field mappings pre-configured for Splunk Enterprise Security. Each Sigma rule converts to production-ready SPL with the correct index, sourcetype, and field mappings applied.
Atomic Test Mappings
Each Sigma rule is paired with one or more Atomic Red Team test IDs. The pipeline executes the mapped technique, then queries the converted SPL to confirm the rule fires, automated pass/fail evidence on every run.
ATT&CK Coverage Report
Auto-generated MITRE ATT&CK coverage output from the CI pipeline, showing which technique IDs have passing tests, which are failing, and which have no coverage. A living record of your detection posture.
Coverage Breakdown
140 rules. 3 platforms. Every one tested.
Coverage spans the platforms most commonly targeted by modern adversaries, with a deliberate emphasis on the identity and cloud attack paths that endpoint-only detection programs miss entirely.
Windows
Windows
Endpoint & Host Detections
Covering process creation, PowerShell abuse, registry persistence, scheduled tasks, WMI lateral movement, credential dumping, and defense evasion techniques.
ATT&CK coverage includes
→ Execution (T1059, T1047, T1053)
→ Credential Access (T1003, T1555)
→ Defense Evasion (T1027, T1562)
→ Lateral Movement (T1021, T1570)
Linux
Linux
Server & Workload Detections
Covering privilege escalation, sudo abuse, crontab persistence, suspicious shell execution, network connections from unexpected processes, and rootkit-like behavior patterns.
ATT&CK coverage includes
→ Persistence (T1053, T1543)
→ Privilege Escalation (T1548, T1611)
→ Execution (T1059, T1609)
→ Discovery (T1087, T1082)
M365 & Azure
Cloud
Identity & Cloud Detections
The identity attack surface most endpoint-only detection programs miss. Covering OAuth abuse, mailbox rule creation, Azure AD privilege escalation, service principal abuse, and conditional access bypass patterns.
ATT&CK coverage includes
→ Initial Access (T1078, T1566)
→ Persistence (T1098, T1136)
→ Collection (T1114, T1530)
→ Exfiltration (T1537, T1567)
How the Pipeline Works
Four stages. One automated loop.
The pipeline runs automatically on every commit. A detection engineer writes or updates a Sigma rule, from that point, the rest is automated: conversion, technique execution, validation, and CI gate. No manual testing required.
01
Write in Sigma
Author detection logic in vendor-neutral Sigma YAML. Tag the rule with the ATT&CK technique ID and the Atomic Red Team test identifier it validates against. Commit to the repo.
02
Convert to SPL
The CI pipeline converts the Sigma rule to Splunk SPL using the pre-configured pySigma pipeline. Field mappings are applied automatically. The resulting SPL query is staged for validation testing.
03
Execute the Atomic
The mapped Atomic Red Team test executes in the test environment, firing the real technique and generating actual telemetry. This is not simulated data or replayed logs. The technique runs.
04
Validate & Gate
The converted SPL query runs against the telemetry from step 3. If it returns results, the rule passes. If it returns nothing, the CI gate fails and the merge is blocked. Every rule in production has earned its place.
Who Uses This
Built by detection engineers. Useful across the entire blue team.
The pipeline was built for teams that take detection engineering seriously, but the outputs benefit everyone from the individual SOC analyst to the CISO who needs to answer "are our detections actually working?"
Detection Engineers
Stop writing rules that pass in staging and break in production. The CI/CD pipeline gives you proof, not assumption, that your Sigma rule fires against the real technique in your real environment, every time you push.
SIEM Engineers
Every SIEM update, parser change, or index reconfiguration is now a regression test event, not a silent coverage break. The pipeline catches field mapping drift before it becomes an incident you don't detect.
SOC Analysts & Leads
The auto-generated ATT&CK coverage report shows which techniques your rules are actively detecting, tested, not theoretical. Know the real coverage map before your next tabletop or audit review.
CISOs & Security Leaders
Finally answer the board's question — "do our detections work?" — with CI pass/fail history instead of a consultant's assessment. ATT&CK coverage gaps documented, prioritized, and tracked over time.
Get Started
Fork the repo. Run your first regression test this week.
01
Fork & Configure
Fork the repository on GitHub. Configure the CI environment variables for your Splunk instance and Atomic Red Team setup. The workflow file is pre-written, point it at your environment and it runs on the first push.
02
Run Against Your Stack
The 140 included rules run through your pipeline on day one. Some will pass immediately. Others will surface field mapping differences between the community pipeline and your specific log sources, those are your first detection engineering work items.
03
Contribute & Extend
Write new Sigma rules, add Atomic test pairings, and submit pull requests back to the community repository. Every contributed rule that passes CI becomes part of the shared 140+ library, expanding tested coverage for every user of the pipeline.
Take It Further
The pipeline validates your detections. SCYTHE tests the controls behind them.
Sigma regression testing confirms your SIEM rules fire. SCYTHE's platform and frameworks take the next step, running full adversary campaigns against your production environment to validate everything beyond rule logic.
PTEF v4
The Purple Team Exercise Framework gives your detection rules the adversary they need to prove themselves, structured exercises, graded scoring, and a Detection Engineering lifecycle that feeds directly into a regression pipeline like this one.
Learn More →SCYTHE Platform
Move beyond Atomics to full adversary campaign execution. SCYTHE runs multi-stage emulation plans against your production stack, validating detection rules, SOC workflows, and response processes in one continuous test.
Learn More →SIEM Detection Engineering
SCYTHE's SIEM Detection Engineering solution pairs continuous adversary emulation with your existing SIEM stack, running technique-level tests to validate that rules fire, alerts trigger, and workflows respond correctly.
Learn More →Detection-as-Code. Proven.
Stop assuming your detections work. Start proving it.
140 Sigma rules. Windows, Linux, M365/Azure. Automated CI/CD. Atomic Red Team regression testing. ATT&CK mapped. Free and open source.
Get the Pipeline
Your detections shouldn't be a matter of faith.
Download the Sigma Regression Testing Pipeline — free, open source, and ready to run against your Splunk environment. Fill out the form and we'll send you the repo, setup guide, and documentation.
What's in the download
✓ 140 Sigma Rules (Windows, Linux, M365/Azure)
✓ GitHub Actions CI/CD Workflow Configuration
✓ pySigma Splunk Conversion Pipeline
✓ Atomic Red Team Test Mappings
✓ ATT&CK Coverage Report Generator
✓ Setup Guide & Contributing Documentation
Free Download
Get the Sigma Pipeline
No spam. No vendor lock-in. The pipeline is free and open source.
We'll only use your info to send you the download and relevant SCYTHE updates.