So You Want to Bring AI Into Your Defense

A nod to the MCU’s Captain America PSAs.

I was just on a panel with our partner, SimSpace, for the AI Proving Grounds, and the question came up: “If you were a CISO with a limited budget, would you invest more in AI agents today, or in strengthening existing security operations? Why?”

Strengthen existing security operations first. AI agents are useful accelerators, but they deliver poor returns (and can create new risks) on a weak foundation, especially under tight budgets. Bolted onto weak process, AI agents can create more work before they save any.

AI agents can create more work before they save any, if you bolt them onto weak process.

My Recommended Maturity Model

Phase 1

Blocking and Tackling

Breaches happen because of the basics: social engineering, phishing, poor perimeter hygiene, alert fatigue, or credential theft and loss, whether that’s what we’ve seen historically or the new fear of autonomous AI attacks (potentially masquerading as marketing… ahem). This means your focus should be on identity, visibility, and controls. These are high-ROI areas where incremental investment compounds quickly: better tooling integration, tuned detection rules, automation of repetitive triage, staff training, tabletop exercises, vulnerability management maturity, and network segmentation. Limited dollars spent here reduce the attack surface and mean time to detect and respond more reliably than experimental agents.

Purple teaming is my personal 🦄 recommendation for the best way to drive improvements here, because security is defined by the threat. Like carving a sculpture à la Raphael out of an unfinished chunk of stone, using the threat as a technical signal through your enterprise lets you effectively and realistically improve your entire defensive portfolio of people, process, and technology.

Security is defined by the threat.

Key Metrics, Foundation Phase

These prove the SOC / IR function is becoming dependable:

• Mean time to detect (MTTD). There’s granularity worth understanding here: the levels differ across host logging, SIEM receipt (it takes time to populate), and an alert firing.
• Mean / median time to respond (MTTR). Time from detection to response.
• Alert volume and quality baselines. Total alerts per day/week; percentage of alerts that are true positives or actionable (start measuring even if it’s initially low); top alert sources by volume.
• Incident adherence. Percentage of incidents that followed the documented severity / escalation process; percentage of high-severity incidents with timely stakeholder notification.
• False-positive / noise rate (even if approximate, via sampling). Trend in analyst time spent on non-actionable alerts.

Phase 2

AI Introduction

Use AI for select, prioritized automation. Narrow automation focuses on high-volume, repetitive, low-risk, and largely deterministic tasks inside the existing SOC workflow. It uses clear rules, scripts, or playbooks rather than autonomous agents that interpret context, decide intent, or take broad initiative. This approach sits between pure manual operations and full AI agents. It delivers immediate analyst time savings and consistency while remaining auditable, testable, and reversible, which is critical when budget and headcount are constrained.

• Cost and complexity stay low; you mostly configure existing tools or write modest scripts instead of deploying, tuning, and governing new agent platforms.
• Behavior is predictable (deterministic) and explainable to auditors, leadership, and the team.
• Failures are usually obvious (a playbook didn’t fire, an enrichment timed out) rather than subtle (an agent hallucinated or took an unexpected action).
• It directly reduces alert fatigue and frees senior analysts for threat hunting, detection engineering, and process improvement, which actually raises the security baseline.
• It creates clean, structured data and well-understood workflows. Those become the training wheels and guardrails for any later AI-agent investment.

Here’s where to start, with some summarized ideas I’ve seen working across industry. 🦄

1. Alert enrichment playbooks (highest-ROI starting point)

When a new alert fires (IP, domain, hash, user, host):

• Query internal CMDB / asset inventory for ownership, criticality, and recent changes.
• Pull reputation / threat-intel scores (VirusTotal, your internal threat intelligence platform, the Exploit Prediction Scoring System (EPSS), etc.).
• Check recent related activity in the SIEM or EDR.
• Append the results as structured fields or a comment on the ticket / alert.
• Optionally auto-assign priority or severity based on simple scoring rules.

2. Standardized response playbooks for common, low-ambiguity cases

• Confirmed malware that matches a high-confidence indicator of compromise (IOC): create a ticket, notify the owner, and recommend isolation (or auto-isolate after a dual-control check).
• Compromised user-account indicators that meet strict criteria: force a password reset and session revocation, and notify the user / manager, with a manual review gate for privileged accounts.
• Phishing email reported by a user: extract indicators, search the mail gateway for similar messages, quarantine matching items, and reply to the reporter with a templated status you define.

3. Scripted or low-code orchestration outside full SOAR

• SIEM-native automation / correlation rules and response actions in playbooks.
• Lightweight open-source or low-code tools (commercial or simple Python scripts) that call existing APIs (EDR, identity provider, ticketing system, threat intelligence).
• Scheduled jobs that continuously enrich asset inventories, check certificate expiry, or validate critical security configurations.

Key Metrics, AI Introduction

These grow your maturity measurably:

• Median time from alert creation to an enriched, prioritized ticket.
• Percentage of alerts that no longer require manual enrichment.
• Analyst hours reclaimed per week.
• False-positive / auto-close accuracy (measured by sampling).
• Change in mean time to detect and respond for the specific use cases you automated.
• Number of playbook failures or manual overrides (a quality-control metric on your AI work).

Phase 3

Agentic AI

Once visibility, core controls, processes, and selective AI automation are reliable and measurable, you can selectively introduce AI agents. At this stage they become force multipliers rather than expensive experiments that amplify noise or create new risks.

Generalized LLMs will start to cause your organization problems as you integrate and rely on them more deeply in your stack, which is why you’ll want your own local model:

• SaaS. Like any SaaS solution, you’re subject to their updates when they push them, except you aren’t going to get technical release notes to test and update your dependencies. Things are just going to stop working.
• Guardrails are coming. More of them. Those future guardrails will restrict, in unforeseen ways, how we rely on AI for security operations. “We were able to do it yesterday” will become a recurring statement from the technical team as they try to work around the limitations.
• Economic risk. Remember when Uber rides were really, really cheap? That’s because the model was subsidized by significant capital investment, then a sustainable business model became necessary. We’re already seeing this shift as the models move to tokenization, which will be followed by increased costs that you’re not shackled to with a local model.
• A local model addresses these issues, but it requires a combination of security-operations and AI maturity.

With that foundation, agents earn their place as force multipliers, each one keeping a human in the loop:

• Alert prioritization and triage assistance. AI scores or ranks incoming alerts using multiple weak signals (behavioral context, historical analyst decisions, asset criticality, related activity) that pure rules struggle to capture cleanly. Output is a recommended priority plus a short rationale, not an autonomous close or escalate.
• Advanced enrichment and summarization. Beyond deterministic lookups: natural-language summaries of an entity’s recent activity, clustering of related alerts into potential incidents, or extraction of key indicators from noisy logs and tickets. Still read-only, or write-to-ticket only.
• Detection engineering and hunting support. Suggest new detection logic, translate natural-language threat hypotheses into queries, or surface anomalous patterns for human review. The agent proposes; humans validate and promote.
• Playbook / response recommendation. Given a confirmed or high-confidence incident, suggest the next two or three playbook steps or relevant runbooks based on similar past cases. Execution remains gated.
• Noise reduction and false-positive feedback loops. Learn from analyst dispositions over time to improve scoring, while remaining subordinate to explicit rule-based suppressions.
• Automated remediation. Start with deterministic, recommendation-only actions for low-impact fixes; increase autonomy as you gain confidence in the recommendations and in your human-in-the-loop deployment process.

Key Metrics, Agentic AI

Build on your earlier Key Metrics to assess performance against your previous baselines:

• Reduction in analyst time spent on prioritization / enrichment for the targeted alert classes.
• Precision / recall of agent recommendations versus human ground truth (via sampling, one reason monitoring is so important).
• Impact on overall MTTD / MTTR and false-positive rate.
• Override / rejection rate by analysts (a read on agentic performance, or how much your people need to rage against the machine).
• Operational cost (token / compute and maintenance time) versus hours saved.
• Incident outcomes: did the AI-assisted path surface anything the rules missed, or did it create new delays and errors? (via sampling)

The Bottom Line

A limited-budget CISO’s job is risk reduction per dollar and resilience, not early adoption of the latest category. Strong operations create the telemetry, processes, and institutional knowledge that later make AI agents effective and safer. Jumping to agents first often produces expensive theater rather than measurable security improvement. Once the foundation is solid, targeted AI investment becomes far higher leverage.