Adversary Emulation Metrics Time to Detect

Offensive security professionals and program coordinators have a learning curve as they mature through the different ethical hacking assessment types. In Vulnerability Assessment/Management and Penetration Testing, we use Common Vulnerabilities and Exposures (CVE IDs) and the Common Vulnerability Scoring System (CVSS) to report a finding using two criteria:

• Status: Open or Closed
• Risk: Critical, Priority, High, Medium, Low, and/or Informational
  

When emulating Tactics Techniques and Procedures (TTPs), we rarely use “open” and “closed” because we are testing adversary behaviors, not vulnerabilities in technology. We have to look at different criteria:

• Prevented - was not allowed to run (blocked)
• Logged - a log was created for the behavior locally or centrally
• Alerted - was an alert created based on the logged action(s)
• Detected - was the alert actioned by a human or automation
• Response - was the process followed (see 1-10-60 rule below)
  

In Adversary Emulation, whether Red Team Engagement or Purple Team Exercise, we have a list of behaviors that we are going to emulate. Each behavior will execute a particular action at a particular time. If we log the time of each of these actions, we have an additional metrics for both the Red Team and the Blue Team: 

• Time to Objective (TTO) or Time to Ransom (TTR)
• Time to Detection
• Time to Investigation
• Time to Remediation

Time to Initial Access

Red Teams spend a significant amount of time gaining Initial Access which requires performing Reconnaissance and Resource Development. A metric to measure the Red Team is the time it takes them to gain initial access to an environment. Calculate the engagement start time with the time of the first call back to the Command and Control (C2) server.

Time to Objective

Both adversaries and red teams have objectives. Based on the predefined objective, the Red Team can tag a particular TTP as the objective and calculate the time between initial access and the successful execution of said objective. We can use ransomware as an example. The objective for these malicious threat actors is to get paid. The calculation can be from initial access to getting paid. There are a number of incident response teams using the term Time to Ransom (TTR) in examples like ICEDID and Ryuk to track how adversaries are doing. Red teams should be doing similar times.

For a red team, we generally don’t go all the way to getting paid. Instead, we may exfiltrate data and/or encrypt/delete data on one or more systems. Often, a ransom note is downloaded and displayed to the end user for impact. Any of those can be agreed on as the objective and time calculated. 

We recently released and ran Maze v2 during a Purple Team Exercise. Note that we configured a very quick and noisy C2 with 5 second heartbeats and no jitter. We tagged step 47 as the objective and can calculate Time to Objective or Time to Ransom by subtracting the time that step 47 was successful to the initial C2 connection:

• Endpoint1 - 17 minutes
• Endpoint2 - 9 minutes
• Server1 - 6 minutes
  

Of course we can make the heartbeats and jitter more realistic but that is generally not the main goal of a Purple Team Exercise. If this was a red team engagement, then the times would be different.

1-10-60 Rule

Apart from having a goal of detecting and responding to an attack before the objective is met, there are other detection metrics we can track. Dmitri Alperovitch, the founder of CrowdStrike, introduced the 1-10-60 rule. These are goals around attacker break out times that the Blue Team will want to meet to stay ahead of malicious actors:

• 1 minute - Time to Detection - organizations should set a goal of allowing only one minute to detect an incident or intrusion (automated).
• 10 minutes - Time to Investigation - the length of time it takes to find out if the incident is legitimate and determine next steps (containment, remediation, etc.). The best organizations do this within 10 minutes.
• 60 minutes - Time to Remediation - the period of time needed to eject the intruder and clean up your network, which may involve coordination with the business owner of that asset. The best organizations try to do this within 60 minutes.
  

Measuring these times during a red team engagement requires coordination with Trusted Agents to ensure the proper times are logged and tracked.

 Measuring detection and response from blue teams is a key differentiator from penetration tests and a big reason why “open” and “closed” are not ideal measurements for adversary emulations. 

About SCYTHE

SCYTHE provides an advanced attack emulation platform for the enterprise and cybersecurity consulting market. The SCYTHE platform enables Red, Blue, and Purple teams to build and emulate real-world adversarial campaigns in a matter of minutes. Customers are in turn enabled to validate the risk posture and exposure of their business and employees and the performance of enterprise security teams and existing security solutions. Based in Arlington, VA, the company is privately held and is funded by Gula Tech Adventures, Paladin Capital, Evolution Equity, and private industry investors.

‍