Test & Evaluation

This section outlines the components that have been identified for the Test & Evaluation dimension as well as maturity levels within each components. These components and levels form the basis for assessing how threat-informed an organization’s T&E program is.

  1. Test Focus: This component assesses which kinds of testing your organization conducts, ranging from compliance and IOC-focused assessments to behavior-focused testing.

  2. Test Planning: This component assesses how well your organization plans testing and connects those tests with their overall security posture.

  3. Test Relevance: This component assesses how quickly new CTI is ingested into your testing procedures.

  4. Test Triggers: This component assesses if your testing is influenced by recent security events and/or proactively undergoes regular planning.

  5. Test Results: This component assesses how well testing procedures translate into actionable changes within your organization’s security posture.

Test Focus

What is the focus of your organization’s testing?

  • Testing is compliance-focused, e.g. security control assessment

    • Security control assessments evaluate whether security controls are functioning as intended.

  • Testing is IOC-focused, e.g. vulnerability assessment

    • Commodity tools are off-the-shelf solutions that enable pen testing or red team activities using software that is well-known and easily detected.

  • Testing is behavior-focused, executing a single procedure of ATT&CK techniques

    • Many ATT&CK techniques have different ways, or procedures, that an attacker can use to achieve the same goal. For instance, schtask /create and register-scheduletask will both achieve the technique of Scheduled Task.

  • Testing is behavior-focused, executing multiple procedures of a technique, perhaps using custom tooling

Test Planning

How are tests planned within your organization?

  • Testing is designed to discover detection gaps and validate coverage for your attack surface

  • Testing methodology is informed and prioritized by the threats and risks most relevant to your organization

  • Testing is collaboratively planned with defenders, to include security response and remediation components

  • Testing is linked to organizational metrics or key performance indicators (KPIs) to measure effectiveness in discovering gaps, validating coverage, and performing incident response and remediation

    • Example KPIs: time to initial access, time to detection, time from initial access to lateral movement, number of vulnerabilities identified, % of detection evasion, % of recommendations remediated

*For more thorough details on test planning, refer to the Continuous Threat Exposure Management Framework

Test Relevance

How quickly is new CTI incorporated into your testing?* As new security advisories come out, can your team quickly turn those into test procedures?

  • Not CTI Driven or only relying on outdated CTI

  • Within a month

  • Within a week

  • Within a day

Test Triggers

Are tests planned proactively or reactively?

  • Reactive to external security events

  • Reactive to internal security events

  • Testing is proactively planned on a periodic basis

  • Testing is proactively planned on a continuous basis, perhaps through breach and attack simulation platforms

Test Results

How do test results drive improvements in defensive measures?

  • Actions are taken with internal security team to remediate individual hosts**

  • Findings drive detection and architectural changes

    • Architectural changes improve the design of systems or networks to enhance security, e.g. strengthening key management, isolating critical subnets through network segmentation.

  • Findings drive organizational or policy changes

    • Organizational decisions are made based on the results of security testing, e.g. business strategy shifts, changes in hiring or training.