diff --git a/docs/usage/assets/add_manual_validation_atomic_testing.png b/docs/usage/assets/add_manual_validation_atomic_testing.png deleted file mode 100644 index 0ebcb6cb..00000000 Binary files a/docs/usage/assets/add_manual_validation_atomic_testing.png and /dev/null differ diff --git a/docs/usage/assets/manual_expectation_validation_atomic_testing.png b/docs/usage/assets/manual_expectation_validation_atomic_testing.png deleted file mode 100644 index d252be21..00000000 Binary files a/docs/usage/assets/manual_expectation_validation_atomic_testing.png and /dev/null differ diff --git a/docs/usage/assets/technical_expectation_validation.png b/docs/usage/assets/technical_expectation_validation.png deleted file mode 100644 index 67d90184..00000000 Binary files a/docs/usage/assets/technical_expectation_validation.png and /dev/null differ diff --git a/docs/usage/expectations.md b/docs/usage/expectations.md index 48e78dcf..bd41be69 100644 --- a/docs/usage/expectations.md +++ b/docs/usage/expectations.md @@ -1,149 +1,43 @@ -# Expectations -Expectations define what is expected from an [Asset (endpoint)](assets.md) or -a [Players](people.md#players) when facing an [Inject](inject-overview.md) in terms of -security posture. Each expectation has a score representing how well it has been met by the target. -## Expectation types +#### Why use it? -Expectations can be categorized as either Manual or Automatic, depending on how they are validated. +Use this mode when **every member must succeed**, such as: -### Manual expectations +* mandatory training +* compliance requirements +* baseline security checks -Manual expectations require validation by the **exercise organizer** (animation team). -They are simple, customizable, and **user-controlled**. +#### How does it work? -Examples: +* All targets succeed → **100** +* At least one target fails → **0** -- **Team response validation** - Check whether the **incident response team** correctly acknowledges and escalates a phishing inject during the exercise. +#### Example -- **Player task validation** - Ensure that an **analyst player** correctly follows the playbook and reports an incident in the ticketing system. +Group of 4 players: -### Automatic expectations +* 3 succeed, 1 fails → **0 (Failed)** +* 4 succeed → **100 (Success)** -Automatic expectations are validated automatically under specific conditions. +### At least one target must validate -- `Prevention`: automatically validated by security integrations (e.g., quarantine event). -- `Detection`: automatically validated by security integrations (e.g., incident alert). -- `Vulnerability`: automatically validated based on the presence of CVEs. -- `Expect targets to read the article(s)`: automatically validated once targets have read the injected article. +#### Why use it? -## Validation Mode +Use this mode when **a single successful response is enough**, such as: -There are two modes for validating an expectation : +* SOC detection +* on-call escalation +* redundancy testing -### All targets (per group) must validate the expectation +#### How does it work? -- The result depends on **every member's performance**. -- If one target fails, the entire group fails. -- Final score = **100** if all succeed, otherwise **0**. +* ≥ 1 success → **100** +* No success → **0** -Example: 3 players succeed, 1 fails → Group score = **0 (Failed)**. -Example: 4 players succeed → Group score = **100 (Success)**. +#### Example -### At least one target (per group) must validate the expectation +Group of 4 players: -- Success depends on **at least one target succeeding**. -- The group is considered successful if one target validates. -- Final score = **100** if ≥1 succeeds, otherwise **0**. - -Example: 3 players fail, 1 succeeds → Group score = **100 (Success)**. -Example: 4 players fail → Group score = **0 (Failed)**. - -![Validation mode](assets/validation_mode.png) - -## Expectation manipulation - -### Add an expectation to an Inject - -1. Navigate to the inject's content and click **Add expectations**. -2. Select the type of expectation you want to add. -3. Define its score and validation mode. - -You can add multiple expectations to a single inject. - -![Add expectations to an inject](assets/inject_expectations_list.png) - -### Validate a manual expectation - -- During a simulation, go to the **Animation tab → Validation screen** to manually validate expectations. - -![Validate a manual expectation from the animation tab](assets/manual_expectation_validation_animation_tab.png) - -- For atomic testing with manual injects, validation can be done directly in the **Overview tab**. - -![Add the validation of a manual expectation in atomic testing](assets/add_manual_validation_atomic_testing.png) - -![Add the validation of a manual expectation in atomic testing](assets/manual_expectation_validation_atomic_testing.png) - -### Validate technical injects - -- Users can manually add detection/prevention results for their custom security platforms. -- Automatic validation will also occur when connected to compatible collectors. - -![Add the validation of a technical expectation in atomic testing](assets/add_technical_expectation_validation.png) - -![Add the validation of a technical expectation in atomic testing](assets/technical_expectation_validation.png) - -### Update Rules - -Expectation statuses propagate between entities as follows: - -| Action | Behavior | -|--------------------|------------------------------------------------------------------------| -| Update Agent | Update its status (0 = fail, 100 = success). | -| Update Asset | Asset becomes **valid (100)** only if **all its agents are valid**. | -| Update Asset Group | Compute status depending on validation mode: | -| | - **All assets must validate** → 100 only if all assets succeed. | -| | - **At least one asset must validate** → 100 if at least one succeeds. | -| Update Player | Compute status for the team depending on validation mode. | -| Update Team | Propagate status to players. | - -### Customize expectations - -#### Default score - -Each expectation has a **default score value** at creation, configurable via environment variables: - -| Parameter | Environment variable | Default value | Description | -|:-----------------------------------------------|:-----------------------------------------------|:--------------|:-------------------------------------------| -| openaev.expectation.manual.default-score-value | OPENAEV_EXPECTATION_MANUAL_DEFAULT-SCORE-VALUE | 50 | Default score value for manual expectation | - -#### Expiration time - -Expectations must be validated within a time limit. Defaults are set in the system but can be overridden: - -- Manual Response: By default if not manual expectations are set the expiration will done after 24 hours -- Detection/Prevention: By default if no expectations are set the expiration will done after 6 hour - -| Parameter | Environment variable | Default value | Description | -|:-----------------------------------------------|:-----------------------------------------------|:--------------|:--------------------------------------------------------------------| -| openaev.expectation.technical.expiration-time | OPENAEV_EXPECTATION_TECHNICAL_EXPIRATION-TIME | 21600 | Expiration time for Technical expectation (detection & prevention) | -| openaev.expectation.detection.expiration-time | OPENAEV_EXPECTATION_DETECTION_EXPIRATION-TIME | 21600 | Expiration time for detection expectation | -| openaev.expectation.prevention.expiration-time | OPENAEV_EXPECTATION_PREVENTION_EXPIRATION-TIME | 21600 | Expiration time for prevention expectation | -| openaev.expectation.human.expiration-time | OPENAEV_EXPECTATION_HUMAN_EXPIRATION-TIME | 86400 | Expiration time for human expectation (manual, challenge & article) | -| openaev.expectation.challenge.expiration-time | OPENAEV_EXPECTATION_CHALLENGE_EXPIRATION-TIME | 86400 | Expiration time for challenge expectation | -| openaev.expectation.article.expiration-time | OPENAEV_EXPECTATION_ARTICLE_EXPIRATION-TIME | 86400 | Expiration time for article expectation | -| openaev.expectation.manual.expiration-time | OPENAEV_EXPECTATION_MANUAL_EXPIRATION-TIME | 86400 | Expiration time for manual expectation | - -A default expiration time is set for technical and human expectations. Users can override them for each type of -expectations. - -- In the UI - -![Customize expiration time in UI](assets/expectation_customize_expiration_time.png) - -When creating an expectation, users can set an expiration time. The system's default times are set on the form and users -decide to override it. - -Once the expiration time will be reached, the Expectation Expiration Manager will fail the expectation and will show the -result as `Not Detected` or `Not Prevented` depending on the expectation's type. - -## When is a Simulation Finished? - -A simulation is considered **finished** once all defined expectations are resolved: - -- **Technical expectations** → validated through **Detection, Prevention, Vulnerability checks**. -- **Human expectations** → validated manually by the organizer. +* 1 succeeds, 3 fail → **100 (Success)** +* 4 fail → **0 (Failed)** \ No newline at end of file diff --git a/docs/usage/assets/add_technical_expectation_validation.png b/docs/usage/expectations/assets/add_technical_expectation_validation.png similarity index 100% rename from docs/usage/assets/add_technical_expectation_validation.png rename to docs/usage/expectations/assets/add_technical_expectation_validation.png diff --git a/docs/usage/assets/expectation_customize_expiration_time.png b/docs/usage/expectations/assets/create_manual_expectation.png similarity index 100% rename from docs/usage/assets/expectation_customize_expiration_time.png rename to docs/usage/expectations/assets/create_manual_expectation.png diff --git a/docs/usage/assets/inject_expectations_list.png b/docs/usage/expectations/assets/inject_expectations_list.png similarity index 100% rename from docs/usage/assets/inject_expectations_list.png rename to docs/usage/expectations/assets/inject_expectations_list.png diff --git a/docs/usage/assets/manual_expectation_validation_animation_tab.png b/docs/usage/expectations/assets/manual_expectation_validation_animation_tab.png similarity index 100% rename from docs/usage/assets/manual_expectation_validation_animation_tab.png rename to docs/usage/expectations/assets/manual_expectation_validation_animation_tab.png diff --git a/docs/usage/assets/validation_mode.png b/docs/usage/expectations/assets/validation_mode.png similarity index 100% rename from docs/usage/assets/validation_mode.png rename to docs/usage/expectations/assets/validation_mode.png diff --git a/docs/usage/expectations/management.md b/docs/usage/expectations/management.md new file mode 100644 index 00000000..23f330b1 --- /dev/null +++ b/docs/usage/expectations/management.md @@ -0,0 +1,81 @@ +## Expectations — Scoring, Expiration & UI + +This page explains how to **configure, validate and operate expectations**. + +## Add an Expectation to an Inject + +1. Open the Inject +2. Click **Add expectations** +3. Select the expectation type +4. Configure: score, validation mode, expiration time (optional) + +💡 You can attach **multiple expectations** to a single Inject. + +![Create manual expectation](assets/create_manual_expectation.png) + +## Validate Expectations + +### Manual expectations + +* During a simulation: **Animation → Validation** +* During atomic testing: **Overview** tab + +![Validate a manual expectation from the animation tab](assets/manual_expectation_validation_animation_tab.png) + +### Technical expectations + +* Automatically validated via collectors +* Or validated manually by adding detection or prevention results + +![Add the validation of a technical expectation in atomic testing](assets/add_technical_expectation_validation.png) + +## Scoring + +### Default score + +Each expectation type has a **default score**, applied at creation. + +| Type | Default score | +| ------ | ------------- | +| Manual | 50 | + +Configured via environment variables. + +| Parameter | Environment variable | Default value | Description | +|:-----------------------------------------------|:-----------------------------------------------|:--------------|:-------------------------------------------| +| openaev.expectation.manual.default-score-value | OPENAEV_EXPECTATION_MANUAL_DEFAULT-SCORE-VALUE | 50 | Default score value for manual expectation | + +## Expiration Time + +Expectations must validate **within a defined time window**. + +If the time expires: + +* the expectation **fails automatically** +* the result is marked as: + + * `Not Detected` + * `Not Prevented` + * or equivalent + +### Default expiration values + +| Expectation type | Default | +| ---------------------------------- | -------- | +| Detection / Prevention | 6 hours | +| Human (manual, article, challenge) | 24 hours | + +You can override expiration times: + +* globally (environment variables) +* per expectation (UI) + +| Parameter | Environment variable | Default value | Description | +|:-----------------------------------------------|:-----------------------------------------------|:--------------|:--------------------------------------------------------------------| +| openaev.expectation.technical.expiration-time | OPENAEV_EXPECTATION_TECHNICAL_EXPIRATION-TIME | 21600 | Expiration time for Technical expectation (detection & prevention) | +| openaev.expectation.detection.expiration-time | OPENAEV_EXPECTATION_DETECTION_EXPIRATION-TIME | 21600 | Expiration time for detection expectation | +| openaev.expectation.prevention.expiration-time | OPENAEV_EXPECTATION_PREVENTION_EXPIRATION-TIME | 21600 | Expiration time for prevention expectation | +| openaev.expectation.human.expiration-time | OPENAEV_EXPECTATION_HUMAN_EXPIRATION-TIME | 86400 | Expiration time for human expectation (manual, challenge & article) | +| openaev.expectation.challenge.expiration-time | OPENAEV_EXPECTATION_CHALLENGE_EXPIRATION-TIME | 86400 | Expiration time for challenge expectation | +| openaev.expectation.article.expiration-time | OPENAEV_EXPECTATION_ARTICLE_EXPIRATION-TIME | 86400 | Expiration time for article expectation | +| openaev.expectation.manual.expiration-time | OPENAEV_EXPECTATION_MANUAL_EXPIRATION-TIME | 86400 | Expiration time for manual expectation | diff --git a/docs/usage/expectations/overview.md b/docs/usage/expectations/overview.md new file mode 100644 index 00000000..d1b0c1d1 --- /dev/null +++ b/docs/usage/expectations/overview.md @@ -0,0 +1,109 @@ +# Expectations + +This page explains how Expectations work in OpenAEV: what they are, why they matter, how to configure them, and how they +impact the outcome of a simulation. + +Expectations define what success means when an Inject targets an [Asset (endpoint)](../assets.md) or +a [Player](../people.md#players) during an [Inject](../inject-overview.md) execution. + +## What are Expectations? + +### What is this? + +An **Expectation** defines a **measurable outcome** for an Inject. +It answers the question: + +> *What should happen for this Inject to be considered successful?* + +Each expectation: + +* is attached to an Inject +* has a **score** +* has a **validation mode** +* is validated either **manually** or **automatically** +* contributes to the **simulation result** + +### Why use Expectations? + +Expectations allow you to: + +* Objectively measure **security posture** +* Evaluate both **technical controls** and **human behavior** +* Standardize scoring across exercises +* Decide when a **simulation is complete** + +## Expectation Types (Overview) + +Expectations fall into **two categories**, depending on how validation occurs. + +### Manual Expectations (Human validation) + +Manual expectations are validated by an **organizer**. +They are used to evaluate **human actions** such as decision-making or escalation. + +⚠️ These expectations can only be used in the context of human-based injects, such as email or SMS, where a real person +is expected to read, interpret, and react to inject. + +Typical use cases: + +* tabletop exercises +* incident response drills +* awareness and training scenarios + +### Automatic Expectations (Technical validation) + +Automatic expectations are validated by **technical signals**. + +They rely on: + +* connected security tools +* or manually provided technical results + +⚠️ These expectations can only be used in the context of technical injects, where system or security signals are +produced automatically (for example: logs, alerts, detections, or telemetry). + +They are essential for **technical simulations** and **Breach & Attack Simulation** use cases. + +## Validation Modes (Overview) + +A **Validation Mode** defines how individual target results aggregate at **group level**. + +Each expectation uses **exactly one** validation mode. + +### All targets must validate + +* All targets succeed → **100** +* At least one target fails → **0** + +Typical use cases: + +* compliance checks +* mandatory training +* baseline security requirements + +### At least one target must validate + +* ≥ 1 success → **100** +* No success → **0** + +Typical use cases: + +* SOC detection +* escalation workflows +* redundancy testing + +![Validation mode](assets/validation_mode.png) + +## When is a Simulation Finished? + +A simulation is considered **finished** once **all expectations are resolved**: + +* Technical expectations → detection, prevention, vulnerability +* Human expectations → manual validation + +Until then, the simulation remains **in progress**. + +## Go further + +* [Validation, aggregation & propagation](validation.md) +* [Scoring, expiration & UI](management.md) \ No newline at end of file diff --git a/docs/usage/expectations/validation.md b/docs/usage/expectations/validation.md new file mode 100644 index 00000000..825294d8 --- /dev/null +++ b/docs/usage/expectations/validation.md @@ -0,0 +1,189 @@ +## Expectations — Validation, Aggregation & Propagation + +This page explains **how OpenAEV computes expectation results**, from individual signals to organizational outcomes. + +## Manual Expectations (Detailed) + +A **Manual Expectation** requires validation by the **organizer**. + +OpenAEV does not infer the result — a human explicitly decides whether the expectation is met. + +### When should I use it? + +Use manual expectations to evaluate **human-driven actions** that cannot be detected technically: + +- decision-making +- communication +- process adherence + +### How does it work? + +1. Add a **Manual Expectation** to an Inject. + ![Create manual expectation](assets/create_manual_expectation.png) +2. Define: score, validation mode, expiration time +3. During the simulation, manually validate the expectation from the UI. + ![Validate a manual expectation from the animation tab](assets/manual_expectation_validation_animation_tab.png) + +### Example + +You simulate a phishing campaign. + +Expectation: + +> *The Incident Response team acknowledges and escalates the phishing email.* + +- The SOC receives the inject +- The analyst follows the playbook and opens a ticket +- The organizer validates the expectation + +✅ Result: **Success (100)** + +## Automatic Expectations (Detailed) + +An **Automatic Expectation** is validated when a **technical condition** is met. + +Validation relies on: + +- connected security integrations +- or manually provided technical results + +### Why use it? + +Automatic expectations allow you to: + +- Measure **real security controls** +- Avoid subjective validation +- Scale exercises across many Assets and Agents + +They are essential for **technical simulations** and **Breach & Attack Simulation** use cases. + +### Available automatic expectation types + +| Type | What it validates | +|---------------|-----------------------------------------| +| Prevention | A control blocks the attack | +| Detection | A security alert or incident is raised | +| Vulnerability | A CVE is present on the target | +| Article | Targets read or acknowledge the article | + +![Add expectations to an inject](assets/inject_expectations_list.png) + +### Example + +You launch a malware simulation. + +Expectation: + +> *Endpoint protection detects the payload.* + +- The EDR raises an alert +- OpenAEV receives the detection event + +✅ Result: **Success (100)** + +## Validation Modes (Detailed) + +A **Validation Mode** defines how individual target results are **aggregated at group level**. + +Each expectation uses **exactly one** validation mode. + +### All targets must validate + +#### When should I use it? + +Use this mode when **every member must succeed**, such as: + +- mandatory training +- compliance requirements +- baseline security checks + +#### How does it work? + +- All targets succeed → **100** +- At least one target fails → **0** + +#### Example + +Group of 4 players: + +- 3 succeed, 1 fails → **0 (Failed)** +- 4 succeed → **100 (Success)** + +### At least one target must validate + +#### When should I use it? + +Use this mode when **a single successful response is enough**, such as: + +- SOC detection +- on-call escalation +- redundancy testing + +#### How does it work? + +- ≥ 1 success → **100** +- No success → **0** + +#### Example + +Group of 4 players: + +- 1 succeeds, 3 fail → **100 (Success)** +- 4 fail → **0 (Failed)** + +## Result Aggregation per Expectation + +### What is this? + +A single **Expectation** can receive **multiple validation results** for the same Inject. + +This happens when: + +- several security tools monitor the same Asset +- multiple collectors report results +- results are added both manually and automatically + +![Add the validation of a technical expectation in atomic testing](assets/add_technical_expectation_validation.png) + +### Why does OpenAEV work this way? + +OpenAEV evaluates **effective protection**, not individual tool accuracy. + +If at least one control detects or prevents the attack, the organization **did not fully fail**. + +This avoids: + +- false negatives caused by tool overlap +- penalizing layered defense strategies + +### How is the final result computed? + +1. All results are collected +2. Results are ordered by **severity** +3. The **highest result always wins** + +⚠️ A negative result never overrides a positive one. + +### Example + +| Tool | Result | +|--------------------|--------------| +| Microsoft Defender | Detected | +| CrowdStrike | Not Detected | + +Final expectation result: +✅ **Detected** + +## Status Propagation + +Expectation results propagate from **technical entities** to **organizational entities**. + +| Entity | Rule | +|-------------|--------------------------------------| +| Agent | Direct result (0 or 100) | +| Asset | Valid only if **all Agents succeed** | +| Asset Group | Depends on validation mode | +| Player | Depends on validation mode | +| Team | Aggregated from Players | + +⚠️ Validation mode always applies at **group level**. \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index ce124f40..a5737284 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -153,7 +153,10 @@ nav: - Search and Filter: usage/inject-result-list.md - Inject result: usage/inject-result.md - Targets: usage/targets.md - - Expectations: usage/expectations.md + - Expectations: + - Overview: usage/expectations/overview.md + - Validation: usage/expectations/validation.md + - Management: usage/expectations/management.md - Findings: usage/findings.md - Dashboards: - Custom Dashboards: usage/dashboards/custom-dashboards/custom-dashboards.md