Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file not shown.
Binary file not shown.
Binary file not shown.
156 changes: 25 additions & 131 deletions docs/usage/expectations.md
Original file line number Diff line number Diff line change
@@ -1,149 +1,43 @@
# Expectations

Expectations define what is expected from an [Asset (endpoint)](assets.md) or
a [Players](people.md#players) when facing an [Inject](inject-overview.md) in terms of
security posture. Each expectation has a score representing how well it has been met by the target.

## Expectation types
#### Why use it?

Expectations can be categorized as either Manual or Automatic, depending on how they are validated.
Use this mode when **every member must succeed**, such as:

### Manual expectations
* mandatory training
* compliance requirements
* baseline security checks

Manual expectations require validation by the **exercise organizer** (animation team).
They are simple, customizable, and **user-controlled**.
#### How does it work?

Examples:
* All targets succeed → **100**
* At least one target fails → **0**

- **Team response validation**
Check whether the **incident response team** correctly acknowledges and escalates a phishing inject during the exercise.
#### Example

- **Player task validation**
Ensure that an **analyst player** correctly follows the playbook and reports an incident in the ticketing system.
Group of 4 players:

### Automatic expectations
* 3 succeed, 1 fails → **0 (Failed)**
* 4 succeed → **100 (Success)**

Automatic expectations are validated automatically under specific conditions.
### At least one target must validate

- `Prevention`: automatically validated by security integrations (e.g., quarantine event).
- `Detection`: automatically validated by security integrations (e.g., incident alert).
- `Vulnerability`: automatically validated based on the presence of CVEs.
- `Expect targets to read the article(s)`: automatically validated once targets have read the injected article.
#### Why use it?

## Validation Mode
Use this mode when **a single successful response is enough**, such as:

There are two modes for validating an expectation :
* SOC detection
* on-call escalation
* redundancy testing

### All targets (per group) must validate the expectation
#### How does it work?

- The result depends on **every member's performance**.
- If one target fails, the entire group fails.
- Final score = **100** if all succeed, otherwise **0**.
* ≥ 1 success → **100**
* No success → **0**

Example: 3 players succeed, 1 fails → Group score = **0 (Failed)**.
Example: 4 players succeed → Group score = **100 (Success)**.
#### Example

### At least one target (per group) must validate the expectation
Group of 4 players:

- Success depends on **at least one target succeeding**.
- The group is considered successful if one target validates.
- Final score = **100** if ≥1 succeeds, otherwise **0**.

Example: 3 players fail, 1 succeeds → Group score = **100 (Success)**.
Example: 4 players fail → Group score = **0 (Failed)**.

![Validation mode](assets/validation_mode.png)

## Expectation manipulation

### Add an expectation to an Inject

1. Navigate to the inject's content and click **Add expectations**.
2. Select the type of expectation you want to add.
3. Define its score and validation mode.

You can add multiple expectations to a single inject.

![Add expectations to an inject](assets/inject_expectations_list.png)

### Validate a manual expectation

- During a simulation, go to the **Animation tab → Validation screen** to manually validate expectations.

![Validate a manual expectation from the animation tab](assets/manual_expectation_validation_animation_tab.png)

- For atomic testing with manual injects, validation can be done directly in the **Overview tab**.

![Add the validation of a manual expectation in atomic testing](assets/add_manual_validation_atomic_testing.png)

![Add the validation of a manual expectation in atomic testing](assets/manual_expectation_validation_atomic_testing.png)

### Validate technical injects

- Users can manually add detection/prevention results for their custom security platforms.
- Automatic validation will also occur when connected to compatible collectors.

![Add the validation of a technical expectation in atomic testing](assets/add_technical_expectation_validation.png)

![Add the validation of a technical expectation in atomic testing](assets/technical_expectation_validation.png)

### Update Rules

Expectation statuses propagate between entities as follows:

| Action | Behavior |
|--------------------|------------------------------------------------------------------------|
| Update Agent | Update its status (0 = fail, 100 = success). |
| Update Asset | Asset becomes **valid (100)** only if **all its agents are valid**. |
| Update Asset Group | Compute status depending on validation mode: |
| | - **All assets must validate** → 100 only if all assets succeed. |
| | - **At least one asset must validate** → 100 if at least one succeeds. |
| Update Player | Compute status for the team depending on validation mode. |
| Update Team | Propagate status to players. |

### Customize expectations

#### Default score

Each expectation has a **default score value** at creation, configurable via environment variables:

| Parameter | Environment variable | Default value | Description |
|:-----------------------------------------------|:-----------------------------------------------|:--------------|:-------------------------------------------|
| openaev.expectation.manual.default-score-value | OPENAEV_EXPECTATION_MANUAL_DEFAULT-SCORE-VALUE | 50 | Default score value for manual expectation |

#### Expiration time

Expectations must be validated within a time limit. Defaults are set in the system but can be overridden:

- Manual Response: By default if not manual expectations are set the expiration will done after 24 hours
- Detection/Prevention: By default if no expectations are set the expiration will done after 6 hour

| Parameter | Environment variable | Default value | Description |
|:-----------------------------------------------|:-----------------------------------------------|:--------------|:--------------------------------------------------------------------|
| openaev.expectation.technical.expiration-time | OPENAEV_EXPECTATION_TECHNICAL_EXPIRATION-TIME | 21600 | Expiration time for Technical expectation (detection & prevention) |
| openaev.expectation.detection.expiration-time | OPENAEV_EXPECTATION_DETECTION_EXPIRATION-TIME | 21600 | Expiration time for detection expectation |
| openaev.expectation.prevention.expiration-time | OPENAEV_EXPECTATION_PREVENTION_EXPIRATION-TIME | 21600 | Expiration time for prevention expectation |
| openaev.expectation.human.expiration-time | OPENAEV_EXPECTATION_HUMAN_EXPIRATION-TIME | 86400 | Expiration time for human expectation (manual, challenge & article) |
| openaev.expectation.challenge.expiration-time | OPENAEV_EXPECTATION_CHALLENGE_EXPIRATION-TIME | 86400 | Expiration time for challenge expectation |
| openaev.expectation.article.expiration-time | OPENAEV_EXPECTATION_ARTICLE_EXPIRATION-TIME | 86400 | Expiration time for article expectation |
| openaev.expectation.manual.expiration-time | OPENAEV_EXPECTATION_MANUAL_EXPIRATION-TIME | 86400 | Expiration time for manual expectation |

A default expiration time is set for technical and human expectations. Users can override them for each type of
expectations.

- In the UI

![Customize expiration time in UI](assets/expectation_customize_expiration_time.png)

When creating an expectation, users can set an expiration time. The system's default times are set on the form and users
decide to override it.

Once the expiration time will be reached, the Expectation Expiration Manager will fail the expectation and will show the
result as `Not Detected` or `Not Prevented` depending on the expectation's type.

## When is a Simulation Finished?

A simulation is considered **finished** once all defined expectations are resolved:

- **Technical expectations** → validated through **Detection, Prevention, Vulnerability checks**.
- **Human expectations** → validated manually by the organizer.
* 1 succeeds, 3 fail → **100 (Success)**
* 4 fail → **0 (Failed)**
81 changes: 81 additions & 0 deletions docs/usage/expectations/management.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
## Expectations — Scoring, Expiration & UI

This page explains how to **configure, validate and operate expectations**.

## Add an Expectation to an Inject

1. Open the Inject
2. Click **Add expectations**
3. Select the expectation type
4. Configure: score, validation mode, expiration time (optional)

💡 You can attach **multiple expectations** to a single Inject.

![Create manual expectation](assets/create_manual_expectation.png)

## Validate Expectations

### Manual expectations

* During a simulation: **Animation → Validation**
* During atomic testing: **Overview** tab

![Validate a manual expectation from the animation tab](assets/manual_expectation_validation_animation_tab.png)

### Technical expectations

* Automatically validated via collectors
* Or validated manually by adding detection or prevention results

![Add the validation of a technical expectation in atomic testing](assets/add_technical_expectation_validation.png)

## Scoring

### Default score

Each expectation type has a **default score**, applied at creation.

| Type | Default score |
| ------ | ------------- |
| Manual | 50 |

Configured via environment variables.

| Parameter | Environment variable | Default value | Description |
|:-----------------------------------------------|:-----------------------------------------------|:--------------|:-------------------------------------------|
| openaev.expectation.manual.default-score-value | OPENAEV_EXPECTATION_MANUAL_DEFAULT-SCORE-VALUE | 50 | Default score value for manual expectation |

## Expiration Time

Expectations must validate **within a defined time window**.

If the time expires:

* the expectation **fails automatically**
* the result is marked as:

* `Not Detected`
* `Not Prevented`
* or equivalent

### Default expiration values

| Expectation type | Default |
| ---------------------------------- | -------- |
| Detection / Prevention | 6 hours |
| Human (manual, article, challenge) | 24 hours |

You can override expiration times:

* globally (environment variables)
* per expectation (UI)

| Parameter | Environment variable | Default value | Description |
|:-----------------------------------------------|:-----------------------------------------------|:--------------|:--------------------------------------------------------------------|
| openaev.expectation.technical.expiration-time | OPENAEV_EXPECTATION_TECHNICAL_EXPIRATION-TIME | 21600 | Expiration time for Technical expectation (detection & prevention) |
| openaev.expectation.detection.expiration-time | OPENAEV_EXPECTATION_DETECTION_EXPIRATION-TIME | 21600 | Expiration time for detection expectation |
| openaev.expectation.prevention.expiration-time | OPENAEV_EXPECTATION_PREVENTION_EXPIRATION-TIME | 21600 | Expiration time for prevention expectation |
| openaev.expectation.human.expiration-time | OPENAEV_EXPECTATION_HUMAN_EXPIRATION-TIME | 86400 | Expiration time for human expectation (manual, challenge & article) |
| openaev.expectation.challenge.expiration-time | OPENAEV_EXPECTATION_CHALLENGE_EXPIRATION-TIME | 86400 | Expiration time for challenge expectation |
| openaev.expectation.article.expiration-time | OPENAEV_EXPECTATION_ARTICLE_EXPIRATION-TIME | 86400 | Expiration time for article expectation |
| openaev.expectation.manual.expiration-time | OPENAEV_EXPECTATION_MANUAL_EXPIRATION-TIME | 86400 | Expiration time for manual expectation |
109 changes: 109 additions & 0 deletions docs/usage/expectations/overview.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# Expectations

This page explains how Expectations work in OpenAEV: what they are, why they matter, how to configure them, and how they
impact the outcome of a simulation.

Expectations define what success means when an Inject targets an [Asset (endpoint)](../assets.md) or
a [Player](../people.md#players) during an [Inject](../inject-overview.md) execution.

## What are Expectations?

### What is this?

An **Expectation** defines a **measurable outcome** for an Inject.
It answers the question:

> *What should happen for this Inject to be considered successful?*

Each expectation:

* is attached to an Inject
* has a **score**
* has a **validation mode**
* is validated either **manually** or **automatically**
* contributes to the **simulation result**

### Why use Expectations?

Expectations allow you to:

* Objectively measure **security posture**
* Evaluate both **technical controls** and **human behavior**
* Standardize scoring across exercises
* Decide when a **simulation is complete**

## Expectation Types (Overview)

Expectations fall into **two categories**, depending on how validation occurs.

### Manual Expectations (Human validation)

Manual expectations are validated by an **organizer**.
They are used to evaluate **human actions** such as decision-making or escalation.

⚠️ These expectations can only be used in the context of human-based injects, such as email or SMS, where a real person
is expected to read, interpret, and react to inject.

Typical use cases:

* tabletop exercises
* incident response drills
* awareness and training scenarios

### Automatic Expectations (Technical validation)

Automatic expectations are validated by **technical signals**.

They rely on:

* connected security tools
* or manually provided technical results

⚠️ These expectations can only be used in the context of technical injects, where system or security signals are
produced automatically (for example: logs, alerts, detections, or telemetry).

They are essential for **technical simulations** and **Breach & Attack Simulation** use cases.

## Validation Modes (Overview)

A **Validation Mode** defines how individual target results aggregate at **group level**.

Each expectation uses **exactly one** validation mode.

### All targets must validate

* All targets succeed → **100**
* At least one target fails → **0**

Typical use cases:

* compliance checks
* mandatory training
* baseline security requirements

### At least one target must validate

* ≥ 1 success → **100**
* No success → **0**

Typical use cases:

* SOC detection
* escalation workflows
* redundancy testing

![Validation mode](assets/validation_mode.png)

## When is a Simulation Finished?

A simulation is considered **finished** once **all expectations are resolved**:

* Technical expectations → detection, prevention, vulnerability
* Human expectations → manual validation

Until then, the simulation remains **in progress**.

## Go further

* [Validation, aggregation & propagation](validation.md)
* [Scoring, expiration & UI](management.md)
Loading