Skip to content

✨ Concurrent E2E Test Execution #2675

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dtfranz wants to merge 1 commit into
base: main
Choose a base branch
from
Open

✨ Concurrent E2E Test Execution #2675

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,11 +254,23 @@ $(eval $(call install-sh,standard,operator-controller-standard.yaml))
.PHONY: test
test: manifests generate fmt lint test-unit test-e2e test-regression #HELP Run all tests.

E2E_TIMEOUT ?= 20m
GODOG_ARGS ?=
.PHONY: e2e
e2e: E2E_TIMEOUT ?= 20m
e2e: GODOG_ARGS ?=
e2e: #EXHELP Run the e2e tests.
go test -count=1 -v ./test/e2e/features_test.go -timeout=$(E2E_TIMEOUT) $(if $(GODOG_ARGS),-args $(GODOG_ARGS))
ifeq ($(strip $(GODOG_ARGS)),)
set +e; \
Copy link
Copy Markdown
Contributor

@pedjak pedjak May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (non-blocking): with set +e, Ctrl+C during the parallel phase kills go test but the serial phase still starts — the user needs to Ctrl+C a second time. Adding a trap before set +e would make it exit immediately:

	trap 'exit 130' INT; \
	set +e; \

Minor ergonomic improvement for local dev. Fine to skip or do in a follow-up.

go test -count=1 -v ./test/e2e/features_test.go -timeout=${E2E_TIMEOUT} -args --godog.tags="~@Serial" --godog.concurrency=100; \
parallelExit=$$?; \
go test -count=1 -v ./test/e2e/features_test.go -timeout=${E2E_TIMEOUT} -args --godog.tags="@Serial" --godog.concurrency=1; \
serialExit=$$?; \
if [[ $$parallelExit -ne 0 ]] || [[ $$serialExit -ne 0 ]]; then \
echo "e2e tests failed: parallel=$$parallelExit serial=$$serialExit"; \
exit 1; \
fi
Comment on lines +262 to +270
Copy link
Copy Markdown
Contributor Author

@dtfranz dtfranz May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was an intentional decision to avoid complexity, as doing a timeout removes the ability for developers to command-line interrupt their tests.

else
go test -count=1 -v ./test/e2e/features_test.go -timeout=$(E2E_TIMEOUT) -args $(GODOG_ARGS)
endif

export CLUSTER_REGISTRY_HOST := docker-registry.operator-controller-e2e.svc:5000
.PHONY: extension-developer-e2e
Expand Down Expand Up @@ -316,7 +328,7 @@ test-experimental-e2e: COVERAGE_NAME := experimental-e2e
test-experimental-e2e: export MANIFEST := $(EXPERIMENTAL_RELEASE_MANIFEST)
test-experimental-e2e: export INSTALL_DEFAULT_CATALOGS := false
test-experimental-e2e: PROMETHEUS_VALUES := helm/prom_experimental.yaml
test-experimental-e2e: E2E_TIMEOUT := 25m
test-experimental-e2e: E2E_TIMEOUT ?= 25m
Copy link
Copy Markdown
Contributor Author

@dtfranz dtfranz May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My intent was to allow overriding here - not realizing that the way the timeout is define above in e2e sets it initially for the whole file, which negates the ?=. This should be fixed now that I've set the definition inside the e2e target instead of globally.

test-experimental-e2e: run-internal prometheus e2e e2e-coverage kind-clean #HELP Run experimental e2e test suite on local kind cluster

.PHONY: prometheus
Expand Down
35 changes: 35 additions & 0 deletions test/e2e/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,10 @@ Use these variables in YAML templates:

### 5. Feature Tags

Tags can be used for different purposes in the test suite:

#### Feature Gate Tags

Use tags to conditionally run scenarios based on feature gates:

```gherkin
Expand All @@ -216,6 +220,28 @@ Scenario: Install operator having webhooks

Scenarios are skipped if the feature gate is not enabled on the deployed controller.

#### Serial Execution Tag

By default, scenarios run concurrently (up to 100 parallel scenarios). However, some tests must run serially, typically because they:
- Modify shared cluster resources (e.g., cluster-wide TLS configuration)
- Have resource constraints that prevent parallel execution
- Require exclusive access to a resource

To mark a test for serial execution, add the `@Serial` tag:

```gherkin
@Serial
Feature: TLS profile enforcement on metrics endpoints

Scenario: Test TLS configuration
Given the "catalogd" deployment is configured with custom TLS settings
...
```

The `Makefile` automatically separates scenarios when run without additional `GODOG_ARGS`:
- Scenarios **without** `@Serial` run concurrently in the first test phase
- Scenarios **with** `@Serial` run sequentially in a separate serial test phase

## Running Tests

### Run All Tests
Expand All @@ -230,6 +256,15 @@ or
make test-experimental-e2e
```

Custom godog arguments can be modified by setting the following:
```bash
GODOG_ARGS=--godog.tags=@WebhookProviderCertManager make test-experimental-e2e
```

Note that when this is done the `make` target will no longer automatically split the test run into parallel and serial runs, and test execution time may increase. If you wish to add concurrency back into the arguments, it is recommended to also disable the `@Serial` tests:
```bash
GODOG_ARGS="--godog.tags=~@Serial --godog.concurrency=100" make test-experimental-e2e
```
Comment thread
dtfranz marked this conversation as resolved.

### Run Specific Feature

Expand Down
1 change: 1 addition & 0 deletions test/e2e/features/ha.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
@Serial
Feature: HA failover for catalogd

When catalogd is deployed with multiple replicas, the remaining pods must
Expand Down
1 change: 1 addition & 0 deletions test/e2e/features/proxy.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
@Serial
Feature: HTTPS proxy support for outbound catalog requests

OLM's operator-controller fetches catalog data from catalogd over HTTPS.
Expand Down
2 changes: 2 additions & 0 deletions test/e2e/features/revision.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -672,6 +672,7 @@ Feature: Install ClusterObjectSet
And resource "deployment/test-deployment" is eventually not found

@ProgressDeadline
@Serial
Scenario: COS recovers from ProgressDeadlineExceeded to Succeeded when probes pass
Given min value for ClusterObjectSet .spec.progressDeadlineMinutes is set to 1
And ServiceAccount "olm-sa" with needed permissions is available in test namespace
Expand Down Expand Up @@ -721,6 +722,7 @@ Feature: Install ClusterObjectSet
containers:
- name: delayed-ready
image: busybox:1.36
imagePullPolicy: IfNotPresent
command: ["sleep", "1000"]
readinessProbe:
exec:
Expand Down
5 changes: 5 additions & 0 deletions test/e2e/features/tls.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
@Serial
Feature: TLS profile enforcement on metrics endpoints
Comment thread
dtfranz marked this conversation as resolved.

Background:
Expand All @@ -6,11 +7,15 @@ Feature: TLS profile enforcement on metrics endpoints
# Each scenario patches the deployment with the TLS settings under test and
# restores the original configuration during cleanup, so scenarios are independent.

# This feature file is run serially to avoid potential issues modifying the catalogd
# deployment during functional testing.
Comment thread
dtfranz marked this conversation as resolved.

# All three scenarios test catalogd only: the enforcement logic lives in the shared
# tlsprofiles package, so one component is sufficient. TLS 1.2 is used for cipher
# and curve enforcement because Go's crypto/tls does not allow the server to restrict
# TLS 1.3 cipher suites — CipherSuites config only applies to TLS 1.2. The e2e cert
# uses ECDSA, so ECDHE_ECDSA cipher families are required.

@TLSProfile
Scenario: catalogd metrics endpoint enforces configured minimum TLS version
Given the "catalogd" deployment is configured with custom TLS minimum version "TLSv1.3"
Expand Down