Skip to content

devops: run Docker container tests in tests_others workflow#41073

Open
dgozman wants to merge 1 commit into
microsoft:mainfrom
dgozman:docker-tests-others
Open

devops: run Docker container tests in tests_others workflow#41073
dgozman wants to merge 1 commit into
microsoft:mainfrom
dgozman:docker-tests-others

Conversation

@dgozman
Copy link
Copy Markdown
Collaborator

@dgozman dgozman commented Jun 1, 2026

Summary

  • Port the Docker container smoke tests from the playwright-browsers repo into the tests others workflow
  • Builds the image for Ubuntu jammy/noble on amd64/arm64 and runs the @smoke suite inside the container
  • Azure/ACR pull-through-cache and flakiness upload run only on microsoft/playwright; forks fall back to Docker Hub
  • Uploads results via the shared upload-blob-report action

@dgozman dgozman added the CQ1 label Jun 1, 2026
@github-actions

This comment has been minimized.

Sign in to view
@github-actions

This comment has been minimized.

Sign in to view
@github-actions

This comment has been minimized.

Sign in to view
@github-actions

This comment has been minimized.

Sign in to view
@dgozman
devops: run Docker container tests in tests_others workflow
fdd176c 
Port the Docker container smoke tests from the playwright-browsers repo
into the tests_others workflow. Builds the image for Ubuntu jammy/noble
on amd64/arm64 and runs the @smoke suite inside the container. Azure/ACR
pull-through-cache and flakiness upload run only on microsoft/playwright;
forks fall back to Docker Hub.
@dgozman dgozman force-pushed the docker-tests-others branch from a465146 to fdd176c Compare June 1, 2026 21:10
@dgozman dgozman added CQ1 and removed CQ1 labels Jun 1, 2026
yury-s
yury-s approved these changes Jun 1, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@yury-s yury-s left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd run docker tests only in playwright-browsers

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 1, 2026

Test results for "MCP"

2 failed
❌ [chrome] › mcp/launch.spec.ts:64 › persistent context @mcp-ubuntu-latest-chrome
❌ [webkit] › mcp/cli-test.spec.ts:23 › debug test and snapshot @mcp-macos-latest-webkit

7228 passed, 1103 skipped

Merge workflow run.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 1, 2026

Test results for "tests 1"

3 flaky ⚠️ [installation tests] › playwright-test-package-managers.spec.ts:54 › npm: uninstalling ct removes playwright bin `@package-installations-macos-latest`
⚠️ [chromium-library] › library/browsertype-connect.spec.ts:699 › run-server › should record trace with sources `@chromium-ubuntu-22.04-arm-node20`
⚠️ [firefox-library] › library/har-websocket.spec.ts:170 › should record websocket connection failure `@firefox-ubuntu-22.04-node20`

39450 passed, 775 skipped

Merge workflow run.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 1, 2026

Test results for "tests others"

1 failed
❌ [chromium-library] › library/screenshot.spec.ts:215 › page screenshot › should not hang when event loop is blocked @frozen-time-library-chromium-linux

18 flaky ⚠️ [electron-page] › page/page-localstorage.spec.ts:49 › localStorage.removeItem removes a single item `@electron-ubuntu-latest`
⚠️ [electron-page] › page/page-navigation.spec.ts:28 › should work with cross-process _blank target `@electron-ubuntu-latest`
⚠️ [electron-page] › page/page-navigation.spec.ts:36 › should work with _blank target in form `@electron-ubuntu-latest`
⚠️ [electron-page] › page/page-request-continue.spec.ts:398 › should respect set-cookie in redirect response `@electron-ubuntu-latest`
⚠️ [electron-page] › page/page-route.spec.ts:74 › should not support ? in glob pattern `@electron-ubuntu-latest`
⚠️ [electron-page] › page/page-route.spec.ts:169 › should not override cookie header `@electron-ubuntu-latest`
⚠️ [electron-page] › page/page-localstorage.spec.ts:49 › localStorage.removeItem removes a single item `@electron-macos-latest`
⚠️ [electron-page] › page/page-navigation.spec.ts:28 › should work with cross-process _blank target `@electron-macos-latest`
⚠️ [electron-page] › page/page-navigation.spec.ts:36 › should work with _blank target in form `@electron-macos-latest`
⚠️ [electron-page] › page/page-request-continue.spec.ts:398 › should respect set-cookie in redirect response `@electron-macos-latest`
⚠️ [electron-page] › page/page-route.spec.ts:74 › should not support ? in glob pattern `@electron-macos-latest`
⚠️ [electron-page] › page/page-route.spec.ts:169 › should not override cookie header `@electron-macos-latest`
⚠️ [electron-page] › page/page-localstorage.spec.ts:49 › localStorage.removeItem removes a single item `@electron-windows-latest`
⚠️ [electron-page] › page/page-navigation.spec.ts:28 › should work with cross-process _blank target `@electron-windows-latest`
⚠️ [electron-page] › page/page-navigation.spec.ts:36 › should work with _blank target in form `@electron-windows-latest`
⚠️ [electron-page] › page/page-request-continue.spec.ts:398 › should respect set-cookie in redirect response `@electron-windows-latest`
⚠️ [electron-page] › page/page-route.spec.ts:74 › should not support ? in glob pattern `@electron-windows-latest`
⚠️ [electron-page] › page/page-route.spec.ts:169 › should not override cookie header `@electron-windows-latest`

19935 passed, 660 skipped

Merge workflow run.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 1, 2026

Test results for "tests 2"

1 fatal errors, not part of any test
19 failed
❌ [installation tests] › playwright-electron-should-work.spec.ts:21 › electron should work @package-installations-ubuntu-latest-node26
❌ [installation tests] › playwright-electron-should-work.spec.ts:31 › electron should work with special characters in path @package-installations-ubuntu-latest-node26
❌ [installation tests] › playwright-electron-should-work.spec.ts:44 › should work when wrapped inside @playwright/test and trace is enabled @package-installations-ubuntu-latest-node26
❌ [chromium-library] › library/heap.spec.ts:105 › should not leak dispatchers after closing page @tracing-chromium
❌ [chromium-library] › library/screenshot.spec.ts:215 › page screenshot › should not hang when event loop is blocked @tracing-chromium
❌ [chromium-library] › library/heap.spec.ts:98 › should not leak server-side objects @driver
❌ [installation tests] › playwright-electron-should-work.spec.ts:21 › electron should work @package-installations-ubuntu-latest-node24
❌ [installation tests] › playwright-electron-should-work.spec.ts:31 › electron should work with special characters in path @package-installations-ubuntu-latest-node24
❌ [installation tests] › playwright-electron-should-work.spec.ts:44 › should work when wrapped inside @playwright/test and trace is enabled @package-installations-ubuntu-latest-node24
❌ [firefox-library] › library/heap.spec.ts:105 › should not leak dispatchers after closing page @tracing-firefox
❌ [firefox-library] › library/screenshot.spec.ts:215 › page screenshot › should not hang when event loop is blocked @tracing-firefox
❌ [webkit-library] › library/heap.spec.ts:105 › should not leak dispatchers after closing page @tracing-webkit
❌ [webkit-library] › library/screenshot.spec.ts:215 › page screenshot › should not hang when event loop is blocked @tracing-webkit
❌ [webkit-library] › library/har-websocket.spec.ts:60 › should include websocket handshake headers and status @webkit-windows-latest
❌ [webkit-library] › library/har-websocket.spec.ts:96 › should include websocket messages @webkit-windows-latest
❌ [webkit-library] › library/har-websocket.spec.ts:133 › should include binary websocket messages @webkit-windows-latest
❌ [webkit-library] › library/har-websocket.spec.ts:193 › should record websocket handshake failure @webkit-windows-latest
❌ [android-page] › page/page-evaluate.spec.ts:442 › should throw for too deep reference chain 2
❌ [android-page] › page/page-localstorage.spec.ts:99 › storage methods are scoped to the current origin

41 flaky ⚠️ [chromium-library] › library/inspector/cli-codegen-pytest.spec.ts:20 › should print the correct imports and context options `@msedge-beta-ubuntu-22.04`
⚠️ [chromium-library] › library/video.spec.ts:275 › screencast › should capture navigation `@msedge-beta-ubuntu-22.04`
⚠️ [chromium-page] › page/page-event-console.spec.ts:160 › should not throw when there are console messages in detached iframes `@tracing-chromium`
⚠️ [chromium-library] › library/browsercontext-base-url.spec.ts:37 › should construct a new URL when a baseURL in browserType.launchPersistentContext is passed to page.goto `@msedge-dev-windows-latest`
⚠️ [chromium-library] › library/browsertype-launch.spec.ts:22 › should reject all promises when browser is closed `@msedge-dev-windows-latest`
⚠️ [chromium-library] › library/favicon.spec.ts:20 › should load svg favicon with prefer-color-scheme `@msedge-dev-windows-latest`
⚠️ [chromium-library] › library/inspector/cli-codegen-pick-locator.spec.ts:21 › should inspect locator `@msedge-dev-windows-latest`
⚠️ [chromium-library] › library/inspector/cli-codegen-python.spec.ts:98 › should print load/save storage_state `@chrome-ubuntu-22.04`
⚠️ [chromium-library] › library/browsercontext-user-agent.spec.ts:110 › should work for navigator.userAgentData and sec-ch-ua headers `@msedge-macos-latest`
⚠️ [chromium-library] › library/browsertype-connect.spec.ts:189 › launchServer › should ignore page.pause when headed `@chromium-macos-15-large`
⚠️ [chromium-library] › library/video.spec.ts:682 › screencast › should capture full viewport on hidpi `@chromium-macos-15-xlarge`
⚠️ [chromium-library] › library/browsercontext-user-agent.spec.ts:110 › should work for navigator.userAgentData and sec-ch-ua headers `@msedge-beta-macos-latest`
⚠️ [chromium-library] › library/browsertype-connect.spec.ts:189 › launchServer › should ignore page.pause when headed `@msedge-beta-macos-latest`
⚠️ [chromium-library] › library/browsertype-connect.spec.ts:431 › launchServer › should reject waitForEvent before browser.close finishes `@msedge-beta-macos-latest`
⚠️ [chromium-library] › library/browsertype-connect.spec.ts:686 › launchServer › should filter launch options `@msedge-beta-macos-latest`
⚠️ [chromium-library] › library/inspector/cli-codegen-3.spec.ts:224 › cli codegen › should generate frame locators (4) `@chrome-windows-latest`
⚠️ [chromium-library] › library/browsercontext-user-agent.spec.ts:110 › should work for navigator.userAgentData and sec-ch-ua headers `@msedge-dev-macos-latest`
⚠️ [chromium-library] › library/browsertype-connect.spec.ts:141 › launchServer › should be able to reconnect to a browser `@msedge-dev-macos-latest`
⚠️ [chromium-library] › library/browsertype-connect.spec.ts:189 › launchServer › should ignore page.pause when headed `@msedge-dev-macos-latest`
⚠️ [chromium-library] › library/browsertype-connect.spec.ts:216 › launchServer › should be able to connect two browsers at the same time `@msedge-dev-macos-latest`
⚠️ [chromium-library] › library/browsertype-connect.spec.ts:374 › launchServer › should reject navigation when browser closes `@msedge-dev-macos-latest`
⚠️ [chromium-library] › library/browsertype-connect.spec.ts:632 › launchServer › should properly disconnect when connection closes from the client side `@msedge-dev-macos-latest`
⚠️ [chromium-library] › library/chromium/connect-over-cdp.spec.ts:696 › should skip default overrides with noDefaults `@msedge-dev-macos-latest`
⚠️ [chromium-library] › library/client-certificates.spec.ts:426 › browser › should pass with matching certificates and when a http proxy is used from env `@msedge-dev-macos-latest`
⚠️ [chromium-library] › library/browsertype-connect.spec.ts:141 › launchServer › should be able to reconnect to a browser `@chrome-beta-macos-latest`
⚠️ [chromium-library] › library/browsertype-connect.spec.ts:189 › launchServer › should ignore page.pause when headed `@chrome-beta-macos-latest`
⚠️ [chromium-library] › library/browsertype-connect.spec.ts:294 › launchServer › disconnected event should be emitted when browser is closed or server is closed `@chrome-beta-macos-latest`
⚠️ [chromium-library] › library/browsertype-connect.spec.ts:374 › launchServer › should reject navigation when browser closes `@chrome-beta-macos-latest`
⚠️ [chromium-library] › library/browsertype-connect.spec.ts:1117 › launchServer only › should be able to reconnect to a browser 12 times without warnings `@chrome-beta-macos-latest`
⚠️ [chromium-library] › library/selector-generator.spec.ts:680 › selector generator › should generate multiple: noText in text `@chrome-beta-macos-latest`
⚠️ [firefox-library] › library/defaultbrowsercontext-2.spec.ts:147 › should have default URL when launching browser `@firefox-macos-15-large`
⚠️ [firefox-library] › library/inspector/cli-codegen-3.spec.ts:224 › cli codegen › should generate frame locators (4) `@firefox-macos-15-large`
⚠️ [firefox-library] › library/browsercontext-viewport.spec.ts:29 › should set the proper viewport size `@firefox-macos-15-xlarge`
⚠️ [firefox-library] › library/har-websocket.spec.ts:170 › should record websocket connection failure `@firefox-macos-15-xlarge`
⚠️ [firefox-library] › library/screenshot.spec.ts:215 › page screenshot › should not hang when event loop is blocked `@firefox-macos-15-xlarge`
⚠️ [webkit-library] › library/trace-viewer.spec.ts:115 › should show tracing.group in the action list with location `@webkit-windows-latest`
⚠️ [webkit-page] › page/page-set-input-files.spec.ts:38 › should upload a folder `@webkit-macos-15-large`
⚠️ [android-page] › page/page-localstorage.spec.ts:49 › localStorage.removeItem removes a single item
⚠️ [android-page] › page/page-request-continue.spec.ts:162 › should override method along with url
⚠️ [android-page] › page/page-request-fallback.spec.ts:123 › should amend HTTP headers
⚠️ [android-page] › page/page-route.spec.ts:317 › should not throw if request was cancelled by the page

135220 passed, 5324 skipped, 5 did not run

Merge workflow run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yury-s yury-s yury-s approved these changes

Assignees

No one assigned

Labels

CQ1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dgozman @yury-s