feat(ui-automation): Add rs/1 runtime automation parity

CHANGELOG.md

@@ -5,6 +5,58 @@
 ### Added
 
 - Added `nextSteps` hint lines to MCP `structuredContent` and CLI `--output json` envelopes so agents can consume follow-up actions without scraping text. CLI JSON renders shell command lines; MCP structured content renders MCP tool-call hints. Structured result schemas that include `nextSteps` now use schema version 2; existing version 1 schema files remain available for current validators.
+- Added `snapshot_ui sinceScreenHash` / CLI `--since-screen-hash` so callers can skip full runtime snapshot output when the screen hash is unchanged.
+- Added `batch` for executing multiple AXe UI automation steps in one simulator session.
+- Added `wait_for_ui` for polling runtime UI snapshots until UI predicates such as existence, enabled state, focus, text, or settled layout are satisfied. `textContains` can also wait on visible text without a selector when the match is unique.
+- Added structured element-ref `batch` tap steps, preserved same-screen refs after successful `tap` and `batch` actions, and improved UI automation guidance and next steps for one-observation interactions.
+- Added a `replaceExisting` option to `type_text` so agents can replace an existing text-field value instead of accidentally appending to it.
+
+### Fixed
+
+- Fixed compact runtime snapshots so top-level app and window refs are not advertised as swipe targets just because a generic descendant overflows their frame.
+- Fixed `wait_for_ui` focus waits so elements that do not expose focus state return a typed recoverable error instead of timing out.
+- Fixed invalid `touch` calls so structured output no longer reports a fake touch event when neither `down` nor `up` was requested.
+- Fixed compact runtime snapshots so standalone `other` elements, such as keyboard suggestions, are not advertised as swipe targets unless they behave like scrollable containers.
+- Fixed runtime snapshots so off-screen elements, and clipped elements whose activation point is offscreen, are not advertised as actionable targets.
+- Fixed full-screen swipe gestures so app-level scroll refs avoid unsafe screen edges such as the status bar and notch area.
+- Clarified runtime snapshot tips so agents know element refs are snapshot-specific and must come from the latest `snapshot_ui` or `wait_for_ui` output, and only show swipe guidance when the snapshot includes a scroll ref.
+- Made `wait_for_ui` `textContains` matching case-insensitive so assertions survive platform text normalization such as keyboard auto-capitalization, treat duplicate exact text matches as successful presence assertions, narrow broad selectors by text before reporting ambiguity, reject `text` on non-`textContains` predicates instead of silently ignoring it, and keep recoverable-error candidates compact in structured output.
+- Fixed `tap` on SwiftUI switch element refs by using a touch down/up activation instead of AXe's coordinate tap path.
+- Fixed selector fallback for AXe duplicate-match diagnostics that include parenthesized match counts.
+- Fixed semantic taps and text-field focusing so element refs with duplicate AXe selectors use their resolved snapshot coordinates immediately.
+- Fixed bottom-clipped UI automation targets so taps, touches, and long presses use a visible activation point instead of the hidden center of the accessibility frame.
+- Fixed app-level horizontal swipes so full-screen refs use a content-area y-coordinate instead of missing horizontal carousels by swiping near the hero area.
+- Fixed CLI commands with `simulatorId`-only contracts so `simulatorName` session defaults are resolved to a simulator ID without adding conflicting simulator arguments to tools that already accept `simulatorName`, and fixed simulator lifecycle tools so name-only defaults resolve before simctl operations.
+- Fixed `snapshot_ui` and `wait_for_ui` next steps so they use the resolved simulator ID instead of leaking `SIMULATOR_UUID` placeholders.
+- Fixed the Weather example app so saved-location rows are not reused as search-result rows after editing locations.
+- Fixed the Weather example app's current-location button so it selects the current saved location instead of appearing as a no-op UI automation target.
+- Fixed `type_text` so AXe-unsupported international/accented characters fail before focusing the field, with a clear recoverable error instead of a generic typing failure.
+- Fixed `snapshot_ui` next-step guidance so the suggested tap ref prefers useful tappable controls over text fields, sheet grabbers, close buttons, and clear-search buttons.
+- Fixed compact runtime snapshot JSON so target ordering matches compact text output and prioritizes useful content targets before low-value sheet chrome.
+- Fixed `wait_for_ui` success output so compact text and JSON include the matched elements that satisfied the wait predicate.
+- Fixed `wait_for_ui textContains` so duplicate elements with the same matching visible text satisfy presence-style assertions instead of reporting ambiguity.
+- Fixed CLI `--style minimal` so final text output suppresses generated next steps for daemon-routed tools as intended.
+- Fixed `snapshot_ui` next-step guidance so snapshots with no tappable targets no longer suggest tapping the first non-actionable element.
+- Fixed next-step rendering for tools shared across workflows so follow-up commands prefer the workflow that produced the result instead of drifting to another workflow alias.
+- Fixed `snapshot_ui` next-step guidance so calculator-style utility and operator buttons no longer outrank more useful digit/content controls.
+- Fixed `snapshot_ui` compact text, JSON, and next-step guidance so already-selected segmented controls no longer outrank unselected choices.
+- Fixed compact runtime snapshots and next-step guidance so sheet grabbers remain visible as low-priority targets, allowing agents to expand or dismiss sheets without outranking useful content controls.
+- Fixed compact wait-match rows so static assertion matches render with `none` instead of exposing low-level long-press/touch actions as if they were primary agent actions.
+- Fixed compact runtime snapshot ordering and next-step guidance so destructive controls such as Remove/Delete are demoted behind safer content and navigation targets.
+- Clarified simulator keyboard shortcut failures when Simulator.app is running without a visible device window.
+- Fixed hardware button automation so successful button presses wait briefly for system UI transitions before returning, reducing stale immediate follow-up snapshots.
+- Fixed runtime snapshots so modal sheet hosts remain swipeable after the currently visible sheet content fits inside the viewport.
+- Fixed `wait_for_ui` validation so unknown JSON fields are rejected instead of silently broadening waits.
+- Fixed CLI numeric array flags so comma-separated values such as `--key-codes 23,18,14` are parsed as numbers instead of failing validation.
+- Fixed runtime snapshots so unlabeled internal custom-action nodes, such as SpringBoard icon subviews, are no longer advertised as likely tap targets.
+- Fixed AXe bundling so downloaded artifacts must report the pinned AXe version, and dirty local AXe builds require an explicit opt-in.
+- Fixed runtime snapshot tips so compact output names all target-ref action tools, including `long_press` and `touch`.
+- Clarified key press and key sequence tool descriptions so agents know key codes are AXe/macOS virtual key codes and should prefer `type_text` for text entry.
+- Clarified `wait_for_ui` timeout recovery hints so agents know selector fields match exact values and should use `textContains` for partial visible text.
+- Fixed UI action success next steps so agents are prompted to refresh runtime snapshots before reusing element refs after actions such as swipes.
+- Fixed `snapshot_ui` next-step guidance so state-changing controls such as segmented units and switches remain available in targets without being promoted as generic tap or batch suggestions.
+- Fixed `snapshot_ui` tap next-step priority so content-rich cards are suggested before navigation controls like Settings.
+- Fixed successful UI action results so they include a fresh runtime snapshot and actionable next steps, reducing follow-up refresh calls after taps, typing, swipes, and batches.
 
 ## [2.5.2]
 
@@ -642,4 +694,3 @@ Please note that the UI automation features are an early preview and currently i
 - Initial release of XcodeBuildMCP
 - Basic support for building iOS and macOS applications
 
-

example_projects/.xcodebuildmcp/config.yaml

@@ -4,13 +4,11 @@ sessionDefaultsProfiles:
     workspacePath: ./iOS_Calculator/CalculatorApp.xcworkspace
     scheme: CalculatorApp
     simulatorName: iPhone 17 Pro
-    simulatorId: B38FE93D-578B-454B-BE9A-C6FA0CE5F096
     simulatorPlatform: iOS Simulator
   ios-test:
     projectPath: ./iOS/MCPTest.xcodeproj
     scheme: MCPTest
     simulatorName: iPhone 17 Pro
-    simulatorId: B38FE93D-578B-454B-BE9A-C6FA0CE5F096
     simulatorPlatform: iOS Simulator
   macos-test:
     projectPath: ./macOS/MCPTest.xcodeproj

example_projects/Weather/.xcodebuildmcp/config.yaml

@@ -7,7 +7,7 @@ sentryDisabled: false
 sessionDefaults:
   projectPath: Weather.xcodeproj
   scheme: Weather
-  simulatorName: iPhone 17 Pro
+  simulatorName: iPhone 17 Pro Max
 setupPreferences:
   platforms:
     - iOS

example_projects/Weather/README.md

@@ -2,22 +2,14 @@
 
 Atmos Weather is a native SwiftUI weather app prototype for iOS.
 
-## Launch with mock weather data
+## Launch
 
-Build and run the app with XcodeBuildMCP first:
+Build and run the app with XcodeBuildMCP:
 
 ```bash
 ../../build/cli.js simulator build-and-run
 ```
 
-Then relaunch the installed app with the mock API argument:
-
-```bash
-../../build/cli.js simulator launch-app \
-  --bundle-id com.sentry.weather.Weather \
-  --args=--mock-weather-api
-```
-
 ## JSON fixtures
 
 Fixture JSON files live in:
@@ -98,4 +90,4 @@ Run the app test suite through XcodeBuildMCP:
 ../../build/cli.js simulator test
 ```
 
-UI tests inject `--mock-weather-api` themselves so they do not depend on the production API endpoint.
+The app uses bundled deterministic weather data so UI tests do not depend on the production API endpoint.

example_projects/Weather/Weather/Services/MockWeatherAPIClient.swift

@@ -29,8 +29,10 @@ struct MockWeatherAPIClient: WeatherAPIClient, Sendable {
         guard !trimmed.isEmpty else { return [] }
 
         let needle = trimmed.localizedLowercase
-        return fixtures.searchPool.filter { location in
-            location.name.localizedLowercase.contains(needle)
+        var seenLocationIDs = Set<WeatherLocation.ID>()
+        return (fixtures.locations + fixtures.searchPool).filter { location in
+            guard seenLocationIDs.insert(location.id).inserted else { return false }
+            return location.name.localizedLowercase.contains(needle)
                 || location.subtitle.localizedLowercase.contains(needle)
                 || (location.country?.localizedLowercase.contains(needle) ?? false)
         }

example_projects/Weather/Weather/Views/Overlays/LocationPickerView.swift

@@ -103,7 +103,7 @@ struct LocationPickerView: View {
     }
 
     private var currentLocationButton: some View {
-        Button(action: {}) {
+        Button(action: selectCurrentLocation) {
             HStack(spacing: 12) {
                 Image(systemName: "location.fill")
                     .font(.system(size: 14))
@@ -145,6 +145,7 @@ struct LocationPickerView: View {
                             onSelect: { select(location) },
                             onRemove: { remove(location) }
                         )
+                        .id("saved-\(location.id)-\(isEditing)")
                     }
                 } else if isLoading {
                     ForEach(0..<3, id: \.self) { _ in SearchSkeletonRow() }
@@ -160,6 +161,7 @@ struct LocationPickerView: View {
                             onPreview: { preview(location) },
                             onAdd: { add(location) }
                         )
+                        .id("search-\(location.id)-\(isSaved(location))-\(justAddedID == location.id)")
                     }
                 }
             }
@@ -229,6 +231,11 @@ struct LocationPickerView: View {
         justAddedID = location.id
     }
 
+    private func selectCurrentLocation() {
+        guard let currentLocation = savedLocations.first else { return }
+        select(currentLocation)
+    }
+
     private func clearAddedIndicator() async {
         guard let id = justAddedID else { return }
         try? await Task.sleep(for: .milliseconds(1_400))

example_projects/Weather/Weather/Views/Overlays/LocationRows.swift

@@ -96,6 +96,7 @@ struct SearchLocationRow: View {
                 .frame(maxWidth: .infinity, alignment: .leading)
             }
             .buttonStyle(.plain)
+            .accessibilityValue(saved || added ? "saved" : "not saved")
 
             VStack(alignment: .trailing, spacing: 3) {
                 Text(WeatherUnitFormatter.temperatureString(location.temperatureC, units: units))

example_projects/Weather/Weather/Views/Overlays/SettingsSheetView.swift

@@ -132,6 +132,7 @@ private struct SegmentRow<Option: Identifiable & Hashable>: View {
                     Button(optionLabel(option)) {
                         selection = option
                     }
+                    .accessibilityValue(selection == option ? "selected" : "not selected")
                     .font(.system(size: 13, weight: .medium))
                     .foregroundStyle(selection == option ? .black : .white)
                     .padding(.horizontal, 14)

example_projects/Weather/Weather/WeatherApp.swift

@@ -10,15 +10,7 @@ import SwiftUI
 
 @main
 struct WeatherApp: App {
-    private let weatherService: WeatherService
-
-    init() {
-        if CommandLine.arguments.contains("--mock-weather-api") {
-            weatherService = .mock
-        } else {
-            weatherService = .production
-        }
-    }
+    private let weatherService: WeatherService = .mock
 
     var body: some Scene {
         WindowGroup {

example_projects/Weather/WeatherTests/WeatherTests.swift

@@ -47,6 +47,9 @@ struct WeatherTests {
 
         let byCountry = try await service.searchLocations(matching: "gb")
         #expect(byCountry.map(\.name).contains("London"))
+
+        let savedLocationByName = try await service.searchLocations(matching: "tokyo")
+        #expect(savedLocationByName.contains { $0.name == "Tokyo" })
     }
 
     @Test func emptySearchReturnsNoResults() async throws {

example_projects/Weather/WeatherUITests/WeatherUITests.swift

@@ -69,7 +69,6 @@ final class WeatherUITests: XCTestCase {
     @MainActor
     private func launchApp() -> XCUIApplication {
         let app = XCUIApplication()
-        app.launchArguments.append("--mock-weather-api")
         app.launch()
         return app
     }

example_projects/Weather/WeatherUITests/WeatherUITestsLaunchTests.swift

@@ -20,7 +20,6 @@ final class WeatherUITestsLaunchTests: XCTestCase {
     @MainActor
     func testLaunch() throws {
         let app = XCUIApplication()
-        app.launchArguments.append("--mock-weather-api")
         app.launch()
 
         // Insert steps here to perform after app launch but before taking a screenshot,

example_projects/iOS/.xcodebuildmcp/config.yaml

@@ -3,7 +3,7 @@ enabledWorkflows: ['simulator', 'ui-automation', 'debugging', 'logging']
 sessionDefaults:
   projectPath: ./MCPTest.xcodeproj
   scheme: MCPTest
-  simulatorId: B38FE93D-578B-454B-BE9A-C6FA0CE5F096
+  simulatorName: iPhone 17 Pro
   useLatestOS: true
   platform: iOS Simulator
   bundleId: io.sentry.MCPTest

example_projects/iOS_Calculator/.xcodebuildmcp/config.yaml

@@ -11,12 +11,11 @@ sessionDefaults:
   workspacePath: CalculatorApp.xcworkspace
   scheme: CalculatorApp
   configuration: Debug
-  simulatorId: A2C64636-37E9-4B68-B872-E7F0A82A5670
   simulatorPlatform: iOS Simulator
   useLatestOS: true
   arch: arm64
   suppressWarnings: false
-  derivedDataPath: ./iOS_Calculator/.derivedData
+  derivedDataPath: ./.build/DerivedData
   preferXcodebuild: true
   bundleId: io.sentry.calculatorapp
   simulatorName: iPhone 17 Pro

manifests/tools/batch.yaml

@@ -0,0 +1,16 @@
+id: batch
+module: mcp/tools/ui-automation/batch
+names:
+  mcp: batch
+  cli: batch
+description: Execute structured rs/1 tap steps in one simulator session. Use refs from the latest snapshot_ui or wait_for_ui output, for example {"steps":[{"action":"tap","elementRef":"e1"},{"action":"tap","elementRef":"e2"}]}. Do not pass raw AXe strings such as "tap e7". Omit preDelay/postDelay for switch elementRefs; switches execute as touch down/up steps and reject delays.
+outputSchema:
+  schema: xcodebuildmcp.output.ui-action-result
+  version: '2'
+routing:
+  stateful: true
+annotations:
+  title: Batch UI Actions
+  readOnlyHint: true
+  destructiveHint: false
+  openWorldHint: false

manifests/tools/key_press.yaml

@@ -3,10 +3,12 @@ module: mcp/tools/ui-automation/key_press
 names:
   mcp: key_press
   cli: key-press
-description: Press key by keycode.
+description: Press one hardware key using an AXe HID key code. Prefer type_text for text entry. Common values include 40 Return/Enter, 42 Backspace, 43 Tab, and 44 Space.
 outputSchema:
   schema: xcodebuildmcp.output.ui-action-result
-  version: "2"
+  version: '2'
+routing:
+  stateful: true
 annotations:
   title: Key Press
   readOnlyHint: true

manifests/tools/key_sequence.yaml

@@ -3,10 +3,12 @@ module: mcp/tools/ui-automation/key_sequence
 names:
   mcp: key_sequence
   cli: key-sequence
-description: Press a sequence of keys by their keycodes.
+description: Press hardware keys using AXe HID key codes. Prefer type_text for text entry. Common values include 40 Return/Enter, 42 Backspace, 43 Tab, and 44 Space.
 outputSchema:
   schema: xcodebuildmcp.output.ui-action-result
-  version: "2"
+  version: '2'
+routing:
+  stateful: true
 annotations:
   title: Key Sequence
   readOnlyHint: true

manifests/tools/long_press.yaml

@@ -3,10 +3,12 @@ module: mcp/tools/ui-automation/long_press
 names:
   mcp: long_press
   cli: long-press
-description: Long press at coords.
+description: Long press a UI element by elementRef from a current rs/1 runtime snapshot.
 outputSchema:
   schema: xcodebuildmcp.output.ui-action-result
   version: "2"
+routing:
+  stateful: true
 annotations:
   title: Long Press
   readOnlyHint: true

manifests/tools/snapshot_ui.yaml

@@ -3,28 +3,12 @@ module: mcp/tools/ui-automation/snapshot_ui
 names:
   mcp: snapshot_ui
   cli: snapshot-ui
-description: Print view hierarchy with precise view coordinates (x, y, width, height) for visible elements.
+description: Capture a semantic rs/1 runtime UI snapshot with elementRef targets. Observe once, use tap for one target or batch for multiple same-screen targets, and refresh after navigation, scrolling, sheet changes, or obvious layout changes.
 outputSchema:
   schema: xcodebuildmcp.output.capture-result
-  version: "2"
-nextSteps:
-  - label: Refresh after layout changes
-    toolId: snapshot_ui
-    params:
-      simulatorId: SIMULATOR_UUID
-    when: success
-  - label: Tap on element
-    toolId: tap
-    params:
-      simulatorId: SIMULATOR_UUID
-      x: 0
-      y: 0
-    when: success
-  - label: Take screenshot for verification
-    toolId: screenshot
-    params:
-      simulatorId: SIMULATOR_UUID
-    when: success
+  version: '2'
+routing:
+  stateful: true
 annotations:
   title: Snapshot UI
   readOnlyHint: true

manifests/tools/swipe.yaml

@@ -3,10 +3,12 @@ module: mcp/tools/ui-automation/swipe
 names:
   mcp: swipe
   cli: swipe
-description: Swipe between points.
+description: Swipe within a scrollable UI element using a visible element reference from the current UI.
 outputSchema:
   schema: xcodebuildmcp.output.ui-action-result
   version: "2"
+routing:
+  stateful: true
 annotations:
   title: Swipe
   readOnlyHint: true

manifests/tools/tap.yaml

@@ -3,10 +3,12 @@ module: mcp/tools/ui-automation/tap
 names:
   mcp: tap
   cli: tap
-description: Tap UI element by accessibility id/label (recommended) or coordinates as fallback.
+description: Tap one elementRef from the latest snapshot_ui or wait_for_ui output. Other same-screen refs may remain usable after success; refresh after navigation, scrolling, sheet changes, or obvious layout changes.
 outputSchema:
   schema: xcodebuildmcp.output.ui-action-result
-  version: "2"
+  version: '2'
+routing:
+  stateful: true
 annotations:
   title: Tap
   readOnlyHint: true

manifests/tools/touch.yaml

@@ -3,10 +3,12 @@ module: mcp/tools/ui-automation/touch
 names:
   mcp: touch
   cli: touch
-description: Touch down/up at coords.
+description: Send touch down/up events to a UI element by elementRef from a current rs/1 runtime snapshot.
 outputSchema:
   schema: xcodebuildmcp.output.ui-action-result
   version: "2"
+routing:
+  stateful: true
 annotations:
   title: Touch
   readOnlyHint: true