Skip to content

fix(computer-vision): align vision_camera segmentation overlay with preview#1170

Open
msluszniak wants to merge 2 commits into
mainfrom
@ms/fix-segmentation-overlay-alignment
Open

fix(computer-vision): align vision_camera segmentation overlay with preview#1170
msluszniak wants to merge 2 commits into
mainfrom
@ms/fix-segmentation-overlay-alignment

Conversation

@msluszniak
Copy link
Copy Markdown
Member

Description

In the computer-vision demo's vision_camera screen the semantic-segmentation overlay landed rotated and offset relative to the camera preview (FCN ResNet50 / any DeepLab variant). Two bugs in SegmentationTask.tsx:

  • screenW/screenH were derived from an unconditional frame.height/frame.width swap, which is wrong outside portrait.
  • The argmax.length === screenW * screenH branch was unreachable (runOnFrame is called with resizeToInput=false, so the mask is always at model output resolution) and we always fell into the sqrt(length) square fallback — which is fine for current square models but the subsequent <SkiaImage fit="cover"> over the whole canvas stretched it incorrectly onto the portrait canvas.

This PR:

  • Derives screen-space W/H from frame.orientation ('left'/'right' swap, 'up'/'down' keep). The native side already rotates the mask into screen space via inverseRotateMat, so we just need the matching screen-space dims.
  • Drops the unreachable screenW*screenH branch and uses sqrt(length) unconditionally with a comment noting that non-square model outputs would need dims plumbed from native.
  • Draws the <SkiaImage> into the camera preview's cover-fit rect (Math.max scale + centering, same pattern as OCRTask) with fit="fill", so the mask aligns pixel-for-pixel with the preview.

Introduces a breaking change?

  • Yes
  • No

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Other (chores, tests, code style improvements etc.)

Tested on

  • iOS
  • Android

Testing instructions

  1. Open the computer-vision demo → vision_camera screen.
  2. Select Segment → FCN ResNet50 (or any DeepLab variant).
  3. Aim the camera at a clear asymmetric subject (e.g. a TV).
  4. The colored overlay should now line up with the actual subject in the preview, not rotated/displaced.

Screenshots

Related issues

Fixes #1158.

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

iOS not retested with this patch — the orientation-aware swap is symmetric across platforms and the C++ rotation already handled iOS correctly, but a quick sanity check on a portrait iPhone before merge wouldn't hurt.

…review

Replace the unconditional sensor-native W/H swap with an orientation-aware
one, drop the unreachable `argmax.length === screenW * screenH` branch
(runOnFrame is called with resizeToInput=false, so the mask is always at
model output resolution), and draw the SkiaImage into the camera preview's
cover-fit rect with fit="fill" instead of stretching it onto the whole
portrait canvas. Fixes #1158.
@msluszniak msluszniak marked this pull request as draft May 21, 2026 16:02
@msluszniak msluszniak self-assigned this May 21, 2026
@msluszniak msluszniak added bug fix PRs that are fixing bugs demo app labels May 21, 2026
@msluszniak

This comment was marked as resolved.

Pin vision_camera's `orientationSource` to `'interface'` while the
segmentation task is active so the preview stays in the activity's
(portrait-locked) coord system — the same system the native side rotates
the mask into. Other tasks keep `'device'` since their coords (bboxes,
points) tolerate the device-rotated preview.

Also drop the unreachable `argmax.length === screenW * screenH` branch
(`runOnFrame` is called with `resizeToInput=false`, so the mask is at
model output resolution) and draw the SkiaImage into the camera
preview's cover-fit rect with `fit="fill"` instead of stretching it
across the whole portrait canvas. Refs #1158.
@msluszniak msluszniak marked this pull request as ready for review May 21, 2026 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug fix PRs that are fixing bugs demo app

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vision_camera demo: semantic segmentation overlay misaligned in portrait

1 participant