fix(computer-vision): align vision_camera segmentation overlay with preview#1170
Open
msluszniak wants to merge 2 commits into
Open
fix(computer-vision): align vision_camera segmentation overlay with preview#1170msluszniak wants to merge 2 commits into
msluszniak wants to merge 2 commits into
Conversation
…review Replace the unconditional sensor-native W/H swap with an orientation-aware one, drop the unreachable `argmax.length === screenW * screenH` branch (runOnFrame is called with resizeToInput=false, so the mask is always at model output resolution), and draw the SkiaImage into the camera preview's cover-fit rect with fit="fill" instead of stretching it onto the whole portrait canvas. Fixes #1158.
This comment was marked as resolved.
This comment was marked as resolved.
Pin vision_camera's `orientationSource` to `'interface'` while the segmentation task is active so the preview stays in the activity's (portrait-locked) coord system — the same system the native side rotates the mask into. Other tasks keep `'device'` since their coords (bboxes, points) tolerate the device-rotated preview. Also drop the unreachable `argmax.length === screenW * screenH` branch (`runOnFrame` is called with `resizeToInput=false`, so the mask is at model output resolution) and draw the SkiaImage into the camera preview's cover-fit rect with `fit="fill"` instead of stretching it across the whole portrait canvas. Refs #1158.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
In the
computer-visiondemo's vision_camera screen the semantic-segmentation overlay landed rotated and offset relative to the camera preview (FCN ResNet50 / any DeepLab variant). Two bugs inSegmentationTask.tsx:screenW/screenHwere derived from an unconditionalframe.height/frame.widthswap, which is wrong outside portrait.argmax.length === screenW * screenHbranch was unreachable (runOnFrameis called withresizeToInput=false, so the mask is always at model output resolution) and we always fell into thesqrt(length)square fallback — which is fine for current square models but the subsequent<SkiaImage fit="cover">over the whole canvas stretched it incorrectly onto the portrait canvas.This PR:
frame.orientation('left'/'right'swap,'up'/'down'keep). The native side already rotates the mask into screen space viainverseRotateMat, so we just need the matching screen-space dims.screenW*screenHbranch and usessqrt(length)unconditionally with a comment noting that non-square model outputs would need dims plumbed from native.<SkiaImage>into the camera preview's cover-fit rect (Math.maxscale + centering, same pattern asOCRTask) withfit="fill", so the mask aligns pixel-for-pixel with the preview.Introduces a breaking change?
Type of change
Tested on
Testing instructions
computer-visiondemo → vision_camera screen.Screenshots
Related issues
Fixes #1158.
Checklist
Additional notes
iOS not retested with this patch — the orientation-aware swap is symmetric across platforms and the C++ rotation already handled iOS correctly, but a quick sanity check on a portrait iPhone before merge wouldn't hurt.