feat(linux/wlr): implement SHM capture fallback for headless setups#4946
Open
atassis wants to merge 2 commits intoLizardByte:masterfrom
Open
feat(linux/wlr): implement SHM capture fallback for headless setups#4946atassis wants to merge 2 commits intoLizardByte:masterfrom
atassis wants to merge 2 commits intoLizardByte:masterfrom
Conversation
When DMA-BUF capture fails (e.g. GBM cannot allocate buffers on headless NVIDIA without an active DRM output), fall back to SHM shared memory capture via wl_shm. The SHM path creates a memfd-backed wl_shm_pool, receives pixel data from the compositor via wlr-screencopy, and feeds it to the encoder through the existing wlr_ram_t CPU path. Supported SHM formats: - 4 bpp (XRGB8888/ARGB8888): direct memcpy - 3 bpp (BGR888): pixel conversion to BGRA8888 Key changes: - Bind wl_shm interface in screencopy path - Add create_and_copy_shm() with memfd + mmap allocation - Refactor create_and_copy_dmabuf() to return bool for fallback - Cache GBM failure to avoid per-frame retry - Handle SHM frames in wlr_ram_t::snapshot() via memcpy - Make EGL init non-fatal (SHM path does not require EGL) - Force wlr_ram_t on reinit when SHM fallback is active Tested on headless NVIDIA RTX 5060 Ti with labwc compositor, NVENC HEVC encoding, streaming to Moonlight client.
atassis
force-pushed
the
feat/shm-capture-fallback
branch
from
0262c5d to
2bcea4f
Compare
BGR888 (0x34324742) stores bytes in R,G,B memory order. Previous code copied them straight into BGRA8888, causing red/blue inversion (orange appeared bluish in Moonlight). Swap src positions 0↔2 so B and R land in correct BGRA slots. Also deduplicate SHM format log to fire only once.
|
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Implements SHM (shared memory) capture as a fallback when DMA-BUF screencopy fails.
Currently,
wlr-screencopyonly supports DMA-BUF capture via GBM buffer allocation. On headless NVIDIA setups (no physical display connected), GBM allocation fails because the NVIDIA driver cannot create GBM buffers without an active DRM output. This makes Wayland screencopy unusable on headless NVIDIA — the existing code path logs"SHM capture not implemented"and gives up.This PR implements the SHM fallback that was stubbed out in
buffer_done():wl_shminterface from the Wayland registrymemfd-backedwl_shm_pooland requests the compositor to copy frames into itwlr_ram_tCPU pathwlr_ram_tcan operate in SHM-only modeThe DMA-BUF path is completely unchanged — SHM only activates when GBM fails or the compositor doesn't advertise
zwp_linux_dmabuf_v1.Screenshot
Issues Fixed or Closed
Unknown Monitor connector type [Meta]#2250 (headless NVIDIA capture)Roadmap Issues
Type of Change
Checklist
AI Usage