How LinuxPlay Remote Desktop Works

Scope & Security: Hosting is Linux-only; the client runs on Linux and Windows. For WAN, put the host behind a WireGuard tunnel and connect to the tunnel IP. No built-in password prompts are used—assume your LAN is trusted or use WireGuard.

1) Capture & Monitors

On Linux, the host auto-selects kmsgrab when possible (great for Wayland/KMS; cursor isn’t drawn) or falls back to x11grab. The handshake shares all monitors as WxH+X+Y so the client can open one window per monitor or a specific index.

# Example (x11grab fallback)
ffmpeg -f x11grab -draw_mouse 0 -framerate 60 -video_size 1920x1080 -i :0.0+0,0 …

2) Encoding

Pick h.264 (lowest latency) or h.265 (higher efficiency). Backends include nvenc, qsv, vaapi, or CPU. Preset, GOP, QP, tune, and pixel format are configurable. A comment marker is embedded so player logs can attribute streams to LinuxPlay.

3) Negotiation (TCP 7001)

Client connects and sends HELLO. Host replies OK:<codec>:<monitor-list> and remembers the client IP for unicast.

# Client → Host
HELLO

# Host → Client
OK:h.264:1920x1080+0+0;1920x1080+1920+0

4) Transport

# Host (conceptual): send video for monitor 0 to the client
ffmpeg … -f mpegts udp://<client_ip>:5000
# Client listens locally and decodes
av.open("udp://@0.0.0.0:5000", format="mpegts", options={"fflags":"nobuffer","flags":"low_delay"})

5) Decoding & Rendering

The client uses PyAV (FFmpeg) and auto-chooses vaapi/cuda/qsv/d3d11va when available. Frames are uploaded to an OpenGL widget for tear-free, vsync-off presentation. An optional Ultra mode further cuts buffering for LAN.

6) Input, Clipboard & Files

• Input: Client emits MOUSE_* / KEY_* UDP messages; host injects via pynput (preferred) or xdotool fallback.
• Controller: Client forwards gamepad via UDP 7005 (auto-detects common pads like Xbox, DualSense, 8BitDo). Host receives and injects through a uinput virtual device; no system-wide remapper required.
• Clipboard: Host polls pyperclip; client uses Qt clipboard. Both sides exchange CLIPBOARD_UPDATE messages.
• Files: Drag any file/folder onto the client window to upload to the host’s ~/LinuxPlayDrop via TCP.

7) Network Adaptation

• LAN vs Wi-Fi: Client detects link type and announces it to the host; both sides tweak UDP buffers/delay. Ultra is auto-disabled off-LAN.
• Adaptive bitrate (experimental): Host can toggle to a lower bitrate under load and restart streams seamlessly.
• Health-check: Host pings; client PONGs. Host restarts streams for the current client if needed.

Summary

• Host captures with kmsgrab or x11grab, encodes via NVENC/QSV/VAAPI/CPU.
• TCP handshake shares encoder + monitor layout; streaming is unicast UDP per monitor.
• Client decodes with hardware accel and renders through a PyQt5 + OpenGL path.
• Inputs, controller, clipboard, file upload, heartbeat all run over lightweight UDP/TCP side-channels.
• For WAN, use WireGuard; for lowest latency, prefer h.264 + LAN + Ultra.