Audio support: mic capture and playback in the browser channel #28

New issue

Open

opened 2026-08-22 08:42:41 +01:00 by pedromcaraujo · 0 comments

pedromcaraujo commented

2026-08-22 08:42:41 +01:00

Owner

Problem

Forge channels are ttyd terminals in the browser, and there is no audio path
in either direction:

Capture — a mic attached to the client machine isn't reachable from the
channel, so anything recording-shaped (sox/rec, arecord, a transcribe
helper) has no input device on nix-dev.
Playback — audio produced inside a channel has nowhere to go; there's no
sink wired up on nix-dev and nothing carrying it back to the browser tab.

Concretely this blocks driving CLI audio tooling from a Forge channel — the
usual workaround is to fall back to the laptop, which defeats the point of the
browser workstation.

What "supported" would look like

Browser grants mic permission on the channel page (getUserMedia), audio is
streamed to the engine and exposed as a capture device inside the channel's
session.
Playback goes the other way: a sink inside the session, streamed back to the
tab.
Both are opt-in per channel, off by default — most channels don't want a mic
permission prompt.

Open questions

Transport: extend the existing ttyd/sidecar path, or a separate WebSocket
next to it? ttyd itself has no audio concept, so this is likely a sidecar
concern.
Device plumbing on nix-dev: PipeWire null sink + virtual source, or something
lighter like piping raw PCM into the session and skipping a real device node?
Whether a real device node is needed at all, or whether "pipe raw PCM at a
process" covers the actual use cases.
Multiple concurrent channels sharing one host — device naming and isolation.

No urgency implied; filing it so the gap is recorded rather than rediscovered.

## Problem Forge channels are ttyd terminals in the browser, and there is no audio path in either direction: - **Capture** — a mic attached to the client machine isn't reachable from the channel, so anything recording-shaped (`sox`/`rec`, `arecord`, a transcribe helper) has no input device on nix-dev. - **Playback** — audio produced inside a channel has nowhere to go; there's no sink wired up on nix-dev and nothing carrying it back to the browser tab. Concretely this blocks driving CLI audio tooling from a Forge channel — the usual workaround is to fall back to the laptop, which defeats the point of the browser workstation. ## What "supported" would look like - Browser grants mic permission on the channel page (`getUserMedia`), audio is streamed to the engine and exposed as a capture device inside the channel's session. - Playback goes the other way: a sink inside the session, streamed back to the tab. - Both are opt-in per channel, off by default — most channels don't want a mic permission prompt. ## Open questions - Transport: extend the existing ttyd/sidecar path, or a separate WebSocket next to it? ttyd itself has no audio concept, so this is likely a sidecar concern. - Device plumbing on nix-dev: PipeWire null sink + virtual source, or something lighter like piping raw PCM into the session and skipping a real device node? - Whether a real device node is needed at all, or whether "pipe raw PCM at a process" covers the actual use cases. - Multiple concurrent channels sharing one host — device naming and isolation. No urgency implied; filing it so the gap is recorded rather than rediscovered.

pedromcaraujo added the

enhancement

label

2026-08-28 10:10:12 +01:00