Voice PC Remote

Voice PC Remote lets me control my own Windows PC by voice from a phone: open apps and sites, search, play music on YouTube, change the volume, drive OBS, Office, and Steam, and even ask an AI coding agent to change a project. Speech recognition and the language model run on my HomeLab (PC1), not in the cloud. The PC (PC2) runs an agent that connects out to PC1, checks every request again, and reports what actually happened. I wrote the design document first (requirements, threat model, test cases, phases and gates), then built it phase by phase.

Letting a language model control a real computer is risky: speech is misheard, the model can be wrong or manipulated by text on a web page, and a remote-control endpoint on the internet is a target. The goal was hands-free control that stays private and never runs something it cannot justify.

Personal use. Sign-in is limited to an allow-listed Google account on a device paired with a secret key.

Individual builder — design document, architecture, PC1 orchestrator, PC2 agent, Chrome extension, PWA, evaluation, and deployment.

• Python
• FastAPI
• WebSocket
• faster-whisper
• Ollama
• Gemma (gemma4-e2b)
• React
• TypeScript
• PWA
• SQLite
• Chrome extension
• Docker
• Proxmox LXC + GPU
• Cloudflare Tunnel
• GitHub Actions

1. 1

   Speak on the phone

   The PWA records audio (tap, hold, or hands-free with voice activity detection and the wake word “computer”). PC1 transcribes it with faster-whisper; the phone shows what it heard so I can fix it before pressing Send.

2. 2

   Rules before the model

   Aliases fix common mishearings, and simple commands such as “open Chrome”, “stop”, or “status” match rule shortcuts, so they finish in about a second without the LLM.

3. 3

   Plan with the local LLM

   Anything else goes to Ollama with the tool schemas, where allowed apps and sites are enums. The model returns a tool call, a question back when the request is ambiguous, or a refusal.

4. 4

   Policy decides

   The router denies by default: unknown tools, bad arguments, rate limits, and the kill switch all stop the job. Risky actions (level 3 and up) show a confirmation card with the exact arguments, bound by a hash so they cannot change after approval.

5. 5

   Signed, then checked again on the PC

   PC1 signs the request with HMAC-SHA256 over method, path, timestamp, nonce, and body hash. The agent on PC2 rejects old or replayed requests, re-checks its own allowlist, runs the tool without a shell, and verifies the result.

6. 6

   Answer from the verified result

   The summary on the phone is built from what the PC confirmed, never guessed, and pushed live over WebSocket. Every step is timed and written to hash-chained audit logs on both machines.

• Phase 1 — core pipeline: voice and text commands, rule shortcuts, LLM planner, policy router, signed dispatch, verified results, history, and Stop.
• Phase 2 — browser: web and YouTube search with results on the phone, “open result 2”, playing videos and music, tabs, and stopping at CAPTCHAs instead of solving them.
• Phase 3 — AI coding agent: edits on registered projects only, on ai/* branches, with tests, diffs, and revert on the phone.
• Phase 4 — hands-free: wake word, voice answers, screen description with a vision model, and Wake-on-LAN.
• App flows: the phone draws each app (YouTube, Music, Search, Code, Office, OBS, Steam, social sites) as a short wizard generated by PC1, so tapping and speaking share one pipeline.

• AI and speech

  • Speech-to-text with faster-whisper (CTranslate2, int8 on CPU): vocabulary prompts, forced language, repetition-loop handling, confidence-based “did I hear you right?”.
  • LLM tool calling with Ollama: tool schemas with enums, ask-back for ambiguity, retry on invalid JSON.
  • Choosing a model for a 4 GB GPU by benchmark (accuracy, refusal, latency) and recording the decision in an ADR.
  • Wake-word filtering and client-side voice activity detection for hands-free use.

• Security engineering

  • STRIDE threat model across trust zones; the LLM and screen content are treated as untrusted input (prompt-injection taint).
  • HMAC-SHA256 request signing with timestamps, nonces, and idempotency keys; deny-by-default policy and risk levels.
  • Google OAuth with PKCE plus device pairing, HttpOnly cookies, CSRF header, WebSocket origin checks.
  • Hash-chained audit logs, fail-closed when the audit cannot be written, an offline kill switch, and a Stop that always works.
  • AI code edits held for approval when they add process, network, or secret access (found by my own red-team).

• Backend and architecture

  • FastAPI REST and WebSocket APIs; a job state machine (planning → routing → dispatching → verifying → done, with clarify, confirm, cancel).
  • Ports and adapters: the core has no framework imports, so STT, LLM, and PC2 can be swapped for fakes in tests.
  • Shared contracts package for tool schemas and signing, used by both PC1 and PC2; SQLite in WAL mode; circuit breaker for PC2.

• Windows and browser automation

  • A Windows agent that launches and verifies apps, sends media keys, and drives OBS, Office, and Steam.
  • A Chrome extension that controls the owner's own Chrome (YouTube, search, social sites) and reconnects by itself.
  • AI coding agent on registered git projects: every edit on an ai/* branch, fixed test command, diff and revert from the phone.
  • Wake-on-LAN to wake the PC from the HomeLab.

• Infrastructure and operations

  • Proxmox LXC with GPU passthrough, Ollama and Open WebUI in Docker, the orchestrator as a systemd service.
  • Cloudflare Tunnel so neither PC1 nor PC2 opens a port; the PC agent dials out over WebSocket.
  • CI on GitHub Actions: ruff, pytest with a coverage gate, bandit, pip-audit, npm audit, gitleaks, and the evaluation set.

• Evaluation and testing

  • Golden command set measuring tool accuracy, refusals, clarifications, and latency per stage.
  • Red-team suite (prompt injection, dangerous commands, shells), soak runs, a speech-to-text benchmark with noise, and a wake-word evaluation.
  • End-to-end tests with real HMAC and a real socket relay between PC1 and PC2.

• repositorySource code

• Single-user system tuned for my own PC, apps, and voice.
• Speech-to-text runs on CPU (about 2–4 s per command); a larger GPU would allow bigger Whisper and LLM models.
• The risky-code check is a pattern list and cannot prove code safe; the ai/* branch and revert remain the backstop.

• Evaluation (golden sets, red-team prompts, soak runs) is what turns a voice demo into something safe to leave running.
• Writing the design document with test cases and gates first made each phase easy to finish and verify.
• Real use finds what tests miss: a library update broke every audio command in production until I decoded audio myself.

• 

  Infrastructure

  HomeLab and Local AI

  An old PC rebuilt as a Proxmox server that runs my live projects and a local LLM on its own GPU, published through Cloudflare Tunnel with Grafana and Dozzle for metrics and logs.

  • Proxmox VE
  • LXC
  • Linux
  • Docker Compose
  • Cloudflare Tunnel

  View case study

• 

  Software

  Live

  E-Commerce Platform

  Full-stack store built from scratch and running live: checkout with stock reservation, card (3-D Secure) and PromptPay payments, shipping and tracking, loyalty points, real-time notifications, and an admin CMS — deployed by CI/CD to my HomeLab.

  • Next.js
  • React
  • Node.js
  • Express
  • PostgreSQL

  View case studyOpen live site ↗