Jarvis · Timon

Portfolio case study · 2026

Say it once. It becomes a task.

Jarvis is a desk-bound ESP32-S3 that hears “Hey Jarvis.” Timon is the TDAH-friendly task store it writes to. The device stays dumb — mic, speaker, screen. All thinking runs on Cloudflare Workers, on a $0 neuron budget.

Apollo /health Timon /healthz Run the voice path
37Apollo tools
511Apollo tests green
85 / 100STT strict / normalised
~2.5sAudio → task_id

Architecture

Option A, locked: no firmware change. Apollo owns the conversation. Timon is an HTTP tool.

Device

ESP32 stays a sensor

Waveshare ESP32-S3-Touch-AMOLED-2.06. Wake word, I2S mic/speaker, captions. Shared-secret WSS to apollo. Firmware never learns what Timon is.

Conversation

Apollo owns the turn

Durable Object session, 3 tool rounds max, 8000-byte audio floor. Spanish and English. On save: TTS “Guardado: …” plus play_effect: ding.

Task store

Timon is a tool

POST /api/tasks with Bearer key. Groq JSON intent, D1 hierarchy, SessionDO broadcast. Fail fast, no retries inside the Worker.

Voice command → task
  1. Wake + hold-to-talk. PCM frames over WSS to apollo.
  2. Groq Whisper transcribes. DeepSeek decides timon_create_task.
  3. Apollo POSTs {text, device_id, ts} to Timon (5s timeout).
  4. Timon extractIntent → D1 insert + task_events ledger + SessionDO.
  5. Apollo speaks confirmation. Open Timon clients get task_added.

Tech stack

LayerWhat shipsWhy
Hardware ESP32-S3, I2S mic/speaker, AMOLED, OTA 2.7.1 Always-on desk appliance. Device is deliberately dumb.
Edge Cloudflare Workers, Durable Objects, D1, R2, Vectorize, Queues Session pin, task rows, TTS cache, semantic recall, background jobs.
Voice STT Groq whisper-large-v3-turbo · TTS Gemini (VOICE_PROVIDER=free) Workers AI quota 4006 killed the loop. Free tiers, zero neurons.
LLM Apollo: DeepSeek. Timon intents: Groq qwen/qwen3.8-27b. OpenRouter fallbacks. JSON mode for tasks; conversation model stays on apollo.
Integrations GitHub, Google Calendar, Hue, Resend, Tavily — apollo tool catalog Timon does not call them. Apollo does, then writes tasks.
Secrets Bitwarden pointers, wrangler secrets, never git GROQ_API_KEY, TIMON_API_KEY, TZ=America/Argentina/Buenos_Aires

Development journey

  1. Stage 1 · research

    Do not reinvent the task manager

    TDAH rules locked: full context before starting, parent/child, categories, single-hue UI, reduced motion. Jarvis path = Option A (LLM tool, no firmware change).

  2. Stage 2 · audit

    CONTRACT.md as source of truth

    NID-468 pinned the live worker. Found production intents were 100% heuristic: wrangler had no [ai] binding, so env.AI.run never ran. Docs before more code.

  3. Stage 3 · done

    Real LLM, hierarchy, Jarvis bridge

    Groq JSON intents (NID-469). D1 parent/deps/status (NID-471). POST /api/tasks + apollo timon_create_task (NID-470). Parallel PRs conflicted; NID-484 merged #4/#5/#6, stubbed fetch, fixed D1 mocks.

  4. Stage 4 · in flight

    Context API + CI gate todo

    GET list/PATCH/auth on every /api/*. Stop ungated master deploys (conflict markers and a silent wrangler deploy already bit production).

  5. Stage 5 · this site + UI

    Portfolio site now · tapp UI later backlog

    Minimal Timon UI and STT confirm-before-save still parked. This showcase is the interview surface while those wait.

Trade-off: $0 neurons

Cloudflare Workers AI quota error 4006 took STT and TTS down together. Locked decision: Groq + Gemini over HTTP. Cost stays $0 on that path. Fail fast, no retry loops.

Trade-off: TF never local

Cloudflare changes go PR → plan → merge → apply. Secrets live as Bitwarden pointers. Branches are never deleted. Durable Objects pin old code until the device reboots.

Technical highlights

TDAH-first context

getTaskWithContext returns the task, parent, siblings, subtasks, and blockers in one call — one level deep. Cognitive load and D1 cost, both bounded.

Voice-first, not a chat app

Replies are written to be spoken. Captions on the AMOLED. Abort cuts TTS mid-stream. Spanish transcripts are first-class (owner speaks Spanish to Jarvis).

Realtime without a second stack

SessionDO hibernates WebSockets and broadcasts task_added. Same worker that writes D1. No extra realtime vendor.

Semantic memory

Apollo upserts facts into Vectorize and injects a recall block into the LLM prompt. Nightly consolidation on a cron. Brain VM stays network-isolated behind a tunnel.

Code, as shipped

Excerpts from jfcanon/timon and jfcanon/apollo. Not samples invented for the site.

Fail-fast STT

timon/src/lib/transcribe.js 
formData.append('model', 'whisper-large-v3-turbo');
const response = await fetch(
  'https://api.groq.com/openai/v1/audio/transcriptions',
  { method: 'POST', headers: { Authorization: `Bearer ${env.GROQ_API_KEY}` }, body: formData }
);
if (!response.ok) throw new Error(`Groq STT failed with status ${response.status}`);
// no retries — Talvi idiom

Heuristic fallback, always a task

timon/src/lib/intents.js 
const GROQ_MODEL = 'qwen/qwen3.8-27b';
function fallback(transcript) {
  return {
    title: truncateTitle(transcript), // 50 code points, not UTF-16
    date: null, priority: 'medium', category: null, tags: [],
  };
}
// 10s AbortController; on any error return fallback, still insert

Full context before starting

timon/src/lib/store.js 
return {
  task, parent,
  siblings: siblings.results || [],
  subtasks: subtasks.results || [],
  blockers: blockers.results || [],
};

Jarvis bridge, 5s timeout

apollo/apps/agent/src/tools/timon.ts 
export const timonCreateTaskTool = { name: 'timon_create_task', safety: 'safe' };
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 5_000);
const response = await fetch(`${timonUrl}/api/tasks`, {
  method: 'POST',
  headers: { Authorization: `Bearer ${timonApiKey}` },
  signal: controller.signal,
});

Tests: Timon vitest (intents 27, store, tasks) with fetch stubbed and D1 .all() on unbound statements. Apollo: 511 pass / 0 fail, typecheck clean (RESUME-HERE, 2026-08-21).

Voice path simulator

Client-side only. No microphone, no keys, no network. Walks the real Option A sequence.

Pick a phrase or type one.


        

      


      

        
Results & lessons

        

          

            STT · NID-463 corpus
            
85% strict · 100% normalised

            
20 English phrases, Groq Whisper. Average 2543 ms. Failures were punctuation/number form (“6 p.m”, “Chapter 5”), not meaning. Confirm-before-save is Stage 5.

          

          

            Live workers
            
Both health endpoints up

            
Apollo {"ok":true}. Timon {"status":"ok","service":"timon-worker"}. UI is not on the worker yet — Stage 5 tapp.

          

          

            What broke us
            
Mocks, merges, ungated deploys

            
D1 stmt.all() unbound. Tests that hit live Groq. Three Stage 3 PRs from the same SHA. A wrangler deploy from master without POST /api/tasks. CI is the remaining Stage 4 gate.

          

        

        

          
Lessons

          
Audit the live worker before swapping providers — the “llama-2 swap” was actually the first real LLM. Keep one secret for STT and intent. Never let a Durable Object WebSocket hide a deploy. Write the contract from code, not from memory.