Why we built it
Useful AI interactions — autocomplete, classify, summarize, extract — should not require a round trip to a model API for every keystroke. We built Gerbil so product teams can ship private, real-time micro-AI where the user already is.
Benefit
Instant, private inference on the user’s GPU. No API bill per request, no exfiltration path, and the same API in the browser and on Node.
Thesis
The smallest AI loops belong on-device. When latency, cost, and privacy matter more than frontier model quality, local inference is the right default.
Findings
- WebGPU makes on-device models fast enough for product UX — tens to hundreds of tokens per second on modern laptops, usable on tablet and phone.
- A drop-in AI SDK provider lets teams reuse generateText, streamText, tools, and structured output without standing up inference infra.
- One cached base model plus small LoRA flavors covers many product surfaces without redownloading multi-gigabyte weights each time.
How this feeds Hamster
Hamster is about giving AI the right inputs to ship the right product. Gerbil explores the other half of that equation: when those loops should run locally — private, cheap, and immediate — so product delivery doesn’t depend on a network call for every micro-interaction.