An hour on Algorütm, talking about the console that runs my assistant

I spent an hour on Algorütm talking about the AI assistant that runs on a games console in my living room.

Priit Liivak, Erik Jõgi and Martin Kapp have covered self-hosted AI on the show before, but always the multi-GPU basement rig. Mine is a Steam Machine that was bought to play games and does its real work at night.

8 GB of VRAM fits exactly one model at a time, and that single fact decides the whole design. A tool loads the model it needs and evicts whatever was there. Idle models unload after two minutes. You stop interleaving work and start batching it by what's loaded. One model at a time turns out to be enough for a personal assistant.

Priit asked on the show whether I'll open source the piece that manages all this, and the answer is yes. It's the bit that decides which model is loaded, waits for memory to actually free before starting the next one, and refuses a job rather than freezing the machine. That's what makes a small box usable, not the models. I'm finishing the last few things and it goes public shortly.

We also put it on the spot live. I asked it on air which Algorütm episode is the best one, and it said it has no data on that. I'll take that over a confident guess every time.

Is a local 7B as sharp as a frontier model? No, and I said so on the show. The cloud still does the thinking here. The box does the seeing, the hearing and the speaking.

Thanks to Priit, Erik and Martin for a genuinely good hour. It's in Estonian, link in the first comment.