Ask HN: OSS Distributed Ollama Proxy?
2 nodesocket 0 7/16/2025, 5:24:24 AM
Are there any good open source solutions that manages a cluster of ollama serve nodes and distributes 'chat' requests to nodes that are up and not currently processing 'chat' requests?
A http reverse proxy seems to be up to the job, with perhaps consul or redis as the queue and service discovery. Would prefer written in go or rust.
No comments yet