Show HN: Semcache – I built a semantic cache in Rust

Have a look at my semantic caching project!

It's built to easily integrate in existing LLM workflows, you can use it as a proxy where the cache forwards missed requests without modification to a specified upstream, automatically updating it's cache with the response.

You can also use it as a cache-aside cache with a provided python library.

It works by computing embedding vectors of input queries, and matches them to seen query + response pairs using a vector store.

Everything is in-memory, so it should be blazing fast :)