Please note that the Redis supported there is not "Vector Sets" (the new Redis data type) but one of the indexes types of RedisSearch.
And, about such benchmarks: I tested another vector db benchmark, investigated it a bit, found that it was mostly measuring client implementation latencies and other internal inefficiencies...
In Redis with VSIM I can easily get 50k vSIM/seconds with 300 components vectors with redis-benchmark, yet when I tried to write a quick test for one of those engines I got a lot lower numbers because simply vectors are large (makes serialization in Python slow if not well coded), often these tests are written in high level languages, don't account for differences in client libraries speeds.
TLDR? Benchmarking is hard, for vector systems it is harder, and the results of most of such tests are totally irrelevant.
throwaw12 · 1d ago
> found that it was mostly measuring client implementation latencies and other internal inefficiencies...
As you said benchmarking is hard, but isn't the end to end latency customers will see in their workloads is usually including the client library overheads?
IMO, benchmarks should closely resemble the real world scenarios (excluding variables, e.g. network latency of different cloud providers)
falcor84 · 1d ago
For context for those like myself who weren't familiar with Amazon's S3 Vectors, it's a relatively new S3 bucket type that's optimized for storing vectors for RAG and similar purposes, claiming to the be up to 90% cheaper than storing them in a regular bucket.
Am I missing anything? (I love Vespa.ai)
The README.md contains a screenshot from local testing that's got more results included: https://github.com/zilliztech/VectorDBBench?tab=readme-ov-fi...
And, about such benchmarks: I tested another vector db benchmark, investigated it a bit, found that it was mostly measuring client implementation latencies and other internal inefficiencies...
In Redis with VSIM I can easily get 50k vSIM/seconds with 300 components vectors with redis-benchmark, yet when I tried to write a quick test for one of those engines I got a lot lower numbers because simply vectors are large (makes serialization in Python slow if not well coded), often these tests are written in high level languages, don't account for differences in client libraries speeds.
TLDR? Benchmarking is hard, for vector systems it is harder, and the results of most of such tests are totally irrelevant.
As you said benchmarking is hard, but isn't the end to end latency customers will see in their workloads is usually including the client library overheads?
IMO, benchmarks should closely resemble the real world scenarios (excluding variables, e.g. network latency of different cloud providers)
https://aws.amazon.com/s3/features/vectors/