Always exciting to see Robert Escriva's next database project. Building on top of the incredible engineering of S3 is clever, I'm definitely going to be looking at and learning from the implementation in wal3.
Looking forward to the benchmarks on using S3 for piece and the design of the scale out architecture.
ram_rar · 28m ago
Is this deeply coupled with Chroma or something we can fork out for other applications? I would love to explore this for lightweight change data capture on S3.
rescrv · 2m ago
Absolutely! We took care during the build to address chroma’s needs, but I believe the only requirement is chroma storage. Long term it’ll appear on crates.io as a standalone package.
swyx · 7h ago
> In short, we believe that systems design is about contracts between reality and reality as we see it; reality as we see it is just a fiction. We believe systems exist in two forms: their actual reality and our fictional understanding of them. To us, the programmers, the system's reality is a fiction; we cannot see it, we can only observe it through additional SRE observability and monitoring mechanisms. To the system, this observability is a fiction. Whether due to tooling problems or just measurement error, the fiction presented via metrics, logs, and traces resembles the system itself, but is decidedly not the system itself.
this is very McLuhan/systemantics of you! all abstractions are leaky, but some abstractions let you look at the leaks.
TIL about setsums - one wonders if `fn setsum([String]) -> Digest` works then "nested setsums" must also work for very large scales.
one thing i missed from this post, which otherwise would score perfect marks for a technology introduciton, is benchmarks vs your comparisons on warpstream and friends.
rescrv · 3h ago
Can you elaborate on what you mean by "nesting" setsums? It's an associative and commutative construction, so you can run e.g. map-reduce to map every parquet file to a setsum and then reduce to roll them up.
HammadB · 7h ago
You will always find a copy of McLuhan on the Chroma teams bookshelf !
stopachka · 2h ago
Is there a particular book of McLuhan that ya'll recommend?
rescrv · 3h ago
Hi,
I'm Robert, one of the authors of wal3. Here to answer any questions or explain more about our engineering philosophy.
Looking forward to the benchmarks on using S3 for piece and the design of the scale out architecture.
this is very McLuhan/systemantics of you! all abstractions are leaky, but some abstractions let you look at the leaks.
TIL about setsums - one wonders if `fn setsum([String]) -> Digest` works then "nested setsums" must also work for very large scales.
one thing i missed from this post, which otherwise would score perfect marks for a technology introduciton, is benchmarks vs your comparisons on warpstream and friends.
I'm Robert, one of the authors of wal3. Here to answer any questions or explain more about our engineering philosophy.
Happy hacking, Robert
No comments yet