> In short, we believe that systems design is about contracts between reality and reality as we see it; reality as we see it is just a fiction. We believe systems exist in two forms: their actual reality and our fictional understanding of them. To us, the programmers, the system's reality is a fiction; we cannot see it, we can only observe it through additional SRE observability and monitoring mechanisms. To the system, this observability is a fiction. Whether due to tooling problems or just measurement error, the fiction presented via metrics, logs, and traces resembles the system itself, but is decidedly not the system itself.
this is very McLuhan/systemantics of you! all abstractions are leaky, but some abstractions let you look at the leaks.
TIL about setsums - one wonders if `fn setsum([String]) -> Digest` works then "nested setsums" must also work for very large scales.
one thing i missed from this post, which otherwise would score perfect marks for a technology introduciton, is benchmarks vs your comparisons on warpstream and friends.
rescrv · 1h ago
Can you elaborate on what you mean by "nesting" setsums? It's an associative and commutative construction, so you can run e.g. map-reduce to map every parquet file to a setsum and then reduce to roll them up.
HammadB · 5h ago
You will always find a copy of McLuhan on the Chroma teams bookshelf !
stopachka · 1h ago
Is there a particular book of McLuhan that ya'll recommend?
rescrv · 1h ago
Hi,
I'm Robert, one of the authors of wal3. Here to answer any questions or explain more about our engineering philosophy.
this is very McLuhan/systemantics of you! all abstractions are leaky, but some abstractions let you look at the leaks.
TIL about setsums - one wonders if `fn setsum([String]) -> Digest` works then "nested setsums" must also work for very large scales.
one thing i missed from this post, which otherwise would score perfect marks for a technology introduciton, is benchmarks vs your comparisons on warpstream and friends.
I'm Robert, one of the authors of wal3. Here to answer any questions or explain more about our engineering philosophy.
Happy hacking, Robert
No comments yet