While I superficially agree with the point, the article author states that they haven't tried the alternatives they suggest. Implementing CRDT or OT can be a complex endeavour, especially if you're retrofitting on an existing system that already has a REST or similar API. Use case is ever important, don't spend your all important time implementing a complex state sync if it turns out your customers don't do the kind of collaboration that requires it.
kokanee · 4h ago
It seems like the author has never worked on an interface that uses realtime UI syncing. The points about cumbersome and mundane issues when working with REST UIs are not wrong, but the challenges of working with realtime synced UIs are substantially more difficult. And I'm not talking about conflict resolution, which can be abstracted by many existing solutions. I'm talking about the intricacies of UI behaviors that are specific to each application, when suddenly every interactive element can potentially be interacted with by multiple parties simultaneously.
For most applications, the status quo whereby a little bit of recent state is maintained locally, and periodically pushed to a centralized store, seems to provide the best overall balance of complexity and functionality. One exception is that most applications do a poor job of actually maintaining the local state (e.g. most forms clear when you refresh the page, which is not hard to fix).
qudat · 3h ago
State synchronization is a bit of a hot topic for front end development since most of the “middle end” deals squarely with solving that problem.
However, virtually every company I’ve joined has needed to serve multiple clients. In those cases REST is the status quo. Some people have opted for graphql but even that makes some uneasy because tuning gql is non trivial. CRDTs are completely foreign to most organizations unless they deal specifically with multiplayer modes.
So while it sounds nice to ditch REST it’s just not realistic for most orgs. This is why I’ve been developing a side effect and state sync system that can easily interface with REST, websockets, graphql, etc because it’s built off of structured concurrency.
What part of this implementation is ReST? I see a CRUD API that uses HTTP verbs. State transfer implies multiple steps at least, where I call the initial API which then returns several other API endpoints "transfering the state" typically in a HATEOAS style - perhaps that part isn't required.
> ... although this term is more commonly associated with the design of HTTP-based APIs and what are widely considered best practices regarding the "verbs" (HTTP methods) a resource responds to, while having little to do with REST as originally formulated ...
Ah, I see, the industry has taken to just calling HTTP ReST for no apparent reason.
As far as not being sure about CRDTs, these protocols were made to overcome the obvious and terrible shortcomings of CRUD APIs (lack of communicability and idempotency). Who ever wants to see "the data on the server has changed, do you want to overwrite the data on the server or overwrite your local copy". If you're not doing some sort of event sourcing for state management (ES, OT, CRDT) you're probably doing it wrong or doing it for a simple project.
WhitneyLand · 3h ago
Not sure why you think that’s not rest. Because Roy Fielding wrote a paper that most people don’t care about?
Language isn’t controlled by a single authority or the first person to use a term. Words mean what people use them to mean. You can complain that it’s not the ‘proper’ definition, but if the majority of people use it a certain way, that becomes the definition in practice.
edit: Fielding not Crawford (thx ackfoobar)
ackfoobar · 2h ago
Similarly Java falls outside of Alan Kay's definition of OOP. But unlike Roy Fielding's ReST, I don't see many people bringing that up.
827a · 1h ago
Its more fun at parties to just choose to believe that many of these APIs companies deploy are REST APIs, where the word "REST" is just a word, doesn't stand for anything, and is entirely unrelated to whatever that Roy guy was talking about all those years ago.
cgio · 3h ago
REST as the author finds out is not for state synchronisation. CRDTs are also not about state synchronisation, only so in an eventual sense, so similar to REST, you can make it work, but you are twisting the approach with assumptions and sticky tape. State synchronisation is a “solved” problem, in the form of distributed data stores, or in some more relaxed sense, database replication, CDC etc. Not an expert, so this is just an opinion even if it sounds assertive.
No comments yet
sansseriff · 4h ago
Curious to her other people's opinion of this. I arrived at a similar conclusion after making web apps for research control software in my PhD. I went through a Yjs tutorial and looked into integrating it with fastapi websockets. But this seems like a pretty unusual thing; there just isn't enough people doing this.
Nice user-friendly libraries and tutorials don't exist for smoothing the transition from REST to CRDTs, should your app need that.
coolhand2120 · 4h ago
If you're hesitating to use YJS take the word of someone who took the plunge, it's totally worth it on the other side.
I've written several apps now with it now. Very easy to use and quite robust to failures. It's a bit of a mental load to take on at first but it's totally worth it for the problems it solves out of the box. I've tried other things too from rolling my own ES stack to OT and more.
Lately I've got it running on AWS API Gateway V2 over websockets to lambdas + DynamoDB with a small army of daily users. The only expensive part is the event audit logs I keep due to my inherent mistrust of all computers.
techno-beetle · 59m ago
A few questions regarding CRDT usage from someone that's lightly tested out automerge/autosurgeon in rust late last year, but hasn't used it or any other CRDT for an actual project.
1. Do you use the CRDT document as the source of truth or as just synchronization with a database as the source of truth. If the document is the source of truth, do you keep data in it or copy the data into some other format that's easier to query?
2. How do you handle changes to the schema of the CRDT documents?
In my testing I had a `version` field at the top level of the documents and then a function to migrate forward between versions when a document is loaded, but I'm not sure how to handle when different clients are running different versions concurrently as opposed to all clients updating at the same time.
I had read some articles that alluded to allowing the previous versions to still change the state and then, seemingly, translate it as needed on newer versions, but they seemed to hand-wave away any details of what that would actually look like to implement.
3. How granular do you go with documents in the spectrum of "one per user" to "one per object"?
toomim · 1h ago
I work on the referenced Braid project (https://braid.org), and the author is right on— HTTP needs to evolve from a state transfer protocol to a state synchronization protocol.
What he doesn't seem to realize is (a) how doable this is, and (b) how beneficial it is once you solve it.
First, the Braid extensions to HTTP [1] achieve this already! They transform HTTP from state transfer to state synchronization with four simple, backwards-compatible extensions to HTTP, which work in today's browsers and servers, with a simple polyfill library [2].
You can use this today, with or without any CRDT or OT algorithms. In fact, 99% of apps don't need a fancy CRDT — they just need the subscriptions feature. This is really cool: instead of GET responding just once, a Braid-enhanced GET will keep the response open, and send you new updates each time the resource changes, until you're done with it.
This gives you realtime synchronized HTTP without a Websocket. No CRDT or OT needed.
You only need a CRDT or OT algorithm if multiple writers mutate the same state, and your app needs to guarantee perfect consistency. And if this is needed—no problem! We've got libraries for that that use the other 3 independent Braid extensions — essentially adding a few more headers to carry over the CRDT/OT metadata. They work with any CRDT or OT.
Second, I don't think anyone yet realizes how big of a deal this is, so let me explain. The lack of realtime synchronization in today's HTTP is what drives developers to move their state synchronization traffic to a websocket, with some random pub/sub protocol that meets the needs of their app — but is totally incompatible with other developers, other websites, and other tools. This means that every website uses standard HTTP to serve its pages, but has a non-standard protocol for its internal state. As a result, websites openly link to each other's pages, on the open web — but the internal state of websites has become a walled garden. This basic limitation in the architecture of HTTP has led to the centralization of data silos in websites. It's simply too difficult for a peer-to-peer community of web developers to build apps that re-use each others' internal state, in the same way that they can link to each other's pages, because they lack the infrastructure.
We've been building the tools for this new style of peer-to-peer synchronous web of state. I just presented on it a few weeks back here: https://braid.org/meeting-107
Cheers to the author for correctly identifying this problem. And onward to the glorious future of the synchronous web of state!
And you can show up to our open biweekly meetings on zoom: https://braid.org
healthydyd · 50m ago
This looks pretty amazing. I am definitely going to try it out.
4b11b4 · 3h ago
Electric SQL and their various packages looking pretty. Can't comment on actually using them though
3cats-in-a-coat · 1h ago
"State transfer is not state synchronization" is such an obscure point to make.
State sync over REST can be rather trivial by keeping resources immutable (hence also cacheable), and delivering a root resource during the initial endpoint contact. I.e. like an index page linking to static assets. Which is all very familiar to REST design.
yakshaving_jgt · 4h ago
The article told us we should stop using REST for state sync, yet it didn’t really tell us what to do instead.
AtlasBarfed · 4h ago
This is right smack in the territory of the CAP theorem, which a lot of distributed databases using highly reliable networks can now somewhat functionally ignore.
But in the land of highly variable reliability of clients, you got to think about it you are CP or AP. You HAVE to be partition tolerant because clients are so unreliable.
gxs · 3h ago
Maybe if the headline said Stop using REST for state synchronization in these scenarios, I would have been on board
Dismissing something outright is usually the wrong answer, and for the same reason I did read the article
Using REST for synchronization isn’t some huge transgression and is fine in a lot of scenarios, silly in others.
Keeping record updates that a user makes in one system in sync with a single record in another system that uses those updates is fine
Trying to sync giant databases in bulk on the other hand might not be
For most applications, the status quo whereby a little bit of recent state is maintained locally, and periodically pushed to a centralized store, seems to provide the best overall balance of complexity and functionality. One exception is that most applications do a poor job of actually maintaining the local state (e.g. most forms clear when you refresh the page, which is not hard to fix).
I also agree that the way many APIs are built are pushing complexity to the FE: https://bower.sh/dogma-of-restful-api
However, virtually every company I’ve joined has needed to serve multiple clients. In those cases REST is the status quo. Some people have opted for graphql but even that makes some uneasy because tuning gql is non trivial. CRDTs are completely foreign to most organizations unless they deal specifically with multiplayer modes.
So while it sounds nice to ditch REST it’s just not realistic for most orgs. This is why I’ve been developing a side effect and state sync system that can easily interface with REST, websockets, graphql, etc because it’s built off of structured concurrency.
https://starfx.bower.sh/
https://bower.sh/why-structured-concurrency
https://en.wikipedia.org/wiki/REST
> ... although this term is more commonly associated with the design of HTTP-based APIs and what are widely considered best practices regarding the "verbs" (HTTP methods) a resource responds to, while having little to do with REST as originally formulated ...
Ah, I see, the industry has taken to just calling HTTP ReST for no apparent reason.
As far as not being sure about CRDTs, these protocols were made to overcome the obvious and terrible shortcomings of CRUD APIs (lack of communicability and idempotency). Who ever wants to see "the data on the server has changed, do you want to overwrite the data on the server or overwrite your local copy". If you're not doing some sort of event sourcing for state management (ES, OT, CRDT) you're probably doing it wrong or doing it for a simple project.
Language isn’t controlled by a single authority or the first person to use a term. Words mean what people use them to mean. You can complain that it’s not the ‘proper’ definition, but if the majority of people use it a certain way, that becomes the definition in practice.
edit: Fielding not Crawford (thx ackfoobar)
No comments yet
Nice user-friendly libraries and tutorials don't exist for smoothing the transition from REST to CRDTs, should your app need that.
I've written several apps now with it now. Very easy to use and quite robust to failures. It's a bit of a mental load to take on at first but it's totally worth it for the problems it solves out of the box. I've tried other things too from rolling my own ES stack to OT and more.
Lately I've got it running on AWS API Gateway V2 over websockets to lambdas + DynamoDB with a small army of daily users. The only expensive part is the event audit logs I keep due to my inherent mistrust of all computers.
1. Do you use the CRDT document as the source of truth or as just synchronization with a database as the source of truth. If the document is the source of truth, do you keep data in it or copy the data into some other format that's easier to query?
2. How do you handle changes to the schema of the CRDT documents? In my testing I had a `version` field at the top level of the documents and then a function to migrate forward between versions when a document is loaded, but I'm not sure how to handle when different clients are running different versions concurrently as opposed to all clients updating at the same time. I had read some articles that alluded to allowing the previous versions to still change the state and then, seemingly, translate it as needed on newer versions, but they seemed to hand-wave away any details of what that would actually look like to implement.
3. How granular do you go with documents in the spectrum of "one per user" to "one per object"?
What he doesn't seem to realize is (a) how doable this is, and (b) how beneficial it is once you solve it.
First, the Braid extensions to HTTP [1] achieve this already! They transform HTTP from state transfer to state synchronization with four simple, backwards-compatible extensions to HTTP, which work in today's browsers and servers, with a simple polyfill library [2].
You can use this today, with or without any CRDT or OT algorithms. In fact, 99% of apps don't need a fancy CRDT — they just need the subscriptions feature. This is really cool: instead of GET responding just once, a Braid-enhanced GET will keep the response open, and send you new updates each time the resource changes, until you're done with it.
This gives you realtime synchronized HTTP without a Websocket. No CRDT or OT needed.
You only need a CRDT or OT algorithm if multiple writers mutate the same state, and your app needs to guarantee perfect consistency. And if this is needed—no problem! We've got libraries for that that use the other 3 independent Braid extensions — essentially adding a few more headers to carry over the CRDT/OT metadata. They work with any CRDT or OT.
Second, I don't think anyone yet realizes how big of a deal this is, so let me explain. The lack of realtime synchronization in today's HTTP is what drives developers to move their state synchronization traffic to a websocket, with some random pub/sub protocol that meets the needs of their app — but is totally incompatible with other developers, other websites, and other tools. This means that every website uses standard HTTP to serve its pages, but has a non-standard protocol for its internal state. As a result, websites openly link to each other's pages, on the open web — but the internal state of websites has become a walled garden. This basic limitation in the architecture of HTTP has led to the centralization of data silos in websites. It's simply too difficult for a peer-to-peer community of web developers to build apps that re-use each others' internal state, in the same way that they can link to each other's pages, because they lack the infrastructure.
We've been building the tools for this new style of peer-to-peer synchronous web of state. I just presented on it a few weeks back here: https://braid.org/meeting-107
Cheers to the author for correctly identifying this problem. And onward to the glorious future of the synchronous web of state!
[1] https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-b...
[2] https://github.com/braid-org/braid-http?tab=readme-ov-file
Mailing list: https://groups.google.com/forum/#!forum/braid-http
Discord: https://discord.gg/nvPQN7FgDX
My email: toomim@gmail.com
And you can show up to our open biweekly meetings on zoom: https://braid.org
State sync over REST can be rather trivial by keeping resources immutable (hence also cacheable), and delivering a root resource during the initial endpoint contact. I.e. like an index page linking to static assets. Which is all very familiar to REST design.
But in the land of highly variable reliability of clients, you got to think about it you are CP or AP. You HAVE to be partition tolerant because clients are so unreliable.
Dismissing something outright is usually the wrong answer, and for the same reason I did read the article
Using REST for synchronization isn’t some huge transgression and is fine in a lot of scenarios, silly in others.
Keeping record updates that a user makes in one system in sync with a single record in another system that uses those updates is fine
Trying to sync giant databases in bulk on the other hand might not be