The part that was most surprising to me was how much the performance of serializing floating-point numbers has improved, even just in the past decade [1].
JSON encoding is a huge impediment to interprocess communication in NodeJS.
Sooner or later is seems like everyone gets the idea of reducing event loop stalls in their NodeJS code by trying to offload it to another thread, only to discover they’ve tripled the CPU load in the main thread.
I’ve seen people stringify arrays one entry at a time. Sounds like maybe they are doing that internally now.
If anything I would encourage the V8 team to go farther with this. Can you avoid bailing out for subsets of data? What about the CString issue? Does this bring faststr back from the dead?
jcdavis · 2h ago
Based off of my first ever forays into node performance analysis last year, JSON.stringify was one of the biggest impediments to just about everything around performant node services. The fact that everyone uses stringify to for dict keys, the fact that apollo/express just serializes the entire response into a string instead of incrementally streaming it back (I think there are some possible workarounds for this, but they seemed very hacky)
As someone who has come from a JVM/go background, I was kinda shocked how amateur hour it felt tbh.
hinkley · 1h ago
> Based off of my first ever forays into node performance analysis last year, JSON.stringify was one of the biggest impediments to just about everything around performant node services
Just so. It is, or at least can be, the plurality of the sequential part of any Amdahl's Law calculation for Nodejs.
I'm curious if any of the 'side effect free' commentary in this post is about moving parts of the JSON calculation off of the event loop. That would certainly be very interesting if true.
However for concurrency reasons I suspect it could never be fully off. The best you could likely do is have multiple threads converting the object while the event loop remains blocked. Not entirely unlike concurrent marking in the JVM.
MehdiHK · 2h ago
> JSON.stringify was one of the biggest impediments to just about everything around performant node services
That's what I experienced too. But I think the deeper problem is Node's cooperative multitasking model. A preemptive multitasking (like Go) wouldn't block the whole event-loop (other concurrent tasks) during serializing a large response (often the case with GraphQL, but possible with any other API too). Yeah, it does kinda feel like amateur hour.
dmit · 1h ago
Node is the biggest impediment to performant Node services. The entire value proposition is "What if you could hire people who write code in the most popular programming language in the world?" Well, guess what
brundolf · 1h ago
Yeah. I think I've only ever found one situation where offloading work to a worker saved more time than was lost through serializing/deserializing. Doing heavy work often means working with a huge set of data- which means the cost of passing that data via messages scales with the benefits of parallelizing the work.
hinkley · 1h ago
I think the clues are all there in the MDN docs for web workers. Having a worker act as a forward proxy for services; you send it a URL, it decides if it needs to make a network request, it cooks down the response for you and sends you the condensed result.
Most tasks take more memory in the middle that at the beginning and end. And if you're sharing memory between processes that can only communicate by setting bytes, then the memory at the beginning and end represents the communication overhead. The latency.
But this is also why things like p-limit work - they pause an array of arbitrary tasks during the induction phase, before the data expands into a complex state that has to be retained in memory concurrent with all of its peers. By partially linearizing you put a clamp on peak memory usage that Promise.all(arr.map(...)) does not, not just the thundering herd fix.
dwattttt · 1h ago
Now to just write the processing code in something that compiles to WebAssembly, and you can start copying and sending ArrayBuffers to your workers!
Or I guess you can do it without the WebAssembly step.
ot · 34m ago
The SWAR escaping algorithm [1] is very similar to the one I implemented in Folly JSON a few years ago [2]. The latter works on 8 byte words instead of 4 bytes, and it also returns the position of the first byte that needs escaping, so that the fast path does not add noticeable overhead on escape-heavy strings.
An important question that was not addressed is whether the general path will be slower to account for what is needed to check first if the fast path can be used.
monster_truck · 1h ago
I don't think v8 gets enough praise. It is fucking insane how fast javascript can be these days
andyferris · 1h ago
Yeah, it is quite impressive!
It's a real example of "you can solve just about anything with a billion dollars" though :)
I'd prefer JavaScript kept evolving (think "strict", but "stricter", "stricter still", ...) to a simpler and easier to compile/JIT language.
ayaros · 14m ago
Yes, this is what I want too. Give me "stricter" mode.
pyrolistical · 24m ago
> Optimizing the underlying temporary buffer
So array list instead of array?
MutedEstate45 · 2h ago
I really like seeing the segmented buffer approach. It's basically the rope data structure trick I used to hand-roll in userland with libraries like fast-json-stringify, now native and way cleaner. Have you run into the bailout conditions much? Any replacer, space, or custom .toJSON() kicks you back to the slow path?
taeric · 1h ago
I confess that I'm at a bit of a loss to know what sort of side effects would be common when serializing something? Is there an obvious class of reasons for this that I'm just accidentally ignoring right off?
vinkelhake · 1h ago
A simple example is `toJSON`. If an object defines that method, it'll get invoked automatically by JSON.stringify and it could have arbitrary side effects.
I think it's less about side effects being common when serializing, just that their fast path avoids anything that could have side effects (like toJSON).
The article touches briefly on this.
kevingadd · 31m ago
Calling a property getter can have side effects, so if you serialize an object with a getter you have to be very cautious to make sure nothing weird happens underneath you during serialization.
People have exploited this sort of side effect to get bug bounties before via type confusion attacks, iirc.
iouser · 1h ago
Did you run any tests/regressions against the security problems that are common with parsers? Seems like the solution might be at risk of creating CVEs later
[1] https://github.com/jk-jeon/dragonbox?tab=readme-ov-file#perf...
Sooner or later is seems like everyone gets the idea of reducing event loop stalls in their NodeJS code by trying to offload it to another thread, only to discover they’ve tripled the CPU load in the main thread.
I’ve seen people stringify arrays one entry at a time. Sounds like maybe they are doing that internally now.
If anything I would encourage the V8 team to go farther with this. Can you avoid bailing out for subsets of data? What about the CString issue? Does this bring faststr back from the dead?
As someone who has come from a JVM/go background, I was kinda shocked how amateur hour it felt tbh.
Just so. It is, or at least can be, the plurality of the sequential part of any Amdahl's Law calculation for Nodejs.
I'm curious if any of the 'side effect free' commentary in this post is about moving parts of the JSON calculation off of the event loop. That would certainly be very interesting if true.
However for concurrency reasons I suspect it could never be fully off. The best you could likely do is have multiple threads converting the object while the event loop remains blocked. Not entirely unlike concurrent marking in the JVM.
That's what I experienced too. But I think the deeper problem is Node's cooperative multitasking model. A preemptive multitasking (like Go) wouldn't block the whole event-loop (other concurrent tasks) during serializing a large response (often the case with GraphQL, but possible with any other API too). Yeah, it does kinda feel like amateur hour.
Most tasks take more memory in the middle that at the beginning and end. And if you're sharing memory between processes that can only communicate by setting bytes, then the memory at the beginning and end represents the communication overhead. The latency.
But this is also why things like p-limit work - they pause an array of arbitrary tasks during the induction phase, before the data expands into a complex state that has to be retained in memory concurrent with all of its peers. By partially linearizing you put a clamp on peak memory usage that Promise.all(arr.map(...)) does not, not just the thundering herd fix.
Or I guess you can do it without the WebAssembly step.
[1] https://source.chromium.org/chromium/_/chromium/v8/v8/+/5cbc...
[2] https://github.com/facebook/folly/commit/2f0cabfb48b8a8df84f...
It's a real example of "you can solve just about anything with a billion dollars" though :)
I'd prefer JavaScript kept evolving (think "strict", but "stricter", "stricter still", ...) to a simpler and easier to compile/JIT language.
So array list instead of array?
I think it's less about side effects being common when serializing, just that their fast path avoids anything that could have side effects (like toJSON).
The article touches briefly on this.
People have exploited this sort of side effect to get bug bounties before via type confusion attacks, iirc.