It’s great to see substrait getting more seriously used!
It has been supported by engines like duckdb but the main serious use case of it I’m aware of is from Apache gluten where it is used to add Velox as the execution engine of Spark.
It’s an ambitious project and certainly has limitations but more projects like this are needed to push it forward.
gavinray · 1h ago
At Hasura/PromptQL, we attempted to use Substrait IR through Datafusion for representing query engine plans but found that not all semantics were supported.
We ended up having to roll our own [0], but I still think Substrait is a fantastic idea (someone has to solve this problem, eventually) and it's got a good group of minds consistently working on it, so my outlook for it is bright.
query plans are tightly coupled to the engine that emits them : cost models, memory layout, parallelism strategy, codegen behavior all vary. substrait enables structural portability, but the actual execution efficiency depends on engine specific rewrites. a plan optimized in duckdb might underperform in sirius unless it's reshaped. how this handled for now?
tucnak · 4h ago
Reminds me of PG-Strom[1] which is a Postgres extension for GPU-bound index access methods (most notably BRIN, select GIS functions) and the like; it relies on proprietary NVIDIA GPUDirect tech for peer-to-peer PCIe transactions between the GPU and NVMe devices. I'm not sure whether amdgpu kernel driver has this capability in the first place, and last I checked (~6 mo. ago) ROCm didn't have this in software.
However, I wonder whether the GPU's are a good fit for this to begin with.
Counterpoint: Xilinx side of the AMD shop has developed Alveo-series accelerators which used to be pretty basic SmartNIC platforms, but have since evolved to include A LOT more programmable logic and compute IP. You may have heard about these in video encoding applications, HFT, Blockchain stuff, what-have-you. A lot of it has to with AI stuff, see Versal[2]. Raw compute figures are often cited as "underwhelming," and it's unfortunate that so many pundits are mistaking the forest for the trees here. I don't think the AI tiles in these devices are really meant for end-to-end LLM inference, even though memory bandwidth in the high-end devices allows it.
The sauce is compute-in-network over fabrics.
Similarly to how PG-Strom would feed the GPU with relational data from disk, or network directly, many AI teams on the datacenter side are now experimenting with data movement, & intermediate computations (think K/V cache management) over 100/200/800+G fabrics. IMHO, compute-in-network is the MapReduce of this decade. Obviously, there's demand for it in the AI space, but a lot of it lends nicely to the more general-purpose applications, like databases. If you're into experimental networking like that, Corundum[3] by Alex Forencich is a great, perhaps the best, open source NIC design for up to 100G line rate. Some of the cards it supports also expose direct-attach NVMe's over MCIO for latency, and typically have as many as two, or four SFP28 ports for bandwidth.
This is a bit naive way to think about it, but it would have to do!
Postgres is not typically considered to "scale well," but oftentimes this is a statement about its tablespaces more than anything; it has foreign data[4] API, which is how you extend Postgres as single point-of-consumption, foregoing some transactional guarantees in the process. This is how pg_analytics[5] brings DuckDB to Postgres, or how Steampipe[6] similarly exposes many Cloud and SaaS applications. Depending on where you stand on this, the so-called alternative SQL engines may seem like moving in the wrong direction. Shrug.
> However, I wonder whether the GPU's are a good fit for this to begin with.
I think the GPU could be a great fit for OLAP, but when it comes to the nasty OLTP use cases the CPU will absolutely dominate.
Strictly serialized transaction processing facilities demand extremely low latency compute to achieve meaningful throughput. When the behavior of transaction B depends on transaction A being fully resolved, there are no magic tricks you can play anymore.
Consider that talking to L1 is at least 1,000x faster than talking to the GPU. Unless you can get a shitload of work done with each CPU-GPU message (and it is usually the case that you can), this penalty is horrifyingly crippling.
tucnak · 2h ago
I think, TrueTime would constitute a "trick," insofar ordering is concerned?
> Consider that talking to L1 is at least 1,000x faster than talking to the GPU.
This is largely true for "traditional" architectures, but s/GPU/TPU and s/L1/CMEM and suddenly this is no big deal anymore. I'd like Googlers to correct me here, but it seems well in line with classic MapReduce, and probably something that they're doing a lot outside of LLM inference... ads?
bob1029 · 1h ago
How does the information get to & from the GPU in the first place?
If a client wishes to use your GPU-based RDBMS engine, it needs to make a trip through the CPU first, does it not?
tucnak · 1h ago
Not necessarily! The setup I'm discussing is explicitly non-GPU, and it's not necessarily a TPU either. Any accelerator card with NoC capability will do: the requests are queued/batched from network, trickle through the adjacent compute/network nodes, and written back to network. This is what "compute-in-network" means; the CPU is never involved, main memory is never involved. You read from network, you write to network, that's it. On-chip memory on these accelerators is orders of magnitude larger than L1 (FPGA's are known for low-latency systolic stuff) and the on-package memory is large HBM stacks similar to those you would find in a GPU.
It has been supported by engines like duckdb but the main serious use case of it I’m aware of is from Apache gluten where it is used to add Velox as the execution engine of Spark.
It’s an ambitious project and certainly has limitations but more projects like this are needed to push it forward.
We ended up having to roll our own [0], but I still think Substrait is a fantastic idea (someone has to solve this problem, eventually) and it's got a good group of minds consistently working on it, so my outlook for it is bright.
[0] https://hasura.github.io/ndc-spec/reference/types.html#query...
However, I wonder whether the GPU's are a good fit for this to begin with.
Counterpoint: Xilinx side of the AMD shop has developed Alveo-series accelerators which used to be pretty basic SmartNIC platforms, but have since evolved to include A LOT more programmable logic and compute IP. You may have heard about these in video encoding applications, HFT, Blockchain stuff, what-have-you. A lot of it has to with AI stuff, see Versal[2]. Raw compute figures are often cited as "underwhelming," and it's unfortunate that so many pundits are mistaking the forest for the trees here. I don't think the AI tiles in these devices are really meant for end-to-end LLM inference, even though memory bandwidth in the high-end devices allows it.
The sauce is compute-in-network over fabrics.
Similarly to how PG-Strom would feed the GPU with relational data from disk, or network directly, many AI teams on the datacenter side are now experimenting with data movement, & intermediate computations (think K/V cache management) over 100/200/800+G fabrics. IMHO, compute-in-network is the MapReduce of this decade. Obviously, there's demand for it in the AI space, but a lot of it lends nicely to the more general-purpose applications, like databases. If you're into experimental networking like that, Corundum[3] by Alex Forencich is a great, perhaps the best, open source NIC design for up to 100G line rate. Some of the cards it supports also expose direct-attach NVMe's over MCIO for latency, and typically have as many as two, or four SFP28 ports for bandwidth.
This is a bit naive way to think about it, but it would have to do!
Postgres is not typically considered to "scale well," but oftentimes this is a statement about its tablespaces more than anything; it has foreign data[4] API, which is how you extend Postgres as single point-of-consumption, foregoing some transactional guarantees in the process. This is how pg_analytics[5] brings DuckDB to Postgres, or how Steampipe[6] similarly exposes many Cloud and SaaS applications. Depending on where you stand on this, the so-called alternative SQL engines may seem like moving in the wrong direction. Shrug.
[1] https://heterodb.github.io/pg-strom/
[2] https://xilinx.github.io/AVED/latest/AVED%2BOverview.html
[3] https://github.com/corundum/corundum
[4] https://wiki.postgresql.org/wiki/Foreign_data_wrappers
[5] https://github.com/paradedb/pg_analytics
[6] https://hub.steampipe.io/#plugins
I think the GPU could be a great fit for OLAP, but when it comes to the nasty OLTP use cases the CPU will absolutely dominate.
Strictly serialized transaction processing facilities demand extremely low latency compute to achieve meaningful throughput. When the behavior of transaction B depends on transaction A being fully resolved, there are no magic tricks you can play anymore.
Consider that talking to L1 is at least 1,000x faster than talking to the GPU. Unless you can get a shitload of work done with each CPU-GPU message (and it is usually the case that you can), this penalty is horrifyingly crippling.
> Consider that talking to L1 is at least 1,000x faster than talking to the GPU.
This is largely true for "traditional" architectures, but s/GPU/TPU and s/L1/CMEM and suddenly this is no big deal anymore. I'd like Googlers to correct me here, but it seems well in line with classic MapReduce, and probably something that they're doing a lot outside of LLM inference... ads?
If a client wishes to use your GPU-based RDBMS engine, it needs to make a trip through the CPU first, does it not?