Isn't this a software problem being solved in hardware? Ideally you would try to avoid going to memory in the first place by fusing the operations, which should be much faster than speeding up memory ops. E.g. you should never do an explicit im2col before a convolution, it should be fused. However it's hard to argue with a 0.019 mm2 area increase.
imtringued · 51m ago
"Fusing im2col with matrix multiplication" is a confused way of saying that the convolution operation should be implemented directly in hardware.
There are two arguments in favor of im2col.
1. "I don't want to implement a dedicated software kernel just for convolutions" aka laziness
2. "I don't want to implement dedicated hardware just for convolution"
The former is a sham, the latter is motivated by silicon area constraints. Implementing convolutions requires exactly the same number of FMAs, so you would end up doubling your chip size and automatically be cursed with 50% utilization from the start unless you do both matrix multiplication and convolutions simultaneously.
"Element wise convolution performs badly because of the irregular memory accesses involved in it." at a first glance sounds like a reasonable argument, but all you're doing with im2col is shifting the "irregular memory accesses" into a separate kernel. It doesn't fundamentally get rid of the "irregular memory accesses".
The problem with the answer is that the irregularity is purely a result of ones perspective. Assuming you implement im2col in hardware, there is in fact nothing difficult about the irregularity. In fact, what is considered irregular here is perfectly predictable from the perspective of the hardware.
All you do is load x pixels from y rows simultaneously, which is extremely data parallel and SIMD friendly. Once the data is in local registers, you can access it any way you want (each register is effectively its own bank), which allows you to easily produce the im2col output stream and feed it straight to your matrix multiplication unit. You could have implemented the convolution directly, but then again you'd only get 50% utilization due to inflexibility.
KnuthIsGod · 7h ago
Cutting edge and innovative AI hardware research from China.
Looks like Amerikan sanctions are driving a new wave of innovation in China.
" This work addresses that gap by introducing the Ten-
sor Manipulation Unit (TMU): a reconfigurable, near-memory
hardware block designed to execute data-movement-intensive
(DMI) operators efficiently. TMU manipulates long datastreams
in a memory-to-memory fashion using a RISC-inspired execution
model and a unified addressing abstraction, enabling broad
support for both coarse- and fine-grained tensor transformations.
The proposed architecture integrates TMU alongside a TPU
within a high-throughput AI SoC, leveraging double buffering
and output forwarding to improve pipeline utilization. Fab-
ricated in SMIC 40 nm technology, the TMU occupies only
0.019 mm2 while supporting over 10 representative TM operators.
Benchmarking shows that TMU alone achieves up to 1413.43×
and 8.54× operator-level latency reduction over ARM A72 and
NVIDIA Jetson TX2, respectively.
When integrated with the in-
house TPU, the complete system achieves a 34.6% reduction in
end-to-end inference latency, demonstrating the effectiveness and
scalability of reconfigurable tensor manipulation in modern AI
SoCs."
yorwba · 1h ago
It's not like AI hardware acceleration is some niche field that nobody would be researching if there were no sanctions. Academics started flocking towards hardware for AI workloads as soon as it became a trendy topic to be working on (of course back then it was mostly convnets). Maybe recent sanctions have increased the total funding pool, but that's not something you can infer by just gesturing at a single paper.
mikewarot · 3h ago
The only memory involved should be at the input and output of a pipeline stage that does an entire layer of an LLM. I'm of the opinion that we'll end up with effectively massive FPGAs with some stages of pipelining that have NO memory access internally, so that you get one token per clock cycle.
100 million tokens per second is currently worth about $130,000,000/day. (Or so ChatGPT 4.1 told me a few days ago)
I'd like to drop that by a factor of at least 1000:1
There are two arguments in favor of im2col.
1. "I don't want to implement a dedicated software kernel just for convolutions" aka laziness
2. "I don't want to implement dedicated hardware just for convolution"
The former is a sham, the latter is motivated by silicon area constraints. Implementing convolutions requires exactly the same number of FMAs, so you would end up doubling your chip size and automatically be cursed with 50% utilization from the start unless you do both matrix multiplication and convolutions simultaneously.
When you read answers like this: https://stackoverflow.com/a/47422548, they are subtly wrong.
"Element wise convolution performs badly because of the irregular memory accesses involved in it." at a first glance sounds like a reasonable argument, but all you're doing with im2col is shifting the "irregular memory accesses" into a separate kernel. It doesn't fundamentally get rid of the "irregular memory accesses".
The problem with the answer is that the irregularity is purely a result of ones perspective. Assuming you implement im2col in hardware, there is in fact nothing difficult about the irregularity. In fact, what is considered irregular here is perfectly predictable from the perspective of the hardware.
All you do is load x pixels from y rows simultaneously, which is extremely data parallel and SIMD friendly. Once the data is in local registers, you can access it any way you want (each register is effectively its own bank), which allows you to easily produce the im2col output stream and feed it straight to your matrix multiplication unit. You could have implemented the convolution directly, but then again you'd only get 50% utilization due to inflexibility.
Looks like Amerikan sanctions are driving a new wave of innovation in China.
" This work addresses that gap by introducing the Ten- sor Manipulation Unit (TMU): a reconfigurable, near-memory hardware block designed to execute data-movement-intensive (DMI) operators efficiently. TMU manipulates long datastreams in a memory-to-memory fashion using a RISC-inspired execution model and a unified addressing abstraction, enabling broad support for both coarse- and fine-grained tensor transformations.
The proposed architecture integrates TMU alongside a TPU within a high-throughput AI SoC, leveraging double buffering and output forwarding to improve pipeline utilization. Fab- ricated in SMIC 40 nm technology, the TMU occupies only 0.019 mm2 while supporting over 10 representative TM operators. Benchmarking shows that TMU alone achieves up to 1413.43× and 8.54× operator-level latency reduction over ARM A72 and NVIDIA Jetson TX2, respectively.
When integrated with the in- house TPU, the complete system achieves a 34.6% reduction in end-to-end inference latency, demonstrating the effectiveness and scalability of reconfigurable tensor manipulation in modern AI SoCs."
100 million tokens per second is currently worth about $130,000,000/day. (Or so ChatGPT 4.1 told me a few days ago)
I'd like to drop that by a factor of at least 1000:1