> Modern RDMA for Ethernet and IP – Supporting intelligent, low-latency transport for high-throughput environments.
> Open Standards and Interoperability – Avoids vendor lock-in while accelerating ecosystem-wide innovation.
> End-to-End Scalability – From routing and provisioning to operations and testing, UEC scales to millions of endpoints.
frantathefranta · 14h ago
Skimming this, it looks like it's trying compete with Infiniband, but combining it with plain Ethernet?
jauntywundrkind · 14h ago
That so much of the document is about how UltaEthernet maps to Libfabric rather confirms that premise! There's also a credit based flow system, and connection manager roles, which are also easily identifiable Infiniband concepts.
I'm interested to see what the optional hardware features are. And what their relationship is to the different UE profiles (AI Base, AI Full, and HPC).
> The Ultra Ethernet Transport (UET) layer is designed to handle the most challenging application scale,
deliver packets reliably and securely, manage and avoid congestion within the network, and react to
contention at the endpoints. Its goals are minimal tail latency and highest network utilization. At the
same time, UET is designed to enable simple hardware and software implementations – such as what
might be required for accelerator-integrated endpoints. UET can be programmed through the OFI
libfabric standard interface. It sets out to address the shortcomings of RoCEv2, specifically its
semantics, transport layer, wire operations, implementation complexities, and scale limits
deaddodo · 14h ago
Self-admittedly less knowledgeable about this subject, but how does this differ from IBoE?
jauntywundrkind · 12h ago
The end of the quote I provided does some very high level contrasting of the well known RoCE v2 RDMA over converged Ethernet, which is the most popular IBoE like thing.
I'm sure there's a lot more nuanced to it all. But I think predictability/utilization/latency in RoCE are worse, that it relies on Explicit Congestion Notification more for flow control. Where-as UE is using Infiniband style credit based flow control, which should insure that any data sent has sufficient throughout allocated to it to be received.
UE seems to be a more direct creation of an Infiniband like network, atop Ethernet but where all players are agreeing to behave in an Infiniband like way with Infiniband predictability, where-as RoCE encapsulates Infiniband data but still behaves more like an Ethernet network lacking the coordination of Infiniband. I'm far from certain; it'd be so fun to have some extensive material to go over to really find out.
bobmcnamara · 14h ago
Seems like it. Sorta how storage grade Ethernet took out fiber channel. Probably shares a lot with it in some of the lower layers.
throw0101d · 13h ago
While currently it seems like the only (?) InfiniBand vendor is Nvida/Mellanox, there used to be more folks selling the gear. For example Intel used to sell switches (had some for an Isilon backend):
> Intel's Omni-Path was very similar but a little different from Infiniband.
While Intel may have had the 'IB-adjacent' Omni-Path, they also had plain-old IB as well. I ran these/similar switches for an Isilon back-end network (starting when Isilon was still an independent company, pre-EMC buyout, pre-Dell buyout, in the OneFS 6.x days):
Intel bought Qlogic's IB division when they sold it off, in 2012; I believe at one point the former QLogic parts were branded Omni-Path before it diverged. (You can still see the DNA of that in how the Omni-Path driver stack is a relative of the Qlogic drivers, last I knew.)
Before that, Mellanox ate Voltaire, who was the other large vendor in the IB space.
So at this point, I believe NVIDIA's Mellanox devices are the only people selling IB chips these days, and glancing at the TOP500 from June, seems like a good half (45.8%) of the supercomputers listed there are using either Ethernet-based or otherwise non-IB stacks.
anonymousDan · 10h ago
So how long before hardware is available that supports this spec? Would the kit likely be cheaper than infiniband (or even ROCE) equivalents?
bgnn · 10h ago
It seems there isn't much difference on physical layer with standard 802.3 100Gb/lane. So in principle this is a software addition. Hardware supporting it should be available quickly.
I'm not familiar with infiniband, so can't comment on the pricing.
lousken · 12h ago
This is cool, but I am more curious about what happens to all the cat 6a cabling in offices, where do we go after 10GBASE-T
bgnn · 10h ago
There's already 25G BASE-T over CAT6 available. I worked on and taped out a 25G PHY in 2019. Cable reach is 30m instead of 100m though. There's work going on 40G for 10m reach.
crote · 10h ago
Desktop computer are barely on 2.5GBASE-T, and there's still 5G in-between. I reckon it'll be 5-10 years until we'll see widespread 10G rollout, and it'll easily last another 10 before we need something better. The bigger issue is wireless access points. High-end ones are already on 10G today, and unlike desktops PoE is quite critical here.
25GBASE-T and 40GBASE-T are dead-on-arrival: the spec exists, but nobody ever bothered to actually ship it. It requires yet another cable upgrade and the maximum distance is too short to be usable. And that's before we even looked at power consumption.
The obvious answer is fiber, but that's orders of magnitude more complicated to roll out. The fiber itself is a massive pain to install when you want to make it to-length on site, the currently-popular connectors require a lot of babysitting, it can't be backwards compatible with the old RJ45 stuff, and it can't do PoE. Pulling a standard single-mode fiber pair with LC connectors to every cubicle and access point? Not exactly an attractive option.
I personally think we're probably going to see some kind of Frankensteined (think GG45-like) RJ45-with-fiber pop up, but that won't be any time soon.
phonon · 7h ago
The Realtek RTL8127 chip for 10GbE will cost $10, draw 1.95 W and is designed for motherboards. So I think you're a bit pessimistic on timelines...
10GBASE-T dates back to 2006, and we've been seeing it on the odd prosumer workstation / elite gamer motherboard for quite a while now. But those markets aren't going to lead to a full office rewiring.
The entry-level market is more interesting: when will we see it on motherboards that retail for $100? When will we see it on the average Dell Optiplex? When will your ISP router come with 10G ports?
Will some people be using 10G before that? Definitely, and I'll be one of them. Will it be something to consider for the average office environment? Nah.
BizarroLand · 12h ago
10g is going to remain sufficient for office use for a long time.
Ultra Ethernet will be primarily for cluster/AI/Super computing, trading, server, and backplane scenarios where high speed low-latency throughput turns into money or bang for your buck.
Debbie in accounting will practically never need more than 10g networking to do her job.
crote · 10h ago
> Debbie in accounting will practically never need more than 10g networking to do her job.
And 640K of RAM ought to be enough for anyone. Who knows, perhaps in 2050 Edward will be mailing Debbie iterations of their 500GB AI fraud model for local inference?
The question isn't if 10G will need to be replaced with something better, but when.
BizarroLand · 9h ago
Unless there are sea changes in the global infrastructure of computing in the next 25 years, that 500gb AI fraud model would be hosted on a server and made available to debbie through a web connection.
Further, almost every person on an internal network still uses 1g, which has been going strong for 25 years so far and will likely only cease being the standard when it's no longer economical for manufacturers to produce 1g hardware.
datadrivenangel · 14h ago
Ethernet but for AI!
jpgvm · 13h ago
UltraEthernet predates the AI boom. I don't blame them for adding the AI buzzwords, for this to succeed people need to think about it in the same vein as IB/RoCE/etc and a lot of the decision makers are unfortunately not technical enough to understand what this is or does.
Ultimately it's just a different (IMO slightly more refined) look at how to support RDMA on Ethernet vs RoCE which is a more ham fisted implementation.
RoCE took an encapsulation approach that has some drawbacks (namely it's reliance on PFC/ECN for congestion management).
This takes a different approach that attempts to actually do first-class re-implementations of Infiniband-ish congestion control with end-to-end credit based flow control similar to Infiniband virtual lanes.
https://ultraethernet.org/ultra-ethernet-consortium-uec-laun...
Their noteworthy features list is:
> Modern RDMA for Ethernet and IP – Supporting intelligent, low-latency transport for high-throughput environments.
> Open Standards and Interoperability – Avoids vendor lock-in while accelerating ecosystem-wide innovation.
> End-to-End Scalability – From routing and provisioning to operations and testing, UEC scales to millions of endpoints.
I'm interested to see what the optional hardware features are. And what their relationship is to the different UE profiles (AI Base, AI Full, and HPC).
> The Ultra Ethernet Transport (UET) layer is designed to handle the most challenging application scale, deliver packets reliably and securely, manage and avoid congestion within the network, and react to contention at the endpoints. Its goals are minimal tail latency and highest network utilization. At the same time, UET is designed to enable simple hardware and software implementations – such as what might be required for accelerator-integrated endpoints. UET can be programmed through the OFI libfabric standard interface. It sets out to address the shortcomings of RoCEv2, specifically its semantics, transport layer, wire operations, implementation complexities, and scale limits
I'm sure there's a lot more nuanced to it all. But I think predictability/utilization/latency in RoCE are worse, that it relies on Explicit Congestion Notification more for flow control. Where-as UE is using Infiniband style credit based flow control, which should insure that any data sent has sufficient throughout allocated to it to be received.
UE seems to be a more direct creation of an Infiniband like network, atop Ethernet but where all players are agreeing to behave in an Infiniband like way with Infiniband predictability, where-as RoCE encapsulates Infiniband data but still behaves more like an Ethernet network lacking the coordination of Infiniband. I'm far from certain; it'd be so fun to have some extensive material to go over to really find out.
* https://www.intel.com/content/dam/support/us/en/documents/ne... (PDF)
Officially there are a bunch of folks in the IB alliance:
* https://www.infinibandta.org/member-listing/
Omni-Path was sold off to Cornelis Networks in 2020. Cornelis just released new 400Gbps products last week, their CN5000 line, with 48 and 576 port switches. https://www.crn.com/news/components-peripherals/intel-spins-...
While Intel may have had the 'IB-adjacent' Omni-Path, they also had plain-old IB as well. I ran these/similar switches for an Isilon back-end network (starting when Isilon was still an independent company, pre-EMC buyout, pre-Dell buyout, in the OneFS 6.x days):
* https://www.dell.com/support/kbdoc/en-us/000019215/install-a...
Before that, Mellanox ate Voltaire, who was the other large vendor in the IB space.
So at this point, I believe NVIDIA's Mellanox devices are the only people selling IB chips these days, and glancing at the TOP500 from June, seems like a good half (45.8%) of the supercomputers listed there are using either Ethernet-based or otherwise non-IB stacks.
I'm not familiar with infiniband, so can't comment on the pricing.
25GBASE-T and 40GBASE-T are dead-on-arrival: the spec exists, but nobody ever bothered to actually ship it. It requires yet another cable upgrade and the maximum distance is too short to be usable. And that's before we even looked at power consumption.
The obvious answer is fiber, but that's orders of magnitude more complicated to roll out. The fiber itself is a massive pain to install when you want to make it to-length on site, the currently-popular connectors require a lot of babysitting, it can't be backwards compatible with the old RJ45 stuff, and it can't do PoE. Pulling a standard single-mode fiber pair with LC connectors to every cubicle and access point? Not exactly an attractive option.
I personally think we're probably going to see some kind of Frankensteined (think GG45-like) RJ45-with-fiber pop up, but that won't be any time soon.
https://www.tomshardware.com/networking/realteks-usd10-tiny-...
10GBASE-T dates back to 2006, and we've been seeing it on the odd prosumer workstation / elite gamer motherboard for quite a while now. But those markets aren't going to lead to a full office rewiring.
The entry-level market is more interesting: when will we see it on motherboards that retail for $100? When will we see it on the average Dell Optiplex? When will your ISP router come with 10G ports?
Will some people be using 10G before that? Definitely, and I'll be one of them. Will it be something to consider for the average office environment? Nah.
Ultra Ethernet will be primarily for cluster/AI/Super computing, trading, server, and backplane scenarios where high speed low-latency throughput turns into money or bang for your buck.
Debbie in accounting will practically never need more than 10g networking to do her job.
And 640K of RAM ought to be enough for anyone. Who knows, perhaps in 2050 Edward will be mailing Debbie iterations of their 500GB AI fraud model for local inference?
The question isn't if 10G will need to be replaced with something better, but when.
Further, almost every person on an internal network still uses 1g, which has been going strong for 25 years so far and will likely only cease being the standard when it's no longer economical for manufacturers to produce 1g hardware.
Ultimately it's just a different (IMO slightly more refined) look at how to support RDMA on Ethernet vs RoCE which is a more ham fisted implementation.
RoCE took an encapsulation approach that has some drawbacks (namely it's reliance on PFC/ECN for congestion management).
This takes a different approach that attempts to actually do first-class re-implementations of Infiniband-ish congestion control with end-to-end credit based flow control similar to Infiniband virtual lanes.
UEC was formed/announced in July 2023:
* https://ultraethernet.org/leading-cloud-service-semiconducto...
ChatGPT was launched in November 2022:
* https://en.wikipedia.org/wiki/ChatGPT
* https://en.wikipedia.org/wiki/GPT-4