In theory .Net 10 should make this obsolete, the headline features[1] are basically all about this. In practice, well, it's heuristics, I'm adding this to a particularly performance sensitive project right now :)
Edit: what's also nice is that C# recognizes Linq as a contract. So long as this has the correct method names and signatures (it does), the Linq syntax will light up automatically. You can also use this trick for your own home-grown things (add Select, Join, Where, etc. overloads) if the Linq syntax is something you like.
Could you elaborate? I don’t see anything about improving the performance of enumerator. Zlinq appears to remove the penalty of allocating enumerators on the heap to be garbage collected. The link you sent mention improvements, but I don’t see how they lead to linq avoiding heap allocations.
giancarlostoro · 4h ago
Not just that but Zlinq also works across all C# environments it seems including versions embedded in game engines like Godot, Unity, .NET Standard, .NET 8 and 9.
kevingadd · 4h ago
I believe they're referring to the stack allocation improvements, which would ideally allow all the LINQ temporary objects to live on the stack. I'm not sure whether it does in practice though.
andyayers · 3h ago
Unfortunately, those improvements don't work for Linq.
Aw, I had no idea that it didn't work out. If they work that out I'd put good money on a colossal perf boost across the board.
jasonthorsness · 5h ago
This is great. I've worked on production .NET services and we often had to avoid LINQ in hot paths because of the allocations. Reimplementing functions with for-loops and other structures was time-consuming and error-prone compared to LINQ method chaining. Chaining LINQ methods is extremely powerful; like filter, map, and reduce in JS but with a bunch of other operators and optimizations. I wish more languages had something like it.
meisel · 3h ago
What are the advantages of this over using higher order functions? In Ruby I can do list.map { }.select { } …. That feels more natural (doesn’t require special language support), has a very rich set of functions (group_by, chunk_while, etc.), and is something the user can extend with their own methods (if they don’t mind monkeypatching)
int_19h · 2h ago
LINQ is higher-order functions - Ruby `map` is 'Enumerable.Select`, Ruby `select` is `Enumerable.Where` etc.
The special syntax is really just syntactic sugar on top of all this that makes things a little bit more readable for complex queries because e.g. you don't have to repeat computed variables every time after binding it once in the chain. Consider:
from x in xs
where x.IsFoo
let y = Frob(x)
where y.IsBar
let z = Frob(y)
where z.IsBaz
order by x, y descending, z
select z;
If you were to rewrite this with explicit method calls and lambdas, it becomes something like:
Note how it needs to weave `x` and `y` through all the Select/Where calls so that they can be used for ordering in the end here, whereas with syntactic sugar the scope of `let` extends to the remainder of the expression (although under the hood it still does roughly the same thing as the handwritten code).
KallDrexx · 3h ago
This is neat, but how does this get away with being zero allocation? It appears to use `Func<T,U>` for its predicates, and since `Func<T>` is a reference type this will allocate. The IL definitely generates definitely seems like it's forming a reference type from what I can tell.
ziml77 · 3h ago
The JIT can optimize this. I know for sure if there's no captures in the lambda it won't allocate. It's likely also smart enough to recognize when a function parameter doesn't have its lifetime extended past the caller's, which is a case where it would also be possible to not allocate.
theolivenbaum · 39m ago
To add on that, you can define your lambdas as static to make sure you're not capturing anything by mistake.
Something like dates.Where(static x => x.Date > DateTime.Now)
bigmattystyles · 4h ago
This is cool - excited to try it - I would note that I've been a dotnet grunt for almost 15 years now. I am good at it, I know how to use the language, I know the ecosystem - this level of familiarity with the language is just not within my grasp. I can understand the code (mostly) reading it, but I never would have been able to conjure up, let alone implement this. Props to the author.
HexDecOctBin · 4h ago
What features does C# has that makes LINQ possible in it and not in other languages?
tehlike · 4h ago
It's part of the compiler - ast. Linq has two forms - one in the linq ordinary syntax
from x
select x.name
And other is just lambda with anonymous types and so on.
Of course, if you want to run this against a query provider, you do need compiler support to instead give you an expression tree, and provider to process it and convert them to a language (often sql) that database can understand.
C# can turn lambdas into expression trees at runtime allowing libraries like EF to transform code like `db.Products.Where(p => p.Price < 100).Select(p => p.Name);` right to SQL by iterating the structure of that code. JavaScript ORMs would be revolutionized if they had this ability.
vosper · 20m ago
> JavaScript ORMs would be revolutionized if they had this ability.
Is this possible in JavaScript?
Arnavion · 1m ago
There is a limited form of such "expression rewriting" using tagged template strings introduced in ES2015. But it wouldn't be particularly useful for the ORM case.
Merad · 3h ago
Basic LINQ on in-memory collections isn't really that different from what you have in other languages. Where things get special is the LINQ used by Entity Framework. It operates on expressions, which allow code to be compiled into the application and manipulated at runtime. For example, the lambda expression that you pass to Where() will be examined by an EF query provider that translates it into the where clause for a SQL query.
hansvm · 2h ago
C# is definitely not the only possible language, but some things stand out:
1. You can extend other people's interfaces. If you care about method chaining, _something_ like that is required (alternative solutions include syntactic support for method chaining as a generic function-call syntax).
2. The language has support for "code as data." The mechanism is expression trees, but it's really powerful to be able to use method chaining to define a computation and then dispatch it to different backends, complete with optimization steps.
3. The language has a sub-language as a form of syntactic sugar, allowing certain blessed constructs to be written as basically inline SQL with full editor support.
CharlieDigital · 1h ago
Expression trees are highly underrated.
Compare C# ORMs to JS/TS for example. In C#, it is possible to use expression trees to build queries. In TS, the only options are as strings or using structural representation of the trees.
Yikes! Look how dicey that gets with even a basic query!
osigurdson · 3h ago
I get that Go maintainers want to keep things simple, but this stuff is pretty useful.
sherburt3 · 4h ago
I feel like pretty much every language with generics has a LINQ, like functools/itertools in Python, lodash for javascript. It’s just a different expression of the same ideas.
jeswin · 4h ago
Nope, very different. Depending on whether the expression is on an Enumerable or a Queryable, the compiler generates an anonymous function or an AST. That is, you can get "code as data" as in say Lisp; and allows expressions to be converted to say SQL based on the backend.
incoming1211 · 6h ago
Is there a reason these sort of improvements cannot be contributed back into .NET itself?
nikeee · 5h ago
ZLinq relies on its own enumerable type called ValueEnumerable, which is a struct. While it would probably work when using this as a drop-in replacement and re-compiling, things will be more complicated in larger applications. There might be some code that depends on the exact signature of the Linq methods. This might not even be detectable in cases involving reflection and could break stuff silently.
Adding another enumerable type would be a very large change that could effectively double the API surface of the entire ecosystem. This could take some time. Some places still don't even support Span<T>. Also there were some design decisions related to Linq where the number of overloads were a consideration.
Adding this API to .NET could probably be done with that extension method that converts to ValueEnumerable. But without support for that enumerable, this would pretty much be a walled garden where you have to convert back and forth between different enumerable types. Not that great if you'd ask me, but possible I guess.
theolivenbaum · 36m ago
There are some minor breaking changes like the order of iteration is not always the same as the official Linq implementation, or Sum might give different values due to checked vs unchecked summing. Probably not an issue for most people, but a subtle breaking change nevertheless.
I can easily imagine the kind of person that goes out and builds something like this would have little patience with the bureaucracy of getting it integrated into .NET.
CharlieDigital · 5h ago
I'd say it's less about bureaucracy and more about what the .NET team has to consider when they make sweeping changes.
Backwards compatibility, security, edge cases, downstream effects on other libraries that are reliant on LINQ, etc.
One guy with an optional library can break things. If the .NET team breaks things in LINQ, it's going to be a bad, bad time for a lot of people.
I think Evan You's approach with Vue is really interesting. Effectively, they have set up a build pipeline that includes testing major downstream projects as well for compatibility. This means that when the Vue team build something like "Vapor Mode" for 3.6, they've already run it against a large body of community projects to check for breaking changes and edge cases. You can see some of the work they do in this video: https://www.youtube.com/watch?v=zvjOT7NHl4Q
akdev1l · 5h ago
I think this approach predates Vue.
I know of two examples:
1. Fedora in collaboration with GCC maintainers keep GCC on the bleeding edge so it can be used to compile the whole Fedora corpus. This validates the compiler against a set of packages which known to work with the previous GCC
2. I think the rust team also builds all crates on crates.io when working on `rustc`. It seems they created a tool to achieve that: https://github.com/rust-lang/crater
I would assume the .NET guys have something similar already but maybe there’s not enough open code to do that
zamalek · 5h ago
Rust also has the advantage of having no ABI. Binary interface is a whole lot more difficult to maintain than code interface.
C# has multiple technologies built to deal with ABI (though it probably all goes unused these days with folder-based deployments, you really need the GAC for it to work).
jasonjayr · 2h ago
IIRC perl tested new releases by running all the unit tests in the CPAN library, waaaaay back when.
clscott · 1h ago
They still do and investigate each failure. If the end result is that the library is “wrong” tickets and patches get sent to the library maintainers.
mrmedix · 5h ago
You have to add an extra function call at the start of the Linq method chain in order to make it zero-allocation. So I don't think it would break backwards compatibility. But adding it does create an additional maintenance burden.
qingcharles · 4h ago
From some experience, the MS guys are actually really eager to get more outside help and many will help guide you through the process if you have something to offer.
Every release has a fairly decent amount of fixes and additions from outside contributors, and while I can see a lot of to/fro on the PRs to get them through, it's probably not quite as bad as you'd expect.
jayd16 · 2h ago
Using reference types are more idiomatic in C#. To some degree they are less bug prone as well (they can be passed around without issue). Most of the core library use them instead of starting with value types and boxing.
The Task library has successfully added ValueTask but it took some doing. LINQ on the other hand can be replaced with unrolled loops or libraries more easily so the pressure just hasn't been there.
I could see something happening in the future but it would take a lot of be work.
From looking at the blog post I suspect the explosion of generic instances could be a serious problem for code size and startup time, but that's probably solvable somehow. The performance certainly seems impressive.
The way LINQ currently works by default makes aggressive use of interfaces like IEnumerable to hide the actual types being iterated over. This has performance consequences (which is part of why ZLinq can beat it) but it has advantages - for example, the same implementation of Where<T>(seq) can be used for various T's instead of having to JIT or AOT-compile a unique body for every distinct class you iterate over.
From looking at ZLinq it seems like it would potentially have an explosion of unique generic struct types as your queries get more complex, since for it to work you potentially end up with types vaguely resembling Query3<Query2<Query1<T>>>>. But it might not actually be that bad in practice.
Edit: what's also nice is that C# recognizes Linq as a contract. So long as this has the correct method names and signatures (it does), the Linq syntax will light up automatically. You can also use this trick for your own home-grown things (add Select, Join, Where, etc. overloads) if the Linq syntax is something you like.
[1]: https://learn.microsoft.com/en-us/dotnet/core/whats-new/dotn...
Some notes on why this is so here: https://github.com/dotnet/runtime/blob/main/docs/design/core...
The special syntax is really just syntactic sugar on top of all this that makes things a little bit more readable for complex queries because e.g. you don't have to repeat computed variables every time after binding it once in the chain. Consider:
If you were to rewrite this with explicit method calls and lambdas, it becomes something like: Note how it needs to weave `x` and `y` through all the Select/Where calls so that they can be used for ordering in the end here, whereas with syntactic sugar the scope of `let` extends to the remainder of the expression (although under the hood it still does roughly the same thing as the handwritten code).Something like dates.Where(static x => x.Date > DateTime.Now)
from x select x.name
And other is just lambda with anonymous types and so on.
For the lambda syntax, you can just do this: https://www.npmjs.com/package/linq
Of course, if you want to run this against a query provider, you do need compiler support to instead give you an expression tree, and provider to process it and convert them to a language (often sql) that database can understand.
There seems to be some transpilers, or things like that - but i don't know what the state of the art is on this: https://github.com/sinclairzx81/linqbox
Is this possible in JavaScript?
1. You can extend other people's interfaces. If you care about method chaining, _something_ like that is required (alternative solutions include syntactic support for method chaining as a generic function-call syntax).
2. The language has support for "code as data." The mechanism is expression trees, but it's really powerful to be able to use method chaining to define a computation and then dispatch it to different backends, complete with optimization steps.
3. The language has a sub-language as a form of syntactic sugar, allowing certain blessed constructs to be written as basically inline SQL with full editor support.
Compare C# ORMs to JS/TS for example. In C#, it is possible to use expression trees to build queries. In TS, the only options are as strings or using structural representation of the trees.
Compare this:
To the equivalent in Prisma (structural representation of the tree): Yikes! Look how dicey that gets with even a basic query!Adding another enumerable type would be a very large change that could effectively double the API surface of the entire ecosystem. This could take some time. Some places still don't even support Span<T>. Also there were some design decisions related to Linq where the number of overloads were a consideration.
Adding this API to .NET could probably be done with that extension method that converts to ValueEnumerable. But without support for that enumerable, this would pretty much be a walled garden where you have to convert back and forth between different enumerable types. Not that great if you'd ask me, but possible I guess.
Backwards compatibility, security, edge cases, downstream effects on other libraries that are reliant on LINQ, etc.
One guy with an optional library can break things. If the .NET team breaks things in LINQ, it's going to be a bad, bad time for a lot of people.
I think Evan You's approach with Vue is really interesting. Effectively, they have set up a build pipeline that includes testing major downstream projects as well for compatibility. This means that when the Vue team build something like "Vapor Mode" for 3.6, they've already run it against a large body of community projects to check for breaking changes and edge cases. You can see some of the work they do in this video: https://www.youtube.com/watch?v=zvjOT7NHl4Q
I know of two examples:
1. Fedora in collaboration with GCC maintainers keep GCC on the bleeding edge so it can be used to compile the whole Fedora corpus. This validates the compiler against a set of packages which known to work with the previous GCC
2. I think the rust team also builds all crates on crates.io when working on `rustc`. It seems they created a tool to achieve that: https://github.com/rust-lang/crater
I would assume the .NET guys have something similar already but maybe there’s not enough open code to do that
C# has multiple technologies built to deal with ABI (though it probably all goes unused these days with folder-based deployments, you really need the GAC for it to work).
Every release has a fairly decent amount of fixes and additions from outside contributors, and while I can see a lot of to/fro on the PRs to get them through, it's probably not quite as bad as you'd expect.
The Task library has successfully added ValueTask but it took some doing. LINQ on the other hand can be replaced with unrolled loops or libraries more easily so the pressure just hasn't been there.
I could see something happening in the future but it would take a lot of be work.
There's an official process for API change requests: https://github.com/dotnet/runtime/blob/main/docs/project/api...
The way LINQ currently works by default makes aggressive use of interfaces like IEnumerable to hide the actual types being iterated over. This has performance consequences (which is part of why ZLinq can beat it) but it has advantages - for example, the same implementation of Where<T>(seq) can be used for various T's instead of having to JIT or AOT-compile a unique body for every distinct class you iterate over.
From looking at ZLinq it seems like it would potentially have an explosion of unique generic struct types as your queries get more complex, since for it to work you potentially end up with types vaguely resembling Query3<Query2<Query1<T>>>>. But it might not actually be that bad in practice.