Low-Level Optimization with Zig

Take something as simple as a vector (eg. std::vector in C++). If a user directly sets the size or capacity, the calls to methods like push_back() will behave incorrectly, or may even crash.

Opaque pointers are one way of hiding representation, but they also eliminate the possibility of inlining, unless LTO is in use. If you have members that need to be accessible in inline functions, it's impossible to use opaque pointers.

There is certainly a risk of "implicit interfaces" (Hyrum's Law), where users break even when you're changing the internals, but we can lessen the risk by encapsulating data structures as much as possible. There are other strategies for lessening this risk, like randomizing unspecified behaviors, so that people cannot take dependencies on behaviors that are not guaranteed.

Very similar experience here. Also just recently I really _had_ to use and extend the "internal" part of a legacy library. So potentially days or more than a week of work turned into a couple of hours.

I write a lot of rust. By default, fields are private. It's rare to see a field in my code that omits the `pub` prefix. I sometimes start with private because I forget `pub`, but inevitably I need to make it public!

I like in principle they're there, but in practice, `pub` feels like syntactic clutter, because it's on all my fields! I think this is because I use structs as abstract bags of data, vice patterns with getters/setters.

When using libraries that rely on private fields, I sometimes have to fork them so I can get at the data. If they do provide a way, it makes the (auto-generated) docs less usable than if the fields were public.

I suspect this might come down to the perspective of application/firmware development vice lib development. The few times I do use private fields have been in libs. E.g. if you have matrix you generate from pub fields and similar.

Then I noticed people were using them, preventing me from making important changes. So I created a pseudo-"private" facility using macros, where people had to write FOOLIB_PRIVATE(var) to get at the internal var.

Then I noticed (I kid you not) people started writing FOOLIB_PRIVATE(var) in their own code. Completely circumventing my attempt to hide these internal members. And I can't entirely blame them, they were trying to get something done, and they felt it was the fastest way to do it.

After this experience, I consider it an absolute requirement to have a real "private" struct member facility in a language.

I respect Andrew and I think he's done a hell of a job with Zig. I also understand the concern with the Java precedent and lots of wordy getters/setters around trivial variables. But I feel like Rust (and even C++) is a great counterexample that private struct variables can be done in a reasonable way. Most of the time there's no need to have getters/setters for every individual struct member.

My experience is that users who are trying to get work done will bypass every speed bump you put in the way and just access your internals directly.

If you "just" rely on them not to do that, then your internals will effectively be frozen forever.

It does feel regressive to me. I've seen people easily reach for underscored fields in Python. We can discourage them if the code is reviewed, but then again there's also people who write everything without underscores.

There are lots of use cases for this exact pattern. An acceleration structure to speed up searching complex geometry. The internal state of a streaming parser. A lazy cache of an expensive property that has a convenient accessor. An unsafe pointer that the struct provides consistent, threadsafe access patterns for. I've used this pattern for all these things, and there are many more uses for encapsulation. It's not just an OO concern.

C++ has the PassKey idiom that allows you to whitelist what objects are allowed to access each part of the public API at compile-time. This is a significant improvement but a pain to manage for complex whitelists because the language wasn't designed with this in mind. C++26 has added language features specifically to make this idiom scale more naturally.

I'd love to see more explicit ACLs on APIs as a general programming language feature.

If you can trust that downstream users of your api won’t misuse private-by-convention fields (or won’t punish you for doing so), it’s not a problem. That works a lot of the time: You can trust yourself. You can usually your team. In the opensource world, you can just break compatibility with no repercussions.

But yes, sometimes that trust isn’t there. Sometimes you have customers who will misuse your code and blame you for it. But that isn’t the case for all code. Or even most code.

Can’t you do this in Zig with modules? I thought that’s what the ‘pub’ keyword was for.

You can’t have private fields in a struct that’s publicly available but the same is sort of true in C too. OO style encapsulation isn’t the only way to skin a cat, or to send the cat a message to skin itself as the case may be.

I just fundamentally disagree with this. Not having "proper" private methods/members has not once become a problem for me, but overuse of them absolutely has.

Using indices isn't bad for performance. At the very least, it can massively cut down on memory usage (which is in turn good for performance) if you can use 16-bit or 32-bit indices instead of full 64-bit pointers.

> "unsafe" (which is strictly more dangerous than even C, never mind Zig)

Unsafe Rust is much safer than C.

The only way I can imagine unsafe Rust being more dangerous than C is that you need to keep exception safety in mind in Rust, but not in C.

If you need to make everything in-house this is the experience. For the majority though, the moment you require those things you reach for a crate that solves those problems.

No it's not? The Rust burrow checker, the backbone of Rust's memory safety model, doesn't stop working when you drop into an unsafe block. From the Rust Book:

>To switch to unsafe Rust, use the unsafe keyword and then start a new block that holds the unsafe code. You can take five actions in unsafe Rust that you can’t in safe Rust, which we call unsafe superpowers. Those superpowers include the ability to:

    Dereference a raw pointer
    Call an unsafe function or method
    Access or modify a mutable static variable
    Implement an unsafe trait
    Access fields of a union

For those saying unsafe Rust is strictly safer than C, you're overlooking Rust's extremely strict invariants that users must uphold. These are much stricter than C, and they're extremely easy to accidentally break in unsafe Rust. Breaking them in unsafe Rust is instant UB, even before leaving the unsafe context.

This article has a decent summary in this particular section: https://zackoverflow.dev/writing/unsafe-rust-vs-zig/#unsafe-...

Um, what? Unsafe Rust code still has a lot more safety checks applied than C.

>It’s important to understand that unsafe doesn’t turn off the borrow checker or disable any of Rust’s other safety checks: if you use a reference in unsafe code, it will still be checked. The unsafe keyword only gives you access to these five features that are then not checked by the compiler for memory safety. You’ll still get some degree of safety inside of an unsafe block.

https://doc.rust-lang.org/book/ch20-01-unsafe-rust.html

Laughs in graphics programmer. You end up using indices to track data in buffers all the time when working with graphics APIs.

Unsafe rust still has a bunch of checks C doesn’t have, and using indices into vectors is common code in high performance code (including Zig!)

Rust is a general purpose language that can do systems programming. Zig is a systems programming language.

(Safety Coomers please don't downvote)

It is no accident that there is ongoing discussion that clang should get its own IR, just like it happens with the other frontends, instead of spewing LLVM IR directly into the next phase.

> I haven't really understood the advantage compared to simply running a program at build time.

Since both comptime and runtime code can be mixed, this gives you a lot of safety and control. The comptime in zig emulates the target architecture, this makes things like cross-compilation simply work. For program that generates code, you have to run that generator on the system that's compiling and the generator program itself has to be aware the target it's generating code for.

The core problem is that you're changing the semantics of that integer as you change types, and if that happens automatically then the compiler can't protect you from typos, vibe-coded defects, or any of the other ways kids are generating almost-correct code nowadays. You can mitigate that with other coding patterns (like requiring type parameters in any potentially unsafe arithmetic helper functions and banning builtins which aren't wrapped that way), but under the swiss cheese model of error handling it still massively increases your risky surface area.

The issue is more obvious on the input side of that expression and with a different mask. E.g.:

  const a: u64 = 42314;
  const even_mask: u4 = 0b0101;
  a & even_mask;

What if `even_mask` is instead a comptime_int? You still have the same issue. That was a poor use of comptime ints since now that implicit conversion will always happen, and you lost your compiler errors when you misuse that constant.

Back to your proposal of something that should always be safe: implicitly lowering `a & 15` to a u4. The danger is in using it outside its intended context, and given that we're working with primitive integers you'll likely have a lot of functions floating around capable of handling the result incorrectly, so you really want to at least use the _right_ integer type to have a little type safety for the problem.

For a concrete example, code like that (able to be implicitly lowered because of information obvious to the compiler) is often used in fixed-point libraries. The fixed-point library though does those sorts of operations with the express purpose of having zeroed bits in a wide type to be able to execute operations without loss of precision (the choice of what to do for the final coalescing of those operations when precision is lost being a meaningful design choice, but it's irrelevant right this second). If you're about to do any nontrivial arithmetic on the result of that masking, you don't want to accidentally put it in a helper function with a u4 argument, but with implicit lowering that's something that has no guardrails. It requires the programmer to make zero mistakes.

That example might seem a little contrived, and this isn't something you'll run into every day, but every nontrivial project I've worked on has had _something_ like that, where implicit narrowing is extremely dangerous and also extremely easy to accidentally do.

What about the verbosity? IMO the point of verbosity is to draw your attention to code that you should be paying attention to. If you're in a module where implicit casting would be totally fine, then make a local helper function with a short name to do the thing you want. Having an unsafe thing be noisy by default feels about right though.

While Zig fixes some of these issues, the amount of @ feels like being back in Objective-C land and yeah too many uses of dot and starts.

Then again, I am one of those that actually enjoys using C++, despite all its warts and the ways of WG21 nowadays.

I also dislike the approach with source code only libraries and how importing them feels like being back in JavaScript CommonJS land.

Odin and C3 look interesting, the issue is always what is going to be the killer project, that makes reaching for those alternatives unavoidable.

I might not be a language XYZ cheerleeder, but occasionally do have to just get my hands dirty and do the needfull for an happy customer, regardlees of my point of view on XYZ.

"." = the "namespace" (in this case an enum) is implied, i.e. the compiler can derive it from the function signature / type.

"@" = a language built-in.

Basically it’s a better C/C++ for me and sits between C and C++ in complexity.

I tried Zig for a while years ago but found the casts and other annotations frustrating. And at the time the language was pretty unstable in how it applied those rules. Plus I’ve never found a use for custom allocators, even on embedded.

as for the dots, if you use zig quite a bit you'll see that dot usage is incredibly consistent, and not having the dots will feel wrong, not just in an "I'm used to it sense/stockholm syndrome" but you will feel for example that C is wrong for not having them.

for example, the use of dot to signify "anonymous" for a struct literal. why doesn't C have this? the compiler must make a "contentious" choice if something is a block or a literal. by contentious i mean the compiler knows what its doing but a quick edit might easily make you do something unexpected

It's an interesting question about how Zig will handle additional builtins and data representations. The current way I understand it is that there's an additional opt-in translation layer that converts unsupported/complicated IR to IR which the backend can handle. This is referred to as the compiler's "Legalize" stage. It should help to reduce this issue, and perhaps even make backends like https://github.com/xoreaxeaxeax/movfuscator possible :)

    unmarshaling struct type Foo: in field Bar int: failed to parse value "abc" as integer