A deep dive into Rust and C memory interoperability

Miri and Valgrind will usually catch this kind of issue. I did lots of work mixing C and Rust and that tripped me as well at the beginning. I wrote about it if someone is interested: https://gaultier.github.io/blog/perhaps_rust_needs_defer.htm...

It's funny. When I first tried Rust in 2018 they were still statically linking jemalloc into every binary rustc compiled by default, and that alone very much put me off of the language for a while.

Apparently they did away with jemalloc in favor of the system allocator that same year but nonetheless when I came back to it years later I was very happy to learn of its removal.

Rust bindgen[1] will automatically generate native Rust stucts (and unions) from C headers where possible. Note that c_int, c_char, etc. are just aliases for the corresponding native Rust types.

However, not all C constructs have idomatic Rust equivalents. For example, bitfields don't exist in Rust, and unlike Rust enums, C enums can have any value of the underlying type. And for ABI reasons, it's very commom in C APIs to use a pointer to an opaque type paired with what are effectively accessor function and methods, so mapping them to accessors and methods on a "Handle" type in Rust often is the most idomatic Rust representation of the C interface.

[1]: https://github.com/rust-lang/rust-bindgen

Here's one of my recorded talks going through an example of using a `#[repr(C)]` struct (in this case one that's auto-generated by Bindgen): https://youtu.be/LLAUzghhNHg?t=2168

I wrote a blog article about exactly this, using a C++ class from C++, C and Rust: https://gaultier.github.io/blog/rust_c++_interop_trick.html

It's idiomatic to access struct fields directly in both languages. What more do you want?

Like, repr(C)?

https://doc.rust-lang.org/nomicon/other-reprs.html

I don't know what examples you've been seeing. The interop structs are just regular Rust structs with the `#[repr(C)]` attribute applied to them, to ensure that the Rust compiler lays the struct out exactly as the C compiler for that target ABI would. Rust code can access their fields just fine. There's no strict need for accessor functions.

>But with the upcoming support for passing an allocator to any data structure (in the Rust standard library anyway) I think this gets a lot easier?

Yes and no. Even within libstd, some things require A=GlobalAlloc, eg `std::io::Read::read_to_end(&mut Vec<u8>)` will only accept Vec<u8, GlobalAlloc>. It cannot be changed to work with Vec<u8, A> because that change would make it not dyn-compatible (nee "object-safe").

And as you said it will cut you off from much of the third-party crates ecosystem that also assumes A=GlobalAlloc.

But if the subset of libstd you need supports A=!GlobalAlloc then yes it's helpful.

For me the most interesting thing in Allocator is that it's allowed to say OK, you wanted 185 bytes but I only have a 256 byte allocation here, so, here is 256 bytes.

This means that e.g. a growable container type doesn't have to guess that your allocator probably loves powers of 2 and so it should try growing to 256 bytes not 185 bytes, it can ask for 185 bytes, get 256 and then pass that on to the user. Significant performance is left on the table when everybody is guessing and can't pass on what they know due to ABI limitations.

Rust containers such as Vec are already prepared to do this - for example Vec::reserve_exact does not promise you're getting exactly the capacity you asked for, it won't do the exponential growth trick because (unlike Vec::reserve) you've promised you don't want that, but it would be able to take advantage of a larger capacity provided by the allocator.

I’m not sure what those two things have to do with each other, though I did just wake up. The only thing the new allocator stuff would give you is the ability to allocate a standard library data structure with the Postgres allocator. Scoping and handles and such wouldn’t change, and using your own data structures wouldn’t change.

It’s also very possible I’m missing something!

That's the first thing on the page.

> Interviewer: “What happens if you allocate memory with C’s malloc and try to free it with Rust’s dealloc, if you get a pointer to the memory from C?”

> Me: “If we do it via FFI then there’s a possibility the program may continue working (because the underlying structs share the same memory layout? right? …right?)”

It is, but yeah, the entire article's formatting is pretty weird.

I don't agree with your contemptuous framing. It's incorrect, and per the post's author, "dangerous" — but depending on your background it's not "silly" or "borderline insane". It's just naive, and writing a slab allocator as an exercise or making honest explorations like in this blog post will help cure the naivete.

The article is about how and why mixing allocators fails, not if it fails or how to fix the problem.

usually when interfacing with a library written in c the library will export functions for object destruction. it makes sense for that to be part of the interface instead of using the system allocator because it also gives the library freedom to do extra work during object destruction. if you have simple objects then its possible to just use the system allocator, but if you have graphs or trees of objects then its necessary to have a custom destroy function and there is always some risk in the future you might be forced to need to allocate more complex data structures that require multiple allocations.

it sounds nothing like AI to me! or AI has advanced to the point where it is hard to tell - e.g. I wouldn't expect a sentence like "You’re not just getting 64 bytes of memory. You’re entering into a complex contract with a specific allocator implementation." from one.

I find it hard to believe that an LLM would have come up with this quote to start the article:

> “Memory oppresses me.” - Severian, The Book of the New Sun

That sort of artistic/humourous flourish isn't in character for an LLM.

Do you see Emojis in tables/code now and assume the person is using an llm? I dont really see it.