> Normally, debuing the compiler is fairly straightforward: it is more or less a run of the mill executable.
> In the bootstrap process, the entire thing becomes way more complex. You see, rustc is not invoked directly. The bootstrap script calls a wrapper around the compiler.
> Running that wrapped rustc is not easy to run either: it requires a whole lot of complex, environment flags to be set.
> All that is to say: I don’t know how to debug the Rust compiler. I am 99.9 % sure there is an easy way to do this, documented somewhere I did not think to look. After I post this, somebody will tell me "oh, you just need to do X".
> Still, at the time of writing, I did not know how to do this.
> So, can we attach gdb to the running process? Nope, it crashes way to quickly for that.
It's kind of funny how often this problem crops up and the variety of tricks I have in my back to deal with it. Sometimes I patch the script to invoke gdb --args [the original command] instead, but this is only really worthwhile if it's a simple shell script and also I can track where stdin/stdout are going. Otherwise I might patch the code to sleep a bit before actually running anything to give me a chance to attach GDB. On some platforms you can get notified of process execs and sometimes even intercept that (e.g. as an EDR solution) and sometimes I will use that to suspend the process before it gets a chance to launch. But I kind of wish there was a better way to do this in general…LLDB has a "wait for launch" flag but it just spins in a loop waiting for new processes and it can't catch anything that dies too early.
jcranmer · 1h ago
I have a LD_PRELOAD library that hooks SIGSEGV into spawning gdb on the process using the best guess for the process's terminal (which currently isn't very smart because I haven't yet needed to debug processes that do a lot of stdio redirection).
o11c · 2h ago
Other ideas:
* Run the whole tree of processes under `gdb` with `set detach-on-fork off`.
* LD_PRELOAD a library that inserts the sleeps for you, maybe on startup or maybe on signal/exit.
Ideally, we'd have some kind of infrastructure to name and identify particular processes recursively.
dwheeler · 4h ago
It may not seem like it, but this is impressive progress. Getting a compiler to bootstrap at all is an accomplishment, especially for Rust since that depends on so many things working. Once it can reliably bootstrap, a lot of performance-improving steps can begin. Congrats!
Cogito · 2h ago
Really great read.
Someone mentioned recently that the slowness of rustc is in large part due to llvm. I know that is probably orthogonal to the work here, but I do like the idea of building the compiler with different toolchains, and that there may be follow on effects down the line.
JoshTriplett · 41m ago
Depends on the workload, but yes, codegen is a huge part of the total compilation time.
That said, that doesn't mean LLVM is always where the fixes need to be. For instance, one reason rustc spends a lot of time in LLVM is that rustc feeds more code to LLVM than it should, and relies on the LLVM optimizer to improve it. Over time, we're getting better about how much code we throw at LLVM, and that's providing performance improvements.
torstenvl · 43m ago
It's slow because the borrow checker is NP complete. LLVM may or may not generate slower code than GCC would for rustc, but I doubt it's anywhere close to the primary cause of the lack of snappy.
almostgotcaught · 30m ago
You're wrong it's been debunked that the borrow checker is any appreciable part of the compile time - Steve Klabnik actually verified it on here somewhere.
> In the bootstrap process, the entire thing becomes way more complex. You see, rustc is not invoked directly. The bootstrap script calls a wrapper around the compiler.
> Running that wrapped rustc is not easy to run either: it requires a whole lot of complex, environment flags to be set.
> All that is to say: I don’t know how to debug the Rust compiler. I am 99.9 % sure there is an easy way to do this, documented somewhere I did not think to look. After I post this, somebody will tell me "oh, you just need to do X".
> Still, at the time of writing, I did not know how to do this.
> So, can we attach gdb to the running process? Nope, it crashes way to quickly for that.
It's kind of funny how often this problem crops up and the variety of tricks I have in my back to deal with it. Sometimes I patch the script to invoke gdb --args [the original command] instead, but this is only really worthwhile if it's a simple shell script and also I can track where stdin/stdout are going. Otherwise I might patch the code to sleep a bit before actually running anything to give me a chance to attach GDB. On some platforms you can get notified of process execs and sometimes even intercept that (e.g. as an EDR solution) and sometimes I will use that to suspend the process before it gets a chance to launch. But I kind of wish there was a better way to do this in general…LLDB has a "wait for launch" flag but it just spins in a loop waiting for new processes and it can't catch anything that dies too early.
* Run the whole tree of processes under `gdb` with `set detach-on-fork off`.
* LD_PRELOAD a library that inserts the sleeps for you, maybe on startup or maybe on signal/exit.
Ideally, we'd have some kind of infrastructure to name and identify particular processes recursively.
Someone mentioned recently that the slowness of rustc is in large part due to llvm. I know that is probably orthogonal to the work here, but I do like the idea of building the compiler with different toolchains, and that there may be follow on effects down the line.
That said, that doesn't mean LLVM is always where the fixes need to be. For instance, one reason rustc spends a lot of time in LLVM is that rustc feeds more code to LLVM than it should, and relies on the LLVM optimizer to improve it. Over time, we're getting better about how much code we throw at LLVM, and that's providing performance improvements.
Edit: found it
https://news.ycombinator.com/item?id=44391240