HRT's Python fork: Leveraging PEP 690 for faster imports

I wonder how much can be saved by using a local file system for imports though. In my testing just a mere presense of a home directory on NFS already dramatically slows down imports (by ~10x) due to Python searching for modules in home directory too by default.

Interviewed with HRT awhile back. While I didn't get past the final round, their Python internals interview (which I did pass) was an absolute blast to prepare for, and required a really deep dive into implementation specific details of CPython around things like exactly how collisions are handled in dict, details about memory management, etc. Pretty much had to spend a few weeks in the CPython source to prep, and was, for me, worth the interview just to really learn what's going on.

For most teams I would be pretty skeptical of a internal Python fork, but the Python devs at HRT really know their stuff.

It'd have been really nice to have that PEP in as it'd have helped me not have to write local imports everywhere.

As it is, top-level imports IMHO are only meant to be used for modules required to be used in the startup, everything else should be a local import -- getting everyone convinced of that is the main issue though as it really goes against the regular coding of most Python modules (but the time saved to start up apps I work on does definitely make it worth it).

The author interviewed me and talked about this project, so it was cool seeing a blog post posted about it

I really really want lazy imports in Python, it's would be a godsend for CLIs

I know few modules that can take seconds to import but would have been nice to hear how much they actually gained?

Also maybe, if this approach could yield stats on if some import was needed or not ?

I thought HRT was a Cpp shop? Is Python used in their main business applications, or more for quants / data scientists?

> This process gets dramatically slower for … modules on distributed file systems, modules with slow side-effects

Oh no. Look I'm not saying you're holding it wrong, it's perfectly valid to host your modules on what is presumably NFS as well as having modules with side effects but what if you didn't.

I've been down this road with NFS (and SMB if it matters) and pain is the only thing that awaits you. It seems like they're feeling it. Storing what is spiritually executable code on shared storage was a never ending source of bugs and mysterious performance issues.

Gonna call this an antipattern. Do you need all those modules imported in every script ? Well then you save nothing on loadup time, the time will be spent regardless. Does every script not need those imports ? Well they shouldn't be importing those things and this small set of top level imports should be curated into a better, more fine grained list (and if you want to write tools, you can certainly identify these patterns using tooling similar to that which you wrote for LazyImports).

Oh, it's a trading firm. That's why they can fund an internal fork of Python... That sounds nice...

Imagine if these guys put their intelligence towards improving the world.