HRT's Python fork: Leveraging PEP 690 for faster imports

Its a finance firm - i.e scam firm. "We have a fancy trading algorithm that statistically is never going to outperform just buying VOO and holding it, but the thing is if you get lucky, it could".

Scammers are not tech people. And its pretty from their post.

> In Python, imports occur at runtime. For each imported name, the interpreter must find, load, and evaluate the contents of a corresponding module. This process gets dramatically slower for large modules, modules on distributed file systems, modules with slow side-effects (code that runs during evaluation), modules with many transitive imports, and C/C++ extension modules with many library dependencies.

As they should.

The idea that when you type something in the code and then the interpreter just doesn't execute it is how you end up with Java like services, where you have dependency injection chains that are so massive that when the first time everything has to get lazily injected the code takes a massive amount of time to run. Then you have to go figure out where is the initialization code that slows everything down, and start figuring out how to modify your code to make that load first, which leads to a mess.

If your python module takes a long time to load, this is a module problem. There is a reason why you can import submodules of modules directly, and overall the __init__.py in the module shouldn't import all the submodules by default. Structure your modules so they don't do massive initialization routines and problem solved.

Furthermore, because of pythons dynamic nature, you can do run time imports, including imports in functions. In use, whether you import something up at the top and it gets lazily loaded or you import something right when you have to use it has absolutely no difference other than code syntax, and the latter is actually better because you can see what is going on rather than the lazy loading being hidden away in the interpreter.

Or if you really care, you can implement lazy work process inside the modules, so when you import them and use them the first time it works exactly like lazy imports.

To basically spend time building a new interpreter with lazy loading just to be able to have all your import statements up at the top just screams that those devs prefer ideology over practicality.

But yes, like you I had a great experience

Import-time side effects are definitely nasty though and I wonder what the implications on all downstream code would be. Perhaps a lazy import keyword is a better way forward.

A PEP is very much welcome, but using lazy import libraries is a fairly common, very old, method of speeding things up. My pre PEP 690 code looks like this:

    import typing
    from lazy import LazyImport

    member = LazyImport('my_module.subpackage', 'member')
    member1, member2, = LazyImport('my_module', 'member1', 'member2')
    
    if typing.TYPE_CHECKING:
        # normal import, for linter/IDE/navigation. 
        from my_module.subpackage import member
        from my_module import member1, member2

    import argparse
    parser = argparse.ArgumentParser()
    parser.parse_args()
    import requests

It's dirty, but speeds things up vs putting all imports at the top.

The lazy import approach was pioneered in Mercurial I believe, where it cut down startup times by 3x.