Slow startup time due to optional imports

[crest42]

Is your feature request related to a problem? Please describe.
I have an (admittedly odd) use-case, where I run a single script repeatedly, that executes a few queries and dump some data. The overall runtime of the script is rather high compared to the actual time spent executing the queries and dumping the data. (1.2s overall runtime, ~0.2s actual spent on work)

The reason for this are the imports in options.py. When running the following python snippet:

✗ time python -c "import clickhouse_connect"
python -c "import clickhouse_connect"  1.04s user 0.85s system 239% cpu 0.789 total

We see it takes about 0.8s wall clock time to run. If we remove the imports from options.py and replace them with e. g. np = None:

✗ time python -c "import clickhouse_connect"
python -c "import clickhouse_connect"  0.17s user 0.02s system 99% cpu 0.201 total

We see an improvement in startup to 0.2s (4x faster). Regarding CPU time, the improvement is even more significant with a speedup to ~0.2s from ~1.9s.

Those modules get imported independent of whether they ever used or not. Making those imports lazy, could reduce the overhead for use-cases like mine quite significant. Obviously, I could get the same results by just not installing the modules in the python environment in the first place but I need them in other places so this isn't a viable option for me.

I will happily provide a PR with the solution described below if you consider this an issue as well, but I wanted to check in before spending the time doing actual development.

Describe the solution you'd like
One solution to achieve lazy-loading could be to introduce a "LazyLoader" class in options.py. Instead of importing the modules directly with an import statement we could do something like:

class LazyLoader:
  def __getattr__(self, name):
    if not self.loaded:
      self.load()
    else:
      return __getattr__(self.module, name)

  def load(self):
    try:
      import self.module_name
    ...

np = LazyLoader('numpy')

The obvious benefit of such an implementation is that is would act as an drop-in replacement, where no other code changes would be required. The downside of this change is the additional complexity and the associated risks.

Describe alternatives you've considered
An alternative, often used in other libraries (And described here https://peps.python.org/pep-0810/) is to wrap the import to a function call, such as:

def np():
  try:
    import numpy as np
    return np
  except:
    ...

The benefit of this solution is that it does only add little complexity but all usages of np in clickhouse_connect would need to change from np.something to np().something.

Describe alternatives you've considered
There is also https://docs.python.org/3/library/importlib.html#importlib.util.LazyLoader which could do the trick, although further investigation would be needed if this is really applicable. If yes, I think this could be the prefered solution, providing a drop-in replacement with python stdlib functionality.

Additional context
None

[joe-clickhouse]

Hi @crest42, I'm curious to learn a bit more about your use case here. Is this like a lambda/serverless thing that gets called frequently? Part of a CLI tool? Or something else triggering it? Either way, even with the import issue solved, presumably you're grabbing a client at some point to do some database work? Client creation is a pretty heavy process itself, too. It makes a few calls to the server to get settings/version/timezone/etc. and then does quite a bit of other overhead just to get the client in a happy state ready for work. As I'm sure you know, this isn't the ideal or designed use case for clickhouse-connect so I'm not surprised it's slow. I'm not sure exactly what constraints you have, but I'd recommend keeping the process alive if at all possible. Perhaps building a simple server/daemon that accepts requests would help out a lot here.

All that said, I would have no objections to implementing lazy imports via the PEP 562 __getattr__ approach. This would be a single file change to options.py with zero changes needed elsewhere so if you're up it, go for it!

[crest42]

Hi @joe-clickhouse Thanks for the quick reply. To be quite frank its a pretty obvious architectural flaw on our side. The calling side and the called script actually reside on the same system, so a long lived process is likely our long term remediation.

What we do atm is to do some scheduled database queries every minute or so (Think cron). This alone wouldn't be an issue but we have to run multiple queries and each of those tasks are currently spawning a separate python process (Which is even more unnecessary overhead). Feel free to drop me a private message if you are interested in more details. Happy to share.

Fixing this architecture is our main goal but while we at it we figured this might be worthwhile for a quick relieve. I try to come up with a PR in the next days and you can let me know if that's what you had in mind. Appreciate the feedback!

Cheers,
Robin

[genzgd]

+1 for the lazy imports idea

[joe-clickhouse]

Thanks for the additional context @crest42! Again, no idea how much flexibility you have there, but for the short term if you're able to combine all the scheduled tasks into a single Python script that establishes one ClickHouse connection (or even a connection pool) and then just executes all queries sequentially, you'd likely see a big speed up.

Either way, the lazy loading idea is still a good one and would also probably help here significantly. Good luck! Happy to help if you have any other questions or anything.