In render_pyramid.py I am creating one boto3 session per tile in order to handle the fact that boto3's session is not threadsafe. https://github.com/mojodna/marblecutter/blob/tilezen/examples/render_pyramid.py#L193
I was working on code to copy tiles out of a md5-hash prefixed bucket to a "regular" bucket and noticed that creating a session once per tile was eating up a lot of time, so instead I used a thread.local to store the boto session and initialized it once in the initializer for the Pool.
This took 75%+ off the time to do the simple copy operation. It might not take quite as much time off the pyramid renderer because there's more time spent on the actual rendering, but it would still be useful to do in the rendering scripts.
In
render_pyramid.pyI am creating one boto3 session per tile in order to handle the fact that boto3's session is not threadsafe. https://github.com/mojodna/marblecutter/blob/tilezen/examples/render_pyramid.py#L193I was working on code to copy tiles out of a md5-hash prefixed bucket to a "regular" bucket and noticed that creating a session once per tile was eating up a lot of time, so instead I used a
thread.localto store the boto session and initialized it once in theinitializerfor thePool.This took 75%+ off the time to do the simple copy operation. It might not take quite as much time off the pyramid renderer because there's more time spent on the actual rendering, but it would still be useful to do in the rendering scripts.