Streaming responses

[lbam]

In this line...

django-vectortiles/vectortiles/postgis/mixins.py

Line 58 in 86f0087

return row.tobytes() if row else b""

the entire MVT response from PostGIS is converted to a bytes object. That means it has to be in memory twice, once as a memoryview and once as a bytes. I'm wondering if this is necessary, since Django performs the same conversion. If the conversion were not there, a caller could intercept the memoryview and make a StreamingHttpResponse out of it, so that it only needs to be in memory once.

[StefanBrand]

Good point, it looks as if the tobytes call could possibly be removed. I quickly ran the tests with this change and they run fine. Are you interested in contributing a PR?

[lbam]

I could discuss a PR with my employer, but I'd first need to have #36 resolved.

[lbam]

I've also done some more research and found that this is not the bottleneck, memory-wise. The tile is sent over the wire by PostgreSQL as ASCII hex. psycopg reads the entire tile into memory in that format, then decodes it into a separate buffer, so at that point the memory use is 3x the size of the tile. Fixing this issue would therefore not reduce the peak memory use. For that, the ST_AsMVT query would somehow have to chunk the resulting bytea in PostgreSQL so this library can receive it piecemeal and sent the chunks to the client in a streaming response.

[submarcos]

I could discuss a PR with my employer, but I'd first need to have #36 resolved.

This point is resolved. I'm working on a 1.0 refactor and release. If you contribute a PR I can implement it in 0.2.1 and 1.0 versions

[pstokkink]

Hi, I am working for the same organization as OP used to work for. He's no longer involved, but I wanted to gauge whether you would still be open to a PR that would allow a StreamingMVTView, with StreamingVectorLayers that return chunked mvt data? We have a solution that uses the following SQL snippet as the streaming equivalent of VectorLayer.get_tile:

CHUNK_SIZE = 8192
cursor.execute(
    f"""
    WITH mvt AS (
        SELECT ST_AsMVT(_sub.*, %s, %s, %s) FROM ({sql}) as _sub
    )
    SELECT * FROM (
        /* This is SQL for range(1, len(x), CHUNK_SIZE), sort of. */
        WITH RECURSIVE chunk(i) AS (
                VALUES (1)
            UNION ALL
                SELECT chunk.i+{CHUNK_SIZE} FROM chunk, mvt
                WHERE i < octet_length(mvt.ST_AsMVT)
        ),
        sorted AS (SELECT * FROM chunk ORDER BY i)
        SELECT substring(mvt.ST_AsMVT FROM sorted.i FOR {CHUNK_SIZE}) FROM mvt, sorted
    ) AS chunked_mvt WHERE octet_length(substring) > 0
    """, 
    params=[
      self.get_id(),
      self.tile_extent,
      "geom_prepared",
      *params
    ],
)
for chunk in cursor:
    yield chunk[0]

Please let me know if this is indeed an idea that would be interesting for you to incorporate into this repo. Would be happy to work on a PR.