Behavior of generators being exhausted by multiple renders

[JakobGM]

Hi, there, @pelme and other htpythonistas!

I'm unsure if this should be considered a bug, intended behavior, or just a natural unavoidable consequence of generators being exhaustible, but I wanted to bring up a htpy rendering bug that took us a while to understand today. An htpy element had a child that was expressed as a generator, and the element was rendered twice, once for logging purposes (as a small isolated fragment) and once for rendering as part of the full-DOM response. The problem was that the second render did not contain the children, as the generator had been exhausted by the first render.

Here is a piece of simplified code that exemplifies this issue:

import htpy as h

# Construct element where a child is a generator
top_three = h.ul[(h.li[i] for i in range(1, 4))]

# First render behaves as expected
assert str(top_three) == "<ul><li>1</li><li>2</li><li>3</li></ul>"

# The second render is affected by the generator being exhausted
assert str(top_three) == "<ul></ul>"  # <-- Suceeds
assert str(top_three) == "<ul><li>1</li><li>2</li><li>3</li></ul>"  # <-- Fails

This is a consequence of the generator being exhausted, but I wonder if htpy wants to remove this foot gun in some way? Here are some different suggestions to start the discussion:

1. Persist the result of first-time iteration over exhaustible generators that can be re-used for subsequent iterations.
2. Allow an element to be evaluated and simultaneously frozen for those that want to opt-into a behavior where generators don't cause this issue. Either as a global option or as a utility function or method.
3. Raise an exception if a previously exhausted generator is re-used for subsequent render attempts.
4. Log a warning when re-used.
5. Make the reader aware of this potential footgun wherever generators are used in documentation.
6. Do nothing, this is expected of generators.

I have ordered my suggestions in my order of preference, but I'm interested in hearing what you have to say! Suggestion no. 1. has a memory cost, but in most, if not all, cases it is going to be negligible, I think.

[pelme]

Without thinking this through fully, I think I like 3 the most. If one use generators for streaming a huge page, it would be nice to avoid unnecessary memory allocations.

I think that either you really want one-off generators or you do not. If you get a clear exception if you accidentally reuse them you can a) avoid rendering twice or b) wrap your generator in list().

[JakobGM]

I see your point! Persisting the generator obviates some of the benefit of it being a generator, and list comprehensions will always be available for users to use.

I think (I also haven't thought through this fully, yet) that solution 3. would require us to implement one of the following strict conventions in our htpy-using code base:

1. Never ever use generators, or...
2. never trigger a rendering of a htpy element unless it is for the purposes of final response rendering.

The use of a generator expression could be quite far separated away from the potential re-use, as in this silly example where a warning should be displayed both before and after a list:

warning_message = warning_component(*args, **kwargs)
list_view = [
    warning_message,
    list_content,
    warning_message,  # <- Might be missing content if `warning_component()` uses a generator
]

A third option, of course, is having complete test coverage of all templating code such that these exceptions are raised in the test suite instead of in production.

[pelme]

In your example, yes, when warning_message gets rendered the second time it would then raise. If you are currently doing something like that you would not get a result you expect currently. In that case you could either call warning_component twice or make sure it does not use generators internally.

I think it is a choice/tradeoff: if you want the efficiency of generators - use that and deal with the downsides. Otherwise materialise to a list. The same problem exists in regular Python code too. I think using generators mostly is important when generating a response from a database result or a big file or something like that. In those cases you would be mindful about only consuming it once anyways.

Having tests is always a good idea! 😆

[JakobGM]

Makes sense, consider me convinced!

[pelme]

I opened an issue for this: #102