include without context doesn't work in macro

[dseomn]

When I use include ... without context inside a macro like this:

import jinja2

print(
    jinja2.Environment(
        loader=jinja2.DictLoader(
            {
                "included": "foo",
                "main": """
                    {%- macro foo() %}
                        {%- include "included" without context %}
                    {%- endmacro %}
                    {{- foo() -}}
                """,
            }
        ),
    )
    .get_template("main")
    .render()
)

The macro evaluates to something that looks like a repr() string instead of the contents of the included template:

$ python3 foo.py 
<generator object root.<locals>.macro at 0x7f79f60a0580>

The same include line works outside of a macro, and it works inside the macro if I remove without context:

$ python3 foo.py 
foo

Environment:

• Python version: 3.13.5
• Jinja version: 3.1.6-1

[davidism]

First step would be to compare the output from env.compile(..., raw=True) to see how the generated Python code differs, then figure out what the compiler is doing. It's complex, but I'm happy to answer questions or review a PR.

[dseomn]

It looks like the macro function is a generator in the without context case and a normal function in the other case. Totally guessing since I'm not familiar with the internal API at all, but should the first one return concat(template._get_default_module()._body_stream) instead of using yield from?

In [31]: print(jinja2.Environment().compile('{% macro foo() %}{% include "included" without context %}{% endmacro %}{{ foo() }}', raw=True))

from jinja2.runtime import LoopContext, Macro, Markup, Namespace, TemplateNotFound, TemplateReference, TemplateRuntimeError, Undefined, escape, identity, internalcode, markup_join, missing, str_join
name = None

def root(context, missing=missing, environment=environment):
    resolve = context.resolve_or_missing
    undefined = environment.undefined
    concat = environment.concat
    cond_expr_undefined = Undefined
    if 0: yield None
    l_0_foo = missing
    pass
    def macro():
        t_1 = []
        pass
        template = environment.get_template('included', None)
        yield from template._get_default_module()._body_stream
        return concat(t_1)
    context.exported_vars.add('foo')
    context.vars['foo'] = l_0_foo = Macro(environment, macro, 'foo', (), False, False, False, context.eval_ctx.autoescape)
    yield str(context.call((undefined(name='foo') if l_0_foo is missing else l_0_foo)))

blocks = {}
debug_info = '1=12'

In [32]: print(jinja2.Environment().compile('{% macro foo() %}{% include "included" %}{% endmacro %}{{ foo() }}', raw=True))

from jinja2.runtime import LoopContext, Macro, Markup, Namespace, TemplateNotFound, TemplateReference, TemplateRuntimeError, Undefined, escape, identity, internalcode, markup_join, missing, str_join
name = None

def root(context, missing=missing, environment=environment):
    resolve = context.resolve_or_missing
    undefined = environment.undefined
    concat = environment.concat
    cond_expr_undefined = Undefined
    if 0: yield None
    l_0_foo = missing
    pass
    def macro():
        t_1 = []
        pass
        template = environment.get_template('included', None)
        gen = template.root_render_func(template.new_context(context.get_all(), True, {'foo': l_0_foo}))
        try:
            for event in gen:
                t_1.append(event)
        finally: gen.close()
        return concat(t_1)
    context.exported_vars.add('foo')
    context.vars['foo'] = l_0_foo = Macro(environment, macro, 'foo', (), False, False, False, context.eval_ctx.autoescape)
    yield str(context.call((undefined(name='foo') if l_0_foo is missing else l_0_foo)))

blocks = {}
debug_info = '1=12'

P.S. I didn't know off the top of my head what returning a non-None value would do in a function that also had a yield from expression. To save anybody else time looking it up too, https://docs.python.org/3/reference/simple_stmts.html#the-return-statement says:

In a generator function, the return statement indicates that the generator is done and will cause StopIteration to be raised. The returned value (if any) is used as an argument to construct StopIteration and becomes the StopIteration.value attribute.

[dseomn]

And this is the relevant code, right?

jinja/src/jinja2/compiler.py

Lines 1068 to 1089 in 5ef7011

	if node.with_context:
	self.writeline(
	f"gen = template.root_render_func("
	"template.new_context(context.get_all(), True,"
	f" {self.dump_local_context(frame)}))"
	)
	self.writeline("try:")
	self.indent()
	self.writeline(f"{self.choose_async()}for event in gen:")
	loop_body()
	self.outdent()
	self.writeline(
	f"finally: {self.choose_async('await gen.aclose()', 'gen.close()')}"
	)
	elif self.environment.is_async:
	self.writeline(
	"for event in (await template._get_default_module_async())"
	"._body_stream:"
	)
	loop_body()
	else:
	self.writeline("yield from template._get_default_module()._body_stream")

[dseomn]

I just looked at the compiled code when there's anything else in the macro other than the include statement. It looks like that other stuff appends to t_1. So the yield from that the include statement generates turns the macro function into a generator, which completely changes the meaning of return concat(t_1) at the end, breaking everything else in the macro definition. I think the right solution is to do t_1.extend(template._get_default_module()._body_stream) then or just use a for loop like the other branches in visit_Include?