NamedTuple field as data

[ElOceanografo]

The design of my code in ProbabilisticEchoInversion.jl currently relies on a Turing model being able to take a NamedTuple as an argument, which contains the observed data in one of its fields. E.g. something like this:

using Turing

@model function test(data)
    m ~ Normal()
    data.x ~ Normal(m, 1.0)
end
data = (x = 1.0;)
sample(test(data), NUTS(), 100)

This pattern worked until recently, but starting with Turing v0.41.0, it started throwing the following error:

ERROR: setfield!: immutable struct of type NamedTuple cannot be changed
Stacktrace:
  [1] setproperty!(x::@NamedTuple{x::Float64}, f::Symbol, v::Float64)
    @ Base ./Base.jl:53
  [2] macro expansion
    @ ~/.julia/packages/DynamicPPL/ze1aD/src/compiler.jl:573 [inlined]
  [3] test1(__model__::DynamicPPL.Model{…}, __varinfo__::DynamicPPL.VarInfo{…}, data::@NamedTuple{…})
    @ Main ./REPL[5]:3
  [4] _evaluate!!(model::DynamicPPL.Model{…}, varinfo::DynamicPPL.VarInfo{…})
    @ DynamicPPL ~/.julia/packages/DynamicPPL/ze1aD/src/model.jl:974
  [5] evaluate_threadunsafe!!(model::DynamicPPL.Model{…}, varinfo::DynamicPPL.VarInfo{…})
    @ DynamicPPL ~/.julia/packages/DynamicPPL/ze1aD/src/model.jl:940
  [6] check_model_and_trace(model::DynamicPPL.Model{…}, varinfo::DynamicPPL.VarInfo{…}; error_on_failure::Bool)
    @ DynamicPPL.DebugUtils ~/.julia/packages/DynamicPPL/ze1aD/src/debug_utils.jl:428
  [7] check_model(model::DynamicPPL.Model{…}, varinfo::DynamicPPL.VarInfo{…}; error_on_failure::Bool)
    @ DynamicPPL.DebugUtils ~/.julia/packages/DynamicPPL/ze1aD/src/debug_utils.jl:451
  [8] _check_model(model::DynamicPPL.Model{typeof(test1), (:data,), (), (), Tuple{…}, Tuple{}, DynamicPPL.DefaultContext})
    @ Turing.Inference ~/.julia/packages/Turing/ObWSF/src/mcmc/abstractmcmc.jl:5
  [9] _check_model(model::DynamicPPL.Model{…}, ::NUTS{…})
    @ Turing.Inference ~/.julia/packages/Turing/ObWSF/src/mcmc/abstractmcmc.jl:8
 [10] sample(rng::Random.TaskLocalRNG, model::DynamicPPL.Model{…}, sampler::NUTS{…}, N::Int64; check_model::Bool, chain_type::Type, initial_params::InitFromUniform{…}, initial_state::Nothing, progress::Bool, nadapts::Int64, discard_adapt::Bool, discard_initial::Int64, kwargs::@Kwargs{})
    @ Turing.Inference ~/.julia/packages/Turing/ObWSF/src/mcmc/hmc.jl:101
 [11] sample(rng::Random.TaskLocalRNG, model::DynamicPPL.Model{…}, sampler::NUTS{…}, N::Int64)
    @ Turing.Inference ~/.julia/packages/Turing/ObWSF/src/mcmc/hmc.jl:86
 [12] sample(model::DynamicPPL.Model{…}, spl::NUTS{…}, N::Int64; kwargs::@Kwargs{})
    @ Turing.Inference ~/.julia/packages/Turing/ObWSF/src/mcmc/abstractmcmc.jl:71
 [13] sample(model::DynamicPPL.Model{…}, spl::NUTS{…}, N::Int64)
    @ Turing.Inference ~/.julia/packages/Turing/ObWSF/src/mcmc/abstractmcmc.jl:68
 [14] top-level scope
    @ REPL[7]:1
Some type information was truncated. Use `show(err)` to see complete types.

Wondering if this change intentional, or required by some other change, or if it could be considered a bug?

[penelopeysm]

This is probably a bug! I'll transfer to DynamicPPL, but I won't be able to look at it for a while (someone else might).

[penelopeysm]

DPPL only repro:

using DynamicPPL, Distributions
@model function test(data)
    m ~ Normal()
    data.x ~ Normal(m, 1.0)
end
data = (;x = 1.0)
DynamicPPL.init!!(test(data), VarInfo())
#=
ERROR: setfield!: immutable struct of type NamedTuple cannot be changed
Stacktrace:
 [1] setproperty!(x::@NamedTuple{x::Float64}, f::Symbol, v::Float64)
   @ Base ./Base.jl:53
 [2] test(__model__::Model{…}, __varinfo__::VarInfo{…}, data::@NamedTuple{…})
   @ Main ./REPL[13]:3
[...]
=#

[penelopeysm]

Oh, I'm pretty sure it's because of #1110, which generates output that looks like

data.x = if (DynamicPPL.inargnames)(var"##vn#270", __model__)
                            data.x
                        else
                            (DynamicPPL.getconditioned_nested)(__model__.context, (prefix)(__model__.context, var"##vn#270"))
                        end
(var"##value#272", __varinfo__) = (tilde_observe!!)(__model__.context, (DynamicPPL.check_tilde_rhs)(var"##dist#273"), data.x, var"##vn#270", __varinfo__)

Almost certainly what needs to happen is to create a gensym'd variable on the LHS instead of directly using the LHS expression.

[penelopeysm]

There's also a different line of compiler output that looks dodgy:

data.x = (DynamicPPL.getfixed_nested)(__model__.context, (prefix)(__model__.context, var"##vn#270"))

I reckon that might also need to be wrapped in some kind of BangBang.prefermutation wrapper (there's an example somewhere in src/compiler.jl).

Example of failure mode:

using DynamicPPL, Distributions
@model function test(data)
    m ~ Normal()
    data.x ~ Normal(m, 1.0)
end
data = (;x = 1.0)
fixm = fix(test(data), (; data=(; x = 5.0)))
DynamicPPL.init!!(fixm, VarInfo())
#=
ERROR: setfield!: immutable struct of type NamedTuple cannot be changed
Stacktrace:
 [1] setproperty!(x::@NamedTuple{x::Float64}, f::Symbol, v::Float64)
   @ Base ./Base.jl:53
=#

[penelopeysm]

@ElOceanografo Is it important for you that it works on Turing 0.41, or is it OK to skip straight to 0.42?