Why can't NormalMixture node accept fixed mean/precision parameters?

[meditans]

When using the NormalMixture node, the means m and precisions p must be sampled from distributions rather than specified as fixed values:

# We have to initialize every element of m and p from a distribution
m[1] ~ NormalMeanPrecision(...) # etc...
p[1] ~ GammaShapeRate(...) # etc...
x ~ NormalMixture(switch=z, m=m, p=p)

# This doesn't work - why not?
m = [0.0, 5.0]  # fixed means
p = [1.0, 1.0]  # fixed precisions
x ~ NormalMixture(switch=z, m=m, p=p)

I understand I can use very tight priors to approximate fixed values, but since the inference is still computing posteriors for the switch variables z, why is it necessary for m and p to be probability distributions rather than fixed constants?

I'll give a bit of context with an example. Suppose I'm trying to estimate a 1D position x, via a general x_t = x_{t-1} + delta * v schema. Each position on the 1D line has a feature vector. I have some beacons on the 1D line (I know perfectly the position). I observe the feature vector of the position I'm really in, and I know how similar it is to each feature vector of the beacons. I want to model the fact that if the feature is very similar to the one at beacon i, I probably am very close to beacon i, and so it seemed to me that placing a normal at each beacon position and doing a NormalMixture would be what I need. Am I way off topic?

[bvdmitri]

It should be possible imo, there is no particular reason rather than the rules are written such that they only accept distributions. But we should be able to extend it to accept fixed parameters.

[ThijsJenneskens]

I took a look and I think I can get your example to work. I define this model:

@model function inference(x, m, p)
       z ~ Bernoulli(0.5)
       x ~ NormalMixture(switch = z, m = m, p = p)
end

Then if I try to run inference by calling

results = infer(
       model = inference(m = [0.0, 5.0], p = [1.0, 1.0]),
       data = (x = 1.0,)
)

I get "ERROR: The number of means and precisions in NormalMixture must be at least 2. Got 1 means and 1 precisions." If I change the brackets of m and p to round brackets, so that they are given as tuples:

results = infer(
       model = inference(m = (0.0, 5.0), p = (1.0, 1.0)),
       data = (x = 1.0,)
)

it works! We suspect the issue is that GraphPPL sees vectors as single data points, so this is unfortunately not something we can fix. Does this solve your issue?

[bvdmitri]

Good catch @ThijsJenneskens! Indeed, I didn't initially thought about it, but the NormalMixture requires you to pass a tuple of arguments, not a vector. A vector is being interpreted as a single! (multivariate) data point, so it just complains that the dimensionality is not compatible with the z which expects two data points...

[meditans]

Thank you @ThijsJenneskens, this works! I'm also very interested in the process that lead to this discovery (if you weren't familiar with the code beforehand). I was looking at the NormalMixture code in ReactiveMP and the possibility you showed didn't occur to me!

[skoghoern]

since i am also working with the NMnode these days, is was also curious why it didnt work. and will just leave my understanding as a small side note, in case you were wondering why it actually fails. the error isnt so much about the tuple struct definition in normalmixture.jl, but rather has its origins in the make_node! function from GraphPPL. it calls materialize_interfaces(model, context, sorted_interfaces) where we dispatch on a version of getifcreated(model, context, unroll(interface)) with respect to the returned unroll() var, which basically returns the type of how we defined it in the model definition both for vectors and tuples.
only when we defined it as a tuple, getifcreated() dispatches to a method that is specialized to iterate over each element. This correctly creates multiple distinct variable nodes, one for each interface item.
(When we defined it as a vector, getifcreated dispatches to a method that treats the entire collection as a single constant variable. This results in the creation of only one variable node, regardless of the vector's contents. -> fails later in the factornode() of normalmixture.jl)
so its not too obvious imo. maybe we could add a hint to the "length-check-error message" of the factornode() version in normalmixture.jl to define it as a tuple?

[ThijsJenneskens]

Hi @meditans! I am a bit familiar with the code, which helped. What I did was investigate the normal mixture node, which is implemented in ReactiveMP.jl here. (reactive message passing is the package that implements the nodes RxInfer uses)

In that file, I noticed on lines 20 and 21 that the means and precisions are expected to be tuples, so I decided to try running inference where I give them as tuples. I hope that answers your question; if you have more, don't hesitate to ask!

[bvdmitri]

If I understand correctly, they don't exchange messages

Not really, they do exchange messages as any node needs to exchange messages with neighboring nodes.
NormalMixture node sends a message in a form of a Normal distribution, since we only support mean-field constraint for the NormalMixture node. Normal node also sends a message in a form of a Normal distribution. When they "collide" the posterior has also the form of a Normal distribution.

[meditans]

Thank you @bvdmitri, that's helpful, but then what's the message that the NormalMixture node is sending back? Is it the normal with the higher value in the switch parameter?

[meditans]

If I write:

using RxInfer

@model function example(x_mean)
    x ~ NormalMeanPrecision(x_mean, 1)
    x ~ NormalMixture(switch=[0.4, 0.6], m=(1, 2), p=(1, 2))
end

result = infer(
    model=example(),
    data=(x_mean=1.0,),
    addons=(AddonMemory(),)
)

I get:

julia> result.posteriors[:x]
Marginal(NormalWeightedMeanPrecision{Float64}(xi=3.8, w=2.6))) with (AddonMemory(Product memory:
 Message mapping memory:
    At the node: NormalMeanPrecision
    Towards interface: Val{:out}()
    With local constraint: Marginalisation()
    With addons: (AddonMemory(nothing),)
    With input marginals on Val{(:μ, :τ)}() edges: (PointMass{Float64}(1.0), PointMass{Int64}(1))
    With the result: NormalMeanPrecision{Float64}(μ=1.0, w=1.0)
 Message mapping memory:
    At the node: NormalMixture{2}
    Towards interface: Val{:out}()
    With local constraint: Marginalisation()
    With addons: (AddonMemory(nothing),)
    With input marginals on Val{(:switch, :m, :p)}() edges: (PointMass{Vector{Float64}}([0.4, 0.6]), (PointMass{Int64}(1), PointMass{Int64}(2)), (PointMass{Int64}(1), PointMass{Int64}(2)))
    With the result: NormalWeightedMeanPrecision{Float64}(xi=2.8, w=1.6)
),)

so I see the result is the weighted combination of the gaussians according to the weights.

[bvdmitri]

Sorry for delayed response...the rule is here

[bvdmitri]

Ah, wait, by back you probably meant towards the :out edge, the rule is here and it is indeed a weighted combination of the gaussians. This rule is due to mean-field constraint. I would be happy to somehow support sending "real" mixture, but with mean-field constraint this is the mathematically correct rule