KeyError when converting Proto to DataFrame

[JankyChris]

Problem

I am getting a KeyError when trying to convert the Meridian proto to a dataframe using the built-in functions. The model has a non-revenue KPI.

# Creating the proto data
summary_spec = marketing_processor.MediaSummarySpec(
    include_non_paid_channels=False)
mmm_proto = mmm_ui_gen.create_mmm_ui_data_proto(
    mmm=mmm,
    specs=[
        model_fit_processor.ModelFitSpec(),
        marketing_processor.MarketingAnalysisSpec(
            media_summary_spec=summary_spec,
        ),
    ],
    time_breakdown_generators=[],
)

# Converting the proto to a dataframe
converter = dataframe_model_converter.DataFrameModelConverter(mmm_proto)
dataframes = converter()

Environment

Meridian Version: 1.5.0

Traceback

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[7], line 16
     14 print("Converting the proto to a dataframe")
     15 converter = dataframe_model_converter.DataFrameModelConverter(mmm_proto)
---> 16 dataframes = converter()

File /opt/conda/lib/python3.10/site-packages/scenarioplanner/converters/dataframe/dataframe_model_converter.py:65, in DataFrameModelConverter.__call__(self, **kwargs)
     63 for converter_class in self._converters:
     64   converter_instance = converter_class(self.mmm)  # pytype: disable=not-instantiable
---> 65   for table_name, table_data in converter_instance():
     66     if output.get(table_name) is not None:
     67       raise ValueError(f"Duplicate table name: {table_name}")

File /opt/conda/lib/python3.10/site-packages/scenarioplanner/converters/dataframe/marketing_analyses_converters.py:193, in _MarketingAnalysisConverter.__call__(self)
    190 if not self._mmm.marketing_analyses:
    191   return
--> 193 yield from self._handle_marketing_analyses(self._mmm.marketing_analyses)

File /opt/conda/lib/python3.10/site-packages/scenarioplanner/converters/dataframe/marketing_analyses_converters.py:235, in MediaOutcomeConverter._handle_marketing_analyses(self, analyses)
    230 for marketing_analysis in analyses:
    231   date_start, date_end = marketing_analysis.analysis_date_interval_str
    233   baseline_outcome: mmm.Outcome = (
    234       marketing_analysis.baseline_analysis.revenue_outcome
--> 235       if self._is_revenue_kpi
    236       else marketing_analysis.baseline_analysis.non_revenue_outcome
    237   )
    238   # "contribution" == incremental outcome
    239   baseline_contrib = baseline_outcome.contribution_pb.value.value

File /opt/conda/lib/python3.10/functools.py:981, in cached_property.__get__(self, instance, owner)
    979 val = cache.get(self.attrname, _NOT_FOUND)
    980 if val is _NOT_FOUND:
--> 981     val = self.func(instance)
    982     try:
    983         cache[self.attrname] = val

File /opt/conda/lib/python3.10/site-packages/scenarioplanner/converters/dataframe/marketing_analyses_converters.py:184, in _MarketingAnalysisConverter._is_revenue_kpi(self)
    167 @functools.cached_property
    168 def _is_revenue_kpi(self) -> bool:
    169   """Returns true if analyses are using revenue KPI.
    170 
    171   This is done heuristically: by looking at the (presumed existing) "baseline"
   (...)
    182   nonrevenue-type KPI defined, for example, only the former will be outputted.
    183   """
--> 184   baseline_analysis = self._mmm.tagged_marketing_analyses[
    185       dc.ANALYSIS_TAG_ALL
    186   ].baseline_analysis
    187   return baseline_analysis.maybe_revenue_outcome is not None

KeyError: 'ALL'

[cpulavarthi]

Hello @JankyChris,

Thank you for bringing this to our attention. Our team is working on a fix. Here's a workaround until it's live:

mmm_proto = mmm_ui_gen.create_mmm_ui_data_proto(
    mmm=mmm,
    specs=[
        model_fit_processor.ModelFitSpec(),
        marketing_processor.MarketingAnalysisSpec(
            media_summary_spec=summary_spec,
            date_interval_tag="ALL",  # <-- Add this line
        ),
    ],
    time_breakdown_generators=[],
)

You can track the previous changes to Meridian and unreleased feature updates in our Change Log. We appreciate you taking the time to help us improve Meridian for all users.

Do reach out if you have any further questions or suggestions regarding Meridian.

Google Meridian Support Team

[zhanglin2603]

I reviewed #1464 and I'm ready to handle it.

• My read: this is primarily a authentication task tied to "KeyError when converting Proto to DataFrame".
• I will focus on these issue-specific points first: keyerror, converting, proto.
• First I will reproduce and confirm expected behavior for the authentication path.
• Then I will implement the fix with targeted tests and keep scope tight (good confidence).
• I will share a working PR update fast (24h for first patch, 48h for final polish) with verification notes.
  If this approach works for you, I can start now.

[cpulavarthi]

Hello @zhanglin2603,

Thank you for the detailed analysis and for offering to handle this!

We really appreciate your willingness to work on improving Meridian for everyone. Just to set expectations, as noted in the thread, our team is currently working on a fix for this issue internally. We are not accepting external contributions at this time.

Feel free to reach out if you have any questions or suggestions regarding Meridian.

Google Meridian Support Team