How Is It Scientifically Valid to Have Ground Truth Answers to Be "B" ONLY?

[Sasha-Cui]

I was quite surprised to see that the probing.py evaluation script relies on the find_isolated_capital_b function to assess model performance. This effectively means that a model which always outputs “B” could achieve 100% accuracy, regardless of whether it actually understands privacy norms. Since the choices between A and B are not randomized, it's perhaps unsurprising that models appear to perform so well on this benchmark.

I understand that the probing questions are adapted from Shvartzshnaider et al. (Learning Privacy Expectations by Crowdsourcing Contextual Informational Norms), but their Mechanical Turk results clearly showed variation in human judgments — participants did not consistently respond “No” across the board.

Could you clarify whether the benchmark was constructed with these nuances in mind? I’m concerned about how this design choice might affect the scientific validity of the reported findings.

[shaoyijia]

Hi @Sasha-Cui , Thank you for your interest in our work!

This effectively means that a model which always outputs “B” could achieve 100% accuracy, regardless of whether it actually understands privacy norms.

It worth noticing that the evaluation of language models and LM-empowered agentic systems depends on both data points and construct. In another word, the data points cannot be interpreted alone without articulating how you evaluate it. About this topic, there is a nice tutorial from NeurIPS 2024.

Going back to this work, PrivacyLens contribute an evaluation framework to probe LMs and evaluate LM agents in actions. These data points are collected to instantiate the framework. We didn't include non-privacy sensitive examples because we focus on evaluating LM agents in privacy-sensitive scenarios rather than the probing parts. That being said, I agree that the evaluation would be more comprehensive if non-privacy-sensitive samples are included.

[Sasha-Cui]

Hi, @shaoyijia , thanks for your prompt and helpful response. It seems that you are measuring how good a judge is by calculating the percentage of death sentences she gives out. It seems that you have built an image classification dataset but only included cats in it, because you are not interested in dogs!

One of main takeaway of the paper, based on my reading, is that while the SOTA models are already doing very well on the simple MCQs, they are not doing great on complicated agentic scenarios. (That, then, shows that the agentic scenarios are useful in uncovering the failures of respecting privacy norms.) But these MCQs appear to be unrealistically simple with no contrasting samples, so it doesn't mean much to me that the SOTA models did them all very well. To be convinced of your main message, I am curious if the SOTA models do well on an MCQ privacy benchmark with greater construct validity. Do you know if such a benchmark exists? Thanks.