Reproduction Results Significantly Lower Than Paper-Reported Results Using Provided Baselines Script for MemAlpha

[ZiLaoTou]

I am using the script for memalpha provided in the baselines to reproduce the results, but I found that the test results are significantly lower than those reported in the paper. I would like to ask whether the script used in the paper is the same as the one provided in the baselines, or if I have overlooked some key aspects?

[ZiLaoTou]

@deadpool66

[deadpool66]

Thanks for raising this — happy to clarify.

We are indeed using the official MemAlpha source code for reproduction. The only modifications we made are:

1. Interface adaptation: integrating MemAlpha into our framework via a standardized interface (no algorithmic changes to MemAlpha itself).
2. Client change: switching to asynchronous calls for efficiency, which should not affect model quality.

Regarding the performance gap compared to the original paper, we believe the main reason is the difference in the QA model:

• The MemAlpha paper uses Qwen3-32B for QA.
• We use Qwen3-4B-Instruct-2507.

This choice is intentional. Our work focuses on a unified policy model that handles both memory formation and QA in an end-to-end manner. To ensure a fair comparison across methods, we standardize the model scale and use the same backbone across all components. As a result, we replace the larger QA model used in MemAlpha with a model of comparable scale to our framework.

Given this, some performance drop compared to the original MemAlpha results is expected and mainly attributable to model capacity differences rather than implementation discrepancies.

If helpful, we can also provide results using larger QA models to better align with the original setup.

Let us know if you’d like more details — happy to dig deeper.