Unable to Replicate Table VII User Satisfaction Results with Any Heuristic Variant

[AminAkbariCodes]

Hi EV2Gym team,

Thank you for this excellent simulator! I'm trying to replicate the results from Table VII (V2G Profit Maximization with 10 EV Chargers) in your paper, but I'm encountering fundamental issues with user satisfaction metrics that might suggest a deeper configuration or implementation problem.

My Setup:

• Configuration: V2GProfitPlusLoads.yaml
• 10 chargers, 85 timesteps (15 min each), V2G enabled, Workplace scenario
• 100 stochastic runs

Problem:
I have tested all heuristic variants in baselines/heuristics.py but none achieve the user satisfaction values reported in Table VII:

1. ALAP: Paper reports 100±0% , but I get significantly lower values with both ChargeAsLateAsPossible and ChargeAsLateAsPossibleToDesiredCapacity

2. Round Robin: Paper reports 99±1%, but I get much lower values with RoundRobin, RoundRobin_GF, and RoundRobin_GF_off_allowed

Why This Is Concerning:
ALAP should logically achieve 100% user satisfaction since it charges vehicles as late as possible while meeting departure requirements. The fact that none of the variants achieve this suggests a configuration mismatch or implementation issue rather than just using the wrong algorithm class.
This problem persists even when I modify configuration parameters such as transformer limits, number of chargers, and inflexible loads. I cannot achieve stable high user satisfaction under any configuration.

Code structure:

from ev2gym.models.ev2gym_env import EV2Gym
from ev2gym.baselines.heuristics import ChargeAsLateAsPossible, RoundRobin

config_path = "ev2gym/example_config_files/V2GProfitPlusLoads.yaml"
n_runs = 100

for run in range(n_runs):
    env = EV2Gym(config_file=config_path)
    state, info = env.reset()
    algorithm = ChargeAsLateAsPossible()
    
    done = False
    while not done:
        action = algorithm.get_action(env)
        state, reward, done, truncated, info = env.step(action)
    
    user_satisfaction = info['average_user_satisfaction']

Specific Questions:

1. Would it be possible to share the exact configuration file that was used to generate the Table VII results? I suspect there may be some parameter differences from the example configs that could explain the discrepancy.
2. Could you clarify which specific heuristic class names correspond to each baseline algorithm in Table VII? For example, which variant of Round Robin was used - the basic RoundRobin, RoundRobin_GF, or RoundRobin_GF_off_allowed?
3. Would you be willing to share a reproduction script that demonstrates how to generate the Table VII results? Even a minimal example would be incredibly helpful for understanding the correct setup.
4. Are there any specific initialization seeds, random state settings, or other parameters that need to be configured for reproducibility that might not be obvious from the configuration file alone?

[StavrosOrf]

Hello,

Thank you for making using our simulator!

I see the issue. In this config we have defined that we want to use heterogeneous EV, where each EV can have different capabilities (power, cs_power). The ALAP algorithm was just using the cs_max_power. We fixed this bug in the new version, so now ALAP should be walking well (notice there are still some small approximation errors so it will achieve around 99.9% user satisfaction. The basic RoundRobin algorithm might suffer from the same issues, this is the resaon it has been updated since then.

Answert to questions:

1. You can check for previous EV2Gym versions to retrieve what configurations and environemnts were used in the past. Specifically, EV2gym version 1.0.0 with the V2GProfitPlusLoads.yaml was used to generate the figure. However, sicne then many changes have been made.
2. From all the Round Robin versions only the basic RoundRobin existed back then.
3. You can find the exact env and configs used for the EV2Gym paper here https://github.com/StavrosOrf/EV2Gym/releases/tag/1.0.0, however notice more bugs might appear there
4. To ensure reproducibility you should save a replay file and load it for every new simulation. Then you will laways get the exact same behavior.

I hope I answered your questions and you have nice simulations!