Pitfall does not terminate after timer runs out

[Sebastian-Griesbach]

In the description of Pitfall it says

The game is over if you collect all the treasures or if you die or if the time runs out.

However the environment does not terminate when the timer runs out. The player is simply unable to move but the environment keeps running.
The commonly used truncation limit for training atari games of 30 minutes is longer than the 20 minute timer. Which can lead to a lot of wasted transitions in a state of the game where the agent can no longer perform any action if the timer ran out.
I could of course change the truncation to accommodate this, but the timer running out should be a termination not a truncation as the agent can no longer collect any rewards after.
Is there some easy way for me to fix this in a fork of ALE? I could create a wrapper that counts the frames and terminates after 20 minutes but I'd prefer to fix this in the specific game itself somehow.
In the long term It might be worth considering changing this in ALE. As termination on timeout seems to be the intended behavior.

[Sebastian-Griesbach]

I found that some episodes do terminate at the time limit and some don't. I'm not yet sure when each behavior occurs.

[pseudo-rnd-thoughts]

This is a historic problem #268
The problem is "fixing" this bug and a couple of others could / should invalidate comparisons between the "buggy" an "fixed" ALE versions.
Therefore, I've been neverous to make these changes in order people can trust the project can provide fair comparisons.

The fix would be to do a v1.0 and v6 of the environments along with thorough performance comparisons to check how impactful these changes are to performance and therefore comparisons to old versions

[Sebastian-Griesbach]

Thank you for you answer! sorry for not finding the preexisting Issue. How is this different to other newer versions. Why would we need to do a performance check as long as it is clearly a new version?

[pseudo-rnd-thoughts]

Reproducibility is core to making science work and crucial for fair comparisons.
I don't know what affect this will have to fix this bug, but imagine that this improves performance for most algorithms by 10% on pitfall, this can produce suddenly unfair comparisons to previous work.
Therefore, yes, we can make a v6 of the environments with the environment fixes however we need to be explicit about what the changes are, the performance impact, etc.

I hope this explains my nervousness

[Sebastian-Griesbach]

I completely agree. I'd argue as long as Gymnasium is clear about a version change, the responsibility to compare to the correct version should lie with the people performing the comparison, not Gymnasium. I don't think it would be reasonable to test every possible method that might be relevant on the updated version as part of the update itself. How would one even choose which are the methods that need to be tested on this version?
So I understand your worries but I'd argue as long as we put a v6 on it, this is not yours to worry about.
Apparently this bug has been know for at least 6 years, the alternative can not be that we never fix bugs again because the original DQN paper used this version, can it? You are doing amazing work, and it can't be your responsibility to make sure that no one accidentally confuses a version for ever.

[pseudo-rnd-thoughts]

You're right, @Sebastian-Griesbach

I agree we should make a v6 for the Atari environments.
To achieve this, there should be a flag to enable or disable these bug fixes.
Looking through the issues and talking to friends, I believe Pitfall isn't the only environment with bugs. I know of at least

• reward overflow causing -1 million reward - BattleZone, Asterix, VideoPinball,
• Pitfall timeout
• Breakout termination after two levels completed (we can artificially allow the environment to have infinite levels)

If we can find more bug fixes, that would be great.

My hope is that spring next year, the vectorizer will be fixed, and the multiplayer will be integrated, meaning that I think a v1.0 might be able to happen.
So it would make sense to add all these bug fixes at the same time.

What are your thoughts? Are you aware of any other bugs? Would you be willing to contribute to the bug fixes?
I think we should spread to news to see if any more users know of problems to fix @JesseFarebro @psc-g

[Sebastian-Griesbach]

Happy to hear.
I´d be happy to contribute to those fixes. As of now my understanding of ALE is on the user level. So If I have some spare time I might dig in a little deeper. Having better descriptions of these bugs would probably be useful. So far I do not understand when the termination fails after the timeout in Pitfall and when it does not.
I am currently not aware of any other bugs.
@theovincent in your time working extensively with ALE, have you come across any bugs in specific environments?