Adding timeout termination to PitFall

[Sebastian-Griesbach]

It seems that the current code actually does not contain any intended detection for the Timeout.

logo_timer is non zero when the Activision Logo scrolls in the bottom left.

This is happening

1. At the beginning of the game before the player moved
2. When the player lost all lives
3. When the timer ran out
   Apparently the logo_timer was put here because the lives_bytes alone can temporarily take on a zero value during the death transition.

The current Termination condition m_terminal thus only checks if all lives have been lost. There is no detection for the timer reaching zero. I added two variables tracking minutes and seconds and added it to the termination condition.

I don't see how to reasonably add tests for this as the timer only exists

1. in ram what I am using now
2. Visually which is hard to use in a test.

I wrote a little script that records the last view frames of a episode to visually confirm that the timer runs out right before timeout termination which I can provide if needed.

Edit: I did not know that I can not add issue references in the title: #654, #268

[pseudo-rnd-thoughts]

Thanks for the PR, whats the difference between the logo_timer and the actual timer that you've extracted?

Also, could this timer_exprired feature be enabled or disabled

[Sebastian-Griesbach]

I think the logo_timer controls the timing of the scrolling of the logo. When it's active it scrolls between "Activision" and "Copyright 1982", this is not directly related to the in game timer. It just happens to co-inside with gameover which I assume is why this was originally used.

How should I add the mentioned option? As I understand I can not directly add an argument to the environment initialization. I can use a game "mode" that takes an int, is that the intended way?

[Sebastian-Griesbach]

I used the mode argument to add the option for the original behavior.
With the default mode 0, the environment will terminate after the timer runs out as intend in the documentation.
In mode 1 the old termination behavior persists, where the agent will be unable to move after the timer has run out but the episode will not terminate.

You can test this but running the environment and counting the steps. The timer only start after the first not no-op action that's why we need to take another action in the first step. Here is an example for how to test this:

env = gym.make("ALE/Pitfall-v5", mode=0)
observation, info = env.reset()
done = False
steps = 0

while not done:
    action = 1 if steps == 0 else 0
    observation, reward, terminated, truncated, info = env.step(action)
    steps += 1
    done = terminated or truncated

print(f"Episode finished: {steps} steps")
env.close()

In mode 0 the environment terminates after 20 minutes (18000 steps),
In mode 1 the environment truncates after 30 minutes (27000 steps)

[Sebastian-Griesbach]

So looking at the failed test, it seems that using the mode is not the correct way of implementing this option.

Could you give me a hint @pseudo-rnd-thoughts what would be the intended way to implement such an option instead? I could also do it via a "difficulty" but that also seems hacky.

[Sebastian-Griesbach]

My current best suggestion would be to drop the Option. As the fixed behavior is the intended behavior already present in the documentation.

"You have three lives. The game is over if you collect all the treasures or if you die or if the time runs out."

This would simplify the PR a lot. I can not think of a reason why someone would intentionally want to keep this bugged behavior unless for comparing to old results in which case one should anyways always use the same version the other method was run on.

Let me know how to proceed, I'd be happy to resolve this issue outside of my local installation.