Incorrect job status returned by 'at' adaptor of finished jobs (when using xenon-cli)

[jmaassen]

As reported by @arnikz there are some issues with the "at" adaptor.

xenon-middleware/xenon-cli#81

I'll look into this

[jmaassen]

I can confirm that retrieving the status after the job has finished will result in an UNKNOWN status. This may have to do with the fact that "at" is not a full blown scheduler, and may not remember jobs once they have finished?

Also, there seems to be an issue with the generating the tempfile used to submit to "at" (/tmp/xenon.at.0). The second time a submit a job via xenon-cli, the command is appended to the generated script of the first. This is probably caused by the fact that xenon-cli has no memory between commands, and therefore doesn't know what job number to assign to the new job?

[jmaassen]

After some digging, the problem is as follows:

• xenon-cli is stateless between runs. Therefore, every time we use it to start a job with the 'at' adaptor, its starts from scratch (i.e., by counting jobs from '0' again.)
• at will give us a job number when we submit a job, this is usually not the number that xenon came up with.
• at only remembers queued / running jobs. It does not remember jobs which are done. Retrieving info about the job with xenon-cli when the job is waiting/running is no problem, but as soon as the job finishes all info is gone. We do save the exit code of the command we run in the script we submit to 'at' (by appending it to the end of this script), but at the moment that is all that remains.
• When using xenon as a library, the 'at' adaptor will remember a list of jobs it has seen. When info about a job is requested, but is is no longer known by 'at' we assume it was done. However, this list is lost between xenon-cli runs. Therefore, we lose all knowledge about the job

[jmaassen]

Fixed in improved-at branch