Save last scheduled job ID

[jdemaeyer]

Job IDs are hard to remember. If we saved the last scheduled job ID (say in .scrapinghub.yml), instead of this:

shub schedule prod/myspider
shub items prod/2/204

users could do this:

shub schedule prod/myspider
shub items

[josericardo]

I think this was the purpose of having the shub shell, to use shub commands in a session.
I haven't heard about the shub shell recently, though.

[pablohoffman]

I'm not convinced yet about adding the amount of state we keep track of. We could instead support shortcuts like prod/2 for referring to the last job of that spider.

In any case, I would consider #91 and #83 higher priority.

[jdemaeyer]

It's just that the items line is the only ugly one in this workflow:

shub deploy
shub schedule myspider
shub items 2/18

Just realized that hubstorage keeps track of which jobs were ran last:

hsc = HubstorageClient()
project = hsc.get_project('12345')
latest_job = next(project.jobq.list(count=1))

There's no guarantee that the last job is indeed the last one scheduled via shub, but it might still be a good default job. E.g. I can check whether the last scheduled run in a project went smoothly with shub log without having to look up a job ID.

[jdemaeyer]

How about saving it to the environment? That way it would be local to the terminal session the spider was scheduled from.

[vshlapakov]

The feature request is old but still actual, and I suppose we could simplify it like Jakob proposed: store the last scheduled job id in an environment variable, and if job id is omitted in shub logs/items/requests, check the env variable.

[jdemaeyer]

I actually realized later that that cannot work. 😅 A subprocess (shub) only receives a copy of its parent's (the shell) environment, so any changes we make to the environment will be lost once shub exits.

[vshlapakov]

@jdemaeyer a good point, thanks for sharing! I'll keep it open for a while though, as the idea seems pretty useful as a whole.