HoldingPen: arXiv reappears

[ksachs]

Expected Behavior

harvests of an article we harvested previously should be filtered

I'm not sure whether this is already_harvested or previously_rejected.
In any case: @fschwenn and I don't want to see an article more than once in the holdingpen.

Current Behavior / Example

An arxiv article (1707.02681) was harvested and rejected yesterday (https://labs.inspirehep.net/holdingpen/675354). Today another harvest of the same article is again waiting in the HP (https://labs.inspirehep.net/holdingpen/676378).

https://labs.inspirehep.net/holdingpen/list/?page=1&size=10&q=1707.02681

Screenshots (if appropriate):

[fschwenn]

This seems to happen even for pending articles.
https://labs.inspirehep.net/holdingpen/677472 was harvested while
https://labs.inspirehep.net/holdingpen/676633 was already waiting for a decision. It has not been updated on arXiv (https://arxiv.org/abs/1707.02998) meanwhile.

[kaplun]

This is normal, until we merge: #2423. The current situation is that we don't know if something is an update or not. So there is a stupid check: if something has a creation date older than 2 weeks it's rejected assuming it's duplicate.
Once #2423 is merged (which will happen as soon as we implement the filtering of updated Vs. new records I mentioned on standup) then you will see updates treated as updates.

[kaplun]

Re: what @fschwenn says... mmh...

[kaplun]

The record you mention has been modified last time the 12th according to arXiv, so it make sense that it has been harvested on the 13th... (assuming my explanation that we are not able yet to distinguish updates).

[ksachs]

we don't need the merger. We need to know whether we saw it before. I.e. it should be enough to search for the arXiv-id in INSPIRE - if it's there it's an update - do whatever was done on legacy; if not in INSPIRE query holdingpen - if it's there reject automatically.
At least reduce the time-window to max. 4 days. Even on holidays and weekends I guess we get the first harvest within 4 days.

[ksachs]

there is no difference in the metadata for 676633 and 677472, maybe a status-change triggered the update.
diff_676633.vs.677472.txt

[kaplun]

we don't need the merger.

@ksachs we do because it's all implemented into #2423 which is basically ready. No point in adding an other hack that will only postpone having the functionality.

[ksachs]

OK, understood. So we'll continue with the DESY workflow until then.

[kaplun]

Yeah, it's still great if you monitor, just ignore the issue of duplicate harvesting.

[ksachs]

just a quick check of astro articles from Wednesday that were not harvested directly to INSPIRE and should be handled via HP (as far as I understand it). I don't know how to query HP via python, otherwise I would have checked all subjects.

not in HP
[1707.03130, 1707.03155, 1707.03179, 1707.03180, 1707.03343, 1707.03345, 1707.03390]

once in HP
[1707.02980, 1707.02982, 1707.02985, 1707.02989, 1707.03064, 1707.03120, 1707.03121, 1707.03170, 1707.03171, 1707.03208]

twice in HP
[1707.03009, 1707.03026, 1707.03105, 1707.03108, 1707.03118, 1707.03256, 1707.03356, 1707.02990, 1707.02996]

3-times in HP
[1707.02997,  1707.03078]

[ksachs]

Just to show you that @fschwenn and I have to reject (almost) everything twice. Every day! And usually independently - the records come in again the next day. It's super annoying. What I'm trying to say: the matcher is good enough - just looking for IDs. It's mostly records that are rejected, i.e. no merger needed. You would do us an enormous favor if these double records could be avoided.