Skip to content

Report individual crawl errors in issues #1131

Open
@tidoust

Description

@tidoust

Via #1130.

When crawl on a spec fails, the crawler records the error in an error property in ed/index.json and reuses previous extracts. In some cases, failure is transient, e.g., due to a network hiccup. In other cases though, the error is more permanent, e.g., because the extraction logic bumps into unexpected markup.

The more permanent errors may go unnoticed for some time, because nothing notifies us about the problem. Code should report these errors in an issue (and ideally close the issue if the problem disappears).

Side note: when the crawler crashes completely, the job fails, no need to handle that, GitHub already sends email notifications.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions