Skip to content

Monitor service crashes when GitHub returns HTTP 503 #9

Open
@delan

Description

When this happens, the service needs to be restarted manually.

Nov 15 14:05:09 ci0 monitor-start[2405852]: [2024-11-15T14:05:09Z INFO  monitor::profile] Destroying runner 3493 with base vm name servo-windows10
Nov 15 14:05:09 ci0 monitor-start[2540097]: Domain 'ci-servo-windows10.3493' destroyed
Nov 15 14:05:09 ci0 monitor-start[2540136]: Domain 'ci-servo-windows10.3493' has been undefined
Nov 15 14:05:10 ci0 monitor-start[2540145]: will destroy tank/ci/servo-windows10.3493
Nov 15 14:05:15 ci0 monitor-start[2405852]: [2024-11-15T14:05:15Z INFO  monitor] Did not receive an API request
Nov 15 14:05:16 ci0 monitor-start[2540153]: gh: Server Error (HTTP 503)
Nov 15 14:05:16 ci0 monitor-start[2405852]: [2024-11-15T14:05:16Z ERROR monitor] Monitor thread error: Command exited with status exit status: 1
Nov 15 14:05:16 ci0 systemd[1]: monitor.service: Main process exited, code=exited, status=1/FAILURE
Nov 15 14:05:16 ci0 systemd[1]: monitor.service: Failed with result 'exit-code'.
Nov 15 14:05:16 ci0 systemd[1]: monitor.service: Consumed 4min 55.506s CPU time, 47.8M memory peak, 4K read from disk, 7.6M incoming IP traffic, 2.4M outgoing IP traffic.
  • Restart systemd services on failure (55b3f42)
  • Restart systemd services after extended failures (5 times in 10 seconds) with an hourly timer
  • Don’t crash when GitHub API requests fail

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions