Connection broken: IncompleteRead for dandi-cli; add a generic support for skipping some logs? #208
Description
Exclusively for dandi-cli started to happen recently starting with
16 Jun 30 Cron Daemon Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-cli && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-cli.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch
logs from execution with traceback etc
Subject: Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-cli && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-cli.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch
2025-07-06T08:00:01-0400 [INFO ] tinuous tinuous 0.6.0
2025-07-06T08:00:02-0400 [INFO ] tinuous Fetching resources from github
2025-07-06T08:00:02-0400 [INFO ] tinuous Fetching runs newer than 2025-07-05 06:04:35+00:00
2025-07-06T08:00:02-0400 [INFO ] tinuous Fetching runs for workflow .github/workflows/claude.yml (Claude Code)
2025-07-06T08:00:03-0400 [INFO ] tinuous Fetching runs for workflow .github/workflows/codeql.yml (CodeQL)
2025-07-06T08:00:03-0400 [INFO ] tinuous Fetching runs for workflow .github/workflows/docs.yml (Build Docs)
2025-07-06T08:00:04-0400 [INFO ] tinuous Fetching runs for workflow .github/workflows/labels.yml (Check PR Labels)
2025-07-06T08:00:05-0400 [INFO ] tinuous Fetching runs for workflow .github/workflows/lint.yml (Linters)
2025-07-06T08:00:05-0400 [INFO ] tinuous Fetching runs for workflow .github/workflows/release.yml (Auto-release on PR merge)
2025-07-06T08:00:06-0400 [INFO ] tinuous Fetching runs for workflow .github/workflows/run-tests.yml (Tests)
2025-07-06T08:00:06-0400 [INFO ] tinuous Fetching runs for workflow .github/workflows/test.yml (Tests)
2025-07-06T08:00:07-0400 [INFO ] tinuous Found run 6563
2025-07-06T08:05:25-0400 [INFO ] tinuous Downloading logs for test.yml (Tests) #6563 to 2025/07/06/github/cron/20250706T060452/03b7c07/Tests/6563
2025-07-06T08:05:55-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(3536 bytes read, 560 more expected)', IncompleteRead(3536 bytes read, 560 more expected)); waiting & retrying
2025-07-06T08:06:26-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(4016 bytes read, 80 more expected)', IncompleteRead(4016 bytes read, 80 more expected)); waiting & retrying
2025-07-06T08:06:59-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(4056 bytes read, 40 more expected)', IncompleteRead(4056 bytes read, 40 more expected)); waiting & retrying
2025-07-06T08:07:32-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(3856 bytes read, 240 more expected)', IncompleteRead(3856 bytes read, 240 more expected)); waiting & retrying
2025-07-06T08:08:07-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(4008 bytes read, 88 more expected)', IncompleteRead(4008 bytes read, 88 more expected)); waiting & retrying
2025-07-06T08:08:43-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(4016 bytes read, 80 more expected)', IncompleteRead(4016 bytes read, 80 more expected)); waiting & retrying
2025-07-06T08:09:19-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(4048 bytes read, 48 more expected)', IncompleteRead(4048 bytes read, 48 more expected)); waiting & retrying
2025-07-06T08:09:56-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(4080 bytes read, 16 more expected)', IncompleteRead(4080 bytes read, 16 more expected)); waiting & retrying
2025-07-06T08:10:35-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(8160 bytes read, 32 more expected)', IncompleteRead(8160 bytes read, 32 more expected)); waiting & retrying
2025-07-06T08:11:14-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(4080 bytes read, 16 more expected)', IncompleteRead(4080 bytes read, 16 more expected)); waiting & retrying
2025-07-06T08:11:55-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(4008 bytes read, 88 more expected)', IncompleteRead(4008 bytes read, 88 more expected)); waiting & retrying
2025-07-06T08:12:36-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(3760 bytes read, 336 more expected)', IncompleteRead(3760 bytes read, 336 more expected)); waiting & retrying
2025-07-06T08:13:18-0400 [ERROR ] tinuous Max retries exceeded
Traceback (most recent call last):
File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/urllib3/response.py", line 441, in _error_catcher
yield
File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/urllib3/response.py", line 770, in read_chunked
chunk = self._handle_chunk(amt)
File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/urllib3/response.py", line 723, in _handle_chunk
returned_chunk = self._fp._safe_read(self.chunk_left)
File "/home/dandi/miniconda3/lib/python3.8/http/client.py", line 610, in _safe_read
raise IncompleteRead(data, amt-len(data))
http.client.IncompleteRead: IncompleteRead(4064 bytes read, 32 more expected)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/requests/models.py", line 760, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/urllib3/response.py", line 575, in stream
for line in self.read_chunked(amt, decode_content=decode_content):
File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/urllib3/response.py", line 796, in read_chunked
self._original_response.close()
File "/home/dandi/miniconda3/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/urllib3/response.py", line 458, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(4064 bytes read, 32 more expected)', IncompleteRead(4064 bytes read, 32 more expected))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/mnt/backup/dandi/tinuous-logs/venv/bin/tinuous", line 10, in <module>
sys.exit(main())
File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/decorators.py", line 38, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/__main__.py", line 127, in fetch
paths = obj.download(Path(path))
File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/github.py", line 356, in download
self.client.download_zipfile(self.logs_url, path)
File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/base.py", line 143, in download_zipfile
self.download(path, zippath)
File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/base.py", line 117, in download
for chunk in r.iter_content(chunk_size=8192):
File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/requests/models.py", line 763, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(4064 bytes read, 32 more expected)', IncompleteRead(4064 bytes read, 32 more expected))
using straight `wget` on my laptop we pretty much see similar problem which eventually leads to 403 Forbidden
$ wget -S --header="Authorization: Bearer XXXX" https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs
...
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4932
X-RateLimit-Reset: 1751875203
X-RateLimit-Used: 68
X-RateLimit-Resource: core
...
Resolving results-receiver.actions.githubusercontent.com (results-receiver.actions.githubusercontent.com)... 140.82.114.22
Connecting to results-receiver.actions.githubusercontent.com (results-receiver.actions.githubusercontent.com)|140.82.114.22|:443... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
content-disposition: attachment; filename=logs_41239562373.zip
Content-Type: application/zip
Date: Mon, 07 Jul 2025 07:27:11 GMT
Transfer-Encoding: chunked
x-github-backend: Kubernetes
X-GitHub-Request-Id: XXXX
Length: unspecified [application/zip]
Saving to: ‘logs’
logs [ <=> ] 78.20M 2.96MB/s in 26s
2025-07-07 03:27:41 (3.06 MB/s) - Read error at byte 82018304 (The request is invalid.).Retrying.
--2025-07-07 03:27:42-- (try: 2) https://results-receiver.actions.githubusercontent.com/rest/runs/e0184fe9-1488-40fe-95b7-2a402630b958/logs?filename=logs_41239562373.zip&signature=XXX
Connecting to results-receiver.actions.githubusercontent.com (results-receiver.actions.githubusercontent.com)|140.82.114.22|:443... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
content-disposition: attachment; filename=logs_41239562373.zip
Content-Type: application/zip
Date: Mon, 07 Jul 2025 07:27:42 GMT
Transfer-Encoding: chunked
x-github-backend: Kubernetes
X-GitHub-Request-Id: XXXX
Length: unspecified [application/zip]
Saving to: ‘logs’
logs [ <=> ] 99.35M 3.89MB/s in 30s
2025-07-07 03:28:12 (3.32 MB/s) - Read error at byte 104173568 (The request is invalid.).Retrying.
--2025-07-07 03:28:14-- (try: 3) https://results-receiver.actions.githubusercontent.com/rest/runs/e0184fe9-1488-40fe-95b7-2a402630b958/logs?filename=logs_41239562373.zip&signature=XXXX
Connecting to results-receiver.actions.githubusercontent.com (results-receiver.actions.githubusercontent.com)|140.82.114.22|:443... connected.
HTTP request sent, awaiting response...
HTTP/1.1 403 Forbidden
Date: Mon, 07 Jul 2025 07:28:15 GMT
Content-Length: 0
x-github-backend: Kubernetes
X-GitHub-Request-Id: XXXX
2025-07-07 03:28:15 ERROR 403: Forbidden.so I think we are just doomed to skip such runs/logs?
may be for now, to be explicit, we should just add a feature to explicitly skip them in the config, and make it flexible, e.g. some kind of a code eval based on f-string e.g.
ci:
github:
paths:
logs: '{year}//{month}//{day}/{ci}/{type}/{type_id}/{commit[:7]}/{wf_name}/{number}/'
skip_evals:
- "f'{wf_name}/{number}' == 'Tests/6563'"