The build-and-test CI job is hanging

[dancysoft]

Example from https://app.circleci.com/pipelines/github/yaoyuannnn/gerritlab/1289/workflows/be20e9ab-6367-4a3e-9e62-03fc2d51d84f/jobs/1864

============================= test session starts ==============================
platform linux -- Python 3.8.18, pytest-8.0.0, pluggy-1.4.0
rootdir: /home/circleci/project
collected 10 items                                                             

unit_tests/git_credentials_test.py ..                                    [ 20%]
unit_tests/merge_request_test.py F..
Too long with no output (exceeded 10m0s): context deadline exceeded

[dancysoft]

After enabling verbose output from pytest:

============================= test session starts ==============================
platform linux -- Python 3.8.18, pytest-8.0.0, pluggy-1.4.0
rootdir: /home/circleci/project
collected 10 items                                                             

unit_tests/git_credentials_test.py::test_git_instances PASSED
unit_tests/git_credentials_test.py::test_git_credential_store PASSED
unit_tests/merge_request_test.py::MergeRequestTest::test_create_multiple_mrs Commits to be reviewed, destined for origin/test-0541f69f-4c93-4abd-ac02-420f3ea5e214:
* 25607d27 Add a new file2. [new]
* 7be83e38 Add a new file1. [new]
* ba3712fc Add a new file0. [new]

SUCCESS

New MRs:
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25600 Add a new file0.
    test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I626f9027f8dde8b60f542ac59d28c8f0222cb7cc -> test-0541f69f-4c93-4abd-ac02-420f3ea5e214
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25601 Add a new file1.
    test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I16dcb8583d4db12e1b7a45cd92fb52ac865304a1 -> test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I626f9027f8dde8b60f542ac59d28c8f0222cb7cc
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25602 Add a new file2.
    test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I54f2d37a7d1a18e7ecddd8dbd218a64fcbea5989 -> test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I16dcb8583d4db12e1b7a45cd92fb52ac865304a1

To git@gitlab.com:yaoyuannnn/gitlab-gerrit-review-tests.git

Merging merge requests destined for test-0541f69f-4c93-4abd-ac02-420f3ea5e214:
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25600 Add a new file0.
    [merge status]: mergeable
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25601 Add a new file1.
    [merge status]: mergeable
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25602 Add a new file2.
    [merge status]: mergeable

SUCCESS

New Merged MRs:
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25600 Add a new file0.
    test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I626f9027f8dde8b60f542ac59d28c8f0222cb7cc -> test-0541f69f-4c93-4abd-ac02-420f3ea5e214
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25601 Add a new file1.
    test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I16dcb8583d4db12e1b7a45cd92fb52ac865304a1 -> test-0541f69f-4c93-4abd-ac02-420f3ea5e214
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25602 Add a new file2.
    test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I54f2d37a7d1a18e7ecddd8dbd218a64fcbea5989 -> test-0541f69f-4c93-4abd-ac02-420f3ea5e214
To git@gitlab.com:yaoyuannnn/gitlab-gerrit-review-tests.git
PASSED
unit_tests/merge_request_test.py::MergeRequestTest::test_create_single_mr Commits to be reviewed, destined for origin/test-0541f69f-4c93-4abd-ac02-420f3ea5e214:
* 7ac92a6a Add a new file. [new]

SUCCESS

New MRs:
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25603 Add a new file.
    test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I750ef660fd43107cb94251ef11f45a06c8553046 -> test-0541f69f-4c93-4abd-ac02-420f3ea5e214

To git@gitlab.com:yaoyuannnn/gitlab-gerrit-review-tests.git

Merging merge requests destined for test-0541f69f-4c93-4abd-ac02-420f3ea5e214:
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25603 Add a new file.
    [merge status]: mergeable

SUCCESS

New Merged MRs:
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25603 Add a new file.
    test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I750ef660fd43107cb94251ef11f45a06c8553046 -> test-0541f69f-4c93-4abd-ac02-420f3ea5e214
To git@gitlab.com:yaoyuannnn/gitlab-gerrit-review-tests.git
PASSED
unit_tests/merge_request_test.py::MergeRequestTest::test_insert_new_mr Commits to be reviewed, destined for origin/test-0541f69f-4c93-4abd-ac02-420f3ea5e214:
* a89e0a90 Add a new file1. [new]
* 3b979e60 Add a new file0. [new]

SUCCESS

New MRs:
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25604 Add a new file0.
    test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I451fdb124fd72c555d0276580d9eed0795e86a7f -> test-0541f69f-4c93-4abd-ac02-420f3ea5e214
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25605 Add a new file1.
    test-0541f69f-4c93-4abd-ac02-420f3ea5e214-Ia29c9c9397c5d815f77832cfbc8999cda4b09d19 -> test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I451fdb124fd72c555d0276580d9eed0795e86a7f

To git@gitlab.com:yaoyuannnn/gitlab-gerrit-review-tests.git
Commits to be reviewed, destined for origin/test-0541f69f-4c93-4abd-ac02-420f3ea5e214:
* f5d9fb9a Add a new file1. [!25605]
* a2c4beec Add a inserted MR. [new]
* 3b979e60 Add a new file0. [!25604]

SUCCESS

Updated MRs:
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25604 Add a new file0.
    test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I451fdb124fd72c555d0276580d9eed0795e86a7f -> test-0541f69f-4c93-4abd-ac02-420f3ea5e214
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25605 Add a new file1.
    test-0541f69f-4c93-4abd-ac02-420f3ea5e214-Ia29c9c9397c5d815f77832cfbc8999cda4b09d19 -> test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I91e5a78fc464d3123bfa96bbbbc33d4237d4efdd

New MRs:
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25606 Add a inserted MR.
    test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I91e5a78fc464d3123bfa96bbbbc33d4237d4efdd -> test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I451fdb124fd72c555d0276580d9eed0795e86a7f

To git@gitlab.com:yaoyuannnn/gitlab-gerrit-review-tests.git

Merging merge requests destined for test-0541f69f-4c93-4abd-ac02-420f3ea5e214:
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25604 Add a new file0.
    [merge status]: mergeable
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25606 Add a inserted MR.
    [merge status]: mergeable
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25605 Add a new file1.
    [merge status]: mergeable

SUCCESS

New Merged MRs:
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25604 Add a new file0.
    test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I451fdb124fd72c555d0276580d9eed0795e86a7f -> test-0541f69f-4c93-4abd-ac02-420f3ea5e214
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25606 Add a inserted MR.
    test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I91e5a78fc464d3123bfa96bbbbc33d4237d4efdd -> test-0541f69f-4c93-4abd-ac02-420f3ea5e214
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25605 Add a new file1.
    test-0541f69f-4c93-4abd-ac02-420f3ea5e214-Ia29c9c9397c5d815f77832cfbc8999cda4b09d19 -> test-0541f69f-4c93-4abd-ac02-420f3ea5e214
To git@gitlab.com:yaoyuannnn/gitlab-gerrit-review-tests.git
PASSED
unit_tests/merge_request_test.py::MergeRequestTest::test_merge_mrs Commits to be reviewed, destined for origin/test-0541f69f-4c93-4abd-ac02-420f3ea5e214:
* f7cc5749 Add a new file2. [new]
* ade7c231 Add a new file1. [new]
* bcae89f1 Add a new file0. [new]

SUCCESS

New MRs:
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25607 Add a new file0.
    test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I215275a031621d3911c8628029a8cc34c5758adc -> test-0541f69f-4c93-4abd-ac02-420f3ea5e214
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25608 Add a new file1.
    test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I910e49ced4d25b3090374c700929791c6bc6c70d -> test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I215275a031621d3911c8628029a8cc34c5758adc
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25609 Add a new file2.
    test-0541f69f-4c93-4abd-ac02-420f3ea5e214-If02309d6d455625901e7800e2d2ffe8024327b9e -> test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I910e49ced4d25b3090374c700929791c6bc6c70d

To git@gitlab.com:yaoyuannnn/gitlab-gerrit-review-tests.git

Merging merge requests destined for test-0541f69f-4c93-4abd-ac02-420f3ea5e214:
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25607 Add a new file0.
    [merge status]: mergeable
* https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25608 Add a new file1.
    [merge status]: mergeable

Too long with no output (exceeded 10m0s): context deadline exceeded

While it was blocked, I found https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25607 merged and https://gitlab.com/yaoyuannnn/gitlab-gerrit-review-tests/-/merge_requests/25608 in a weird state with no commits listed and an unusual sequence of events in MR activity log:

• Yuan Yao approved this merge request
• Yuan Yao changed target branch from test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I215275a031621d3911c8628029a8cc34c5758adc to test-0541f69f-4c93-4abd-ac02-420f3ea5e214
• Yuan Yao deleted the test-0541f69f-4c93-4abd-ac02-420f3ea5e214-I215275a031621d3911c8628029a8cc34c5758adc branch. This merge request now targets the test-0541f69f-4c93-4abd-ac02-420f3ea5e214 branch
• Yuan Yao added 15 commits
  ade7c231...b611c7fd - 15 commits from branch test-0541f69f-4c93-4abd-ac02-420f3ea5e214

[dancysoft]

This problem is probably related to https://gitlab.com/gitlab-org/gitlab/-/issues/441446