Skip to content

exp push: HangupException when attempting to push an experiment to a codecommit repo #9416

Open
@mattlbeck

Description

@mattlbeck

Bug Report

Description

I have a codecommit repo and the remote url follows the form https://git-codecommit.<region>.amazonaws.com/v1/repos/<reponame>

A dvc exp push https://git-codecommit.<region>.amazonaws.com/v1/repos/<reponame> balky-chin -vv results in the error and traceack below.

I can workaround this by pushing the fully resolved experiment ref given in the debug using git

git push origin refs/exps/21/b9f0518f32301a128f5c3927fed1c308401b08/balky-chin

After that, commands like dvc exp list and dvc exp pull work fine against the remote.

Not sure what's changed since I helped resolve #9007 and I think its exactly the same scenario...

2023-05-07 20:12:35,563 DEBUG: v2.56.0 (pip), CPython 3.9.16 on Linux-5.10.76-linuxkit-x86_64-with-glibc2.31
2023-05-07 20:12:35,563 DEBUG: command: /opt/conda/bin/dvc exp push https://git-codecommit.eu-west-2.amazonaws.com/v1/repos/example-devbox-project balky-chin -vv
2023-05-07 20:12:35,564 TRACE: Namespace(cprofile=False, yappi=False, yappi_separate_threads=False, viztracer=False, viztracer_depth=None, viztracer_async=False, cprofile_dump=None, pdb=False, instrument=False, instrument_open=False, show_stack=False, quiet=0, verbose=2, cd='.', cmd='push', all_commits=False, rev=None, num=1, force=False, push_cache=True, dvc_remote=None, jobs=None, run_cache=False, git_remote='https://git-codecommit.eu-west-2.amazonaws.com/v1/repos/example-devbox-project', experiment=['balky-chin'], func=<class 'dvc.commands.experiments.push.CmdExperimentsPush'>, parser=DvcParser(prog='dvc', usage=None, description='Data Version Control', formatter_class=<class 'argparse.RawTextHelpFormatter'>, conflict_handler='error', add_help=False))
2023-05-07 20:12:35,851 DEBUG: git push experiment ['refs/exps/21/b9f0518f32301a128f5c3927fed1c308401b08/balky-chin:refs/exps/21/b9f0518f32301a128f5c3927fed1c308401b08/balky-chin'] -> 'https://git-codecommit.eu-west-2.amazonaws.com/v1/repos/example-devbox-project'
2023-05-07 20:12:38,325 ERROR: unexpected error - The remote server unexpectedly closed the connection.                                                                                                                   
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/dvc/cli/__init__.py", line 210, in main
    ret = cmd.do_run()
  File "/opt/conda/lib/python3.9/site-packages/dvc/cli/command.py", line 26, in do_run
    return self.run()
  File "/opt/conda/lib/python3.9/site-packages/dvc/commands/experiments/push.py", line 65, in run
    result = self.repo.experiments.push(
  File "/opt/conda/lib/python3.9/site-packages/dvc/repo/experiments/__init__.py", line 494, in push
    return push(self.repo, *args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/dvc/repo/__init__.py", line 65, in wrapper
    return f(repo, *args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/dvc/repo/scm_context.py", line 151, in run
    return method(repo, *args, **kw)
  File "/opt/conda/lib/python3.9/site-packages/dvc/repo/experiments/push.py", line 117, in push
    push_result = _push(repo, git_remote, exp_ref_set, force)
  File "/opt/conda/lib/python3.9/site-packages/dvc/repo/experiments/push.py", line 159, in _push
    results: Mapping[str, SyncStatus] = repo.scm.push_refspecs(
  File "/opt/conda/lib/python3.9/site-packages/scmrepo/git/__init__.py", line 286, in _backend_func
    result = func(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 580, in push_refspecs
    result = client.send_pack(
  File "/opt/conda/lib/python3.9/site-packages/dulwich/client.py", line 2054, in send_pack
    ref_status = self._handle_receive_pack_tail(
  File "/opt/conda/lib/python3.9/site-packages/dulwich/client.py", line 881, in _handle_receive_pack_tail
    for chan, data in _read_side_band64k_data(proto.read_pkt_seq()):
  File "/opt/conda/lib/python3.9/site-packages/dulwich/client.py", line 458, in _read_side_band64k_data
    for pkt in pkt_seq:
  File "/opt/conda/lib/python3.9/site-packages/dulwich/protocol.py", line 271, in read_pkt_seq
    pkt = self.read_pkt_line()
  File "/opt/conda/lib/python3.9/site-packages/dulwich/protocol.py", line 214, in read_pkt_line
    raise HangupException()
dulwich.errors.HangupException: The remote server unexpectedly closed the connection.

Environment information

DVC version: 2.56.0 (pip)
-------------------------
Platform: Python 3.9.16 on Linux-5.10.76-linuxkit-x86_64-with-glibc2.31
Subprojects:
        dvc_data = 0.47.1
        dvc_objects = 0.21.1
        dvc_render = 0.3.1
        dvc_task = 0.2.1
        scmrepo = 1.0.1
Supports:
        http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2023.4.0, boto3 = 1.26.76)
Config:
        Global: /home/mambauser/.config/dvc
        System: /etc/xdg/dvc
Cache types: symlink
Cache directory: fuse.grpcfuse on grpcfuse
Caches: local
Remotes: s3
Workspace directory: fuse.grpcfuse on grpcfuse
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/d5d9f9155688caf9fa42340dcf3f255e

Metadata

Metadata

Assignees

No one assigned

    Labels

    gitRelated to git and git backendsresearch

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions