Huggingface pytorch training latest (#4688)

* chore: manually installing latest Go version

* Update ['dlc_developer_config.toml']

dlc_developer_config.toml:
{   'build': {   'build_frameworks': ['huggingface_pytorch'],
                 'build_inference': False,
                 'build_training': True},
    'buildspec_override': {   'dlc-pr-huggingface-pytorch-training': 'huggingface/pytorch/training/buildspec.yml'},
    'dev': {   'arm64_mode': False,
               'deep_canary_mode': False,
               'graviton_mode': False,
               'neuronx_mode': False},
    'test': {   'ec2_tests': True,
                'ecs_tests': True,
                'eks_tests': True,
                'sagemaker_local_tests': True,
                'sagemaker_remote_tests': True,
                'sanity_tests': True,
                'security_tests': True}}

* Revert "Update ['dlc_developer_config.toml']"

This reverts commit edc424d.

* chore: ignoring two vuln for unused neko binaries and upgrading go

* Update ['dlc_developer_config.toml']

dlc_developer_config.toml:
{   'build': {   'build_frameworks': ['huggingface_pytorch'],
                 'build_inference': False,
                 'build_training': True},
    'buildspec_override': {   'dlc-pr-huggingface-pytorch-training': 'huggingface/pytorch/training/buildspec.yml'},
    'dev': {   'arm64_mode': False,
               'deep_canary_mode': False,
               'graviton_mode': False,
               'neuronx_mode': False},
    'test': {   'ec2_tests': True,
                'ecs_tests': True,
                'eks_tests': True,
                'sagemaker_local_tests': True,
                'sagemaker_remote_tests': True,
                'sanity_tests': True,
                'security_tests': True}}

* fix: rm correct tarball version

* chore: adding logs to ecr scan test and patching library requirements to pass pip check

* chore: fixing logging

* chore: updating test to remove package from vulnerability list if there are no packages effected

* chore: formatting

* chore: more formatting

* Revert "Update ['dlc_developer_config.toml']"

This reverts commit 4555f31.

---------

Co-authored-by: shaernev <shaernev@amazon.com>

Author: ErnevSharma

huggingface/pytorch/training/docker/2.5/py3/cu124/Dockerfile.gpu

@@ -49,23 +49,29 @@ RUN pip install --no-cache-dir \
     peft==${PEFT_VERSION} \
     flash-attn==${FLASH_ATTN_VERSION}
 
+# Override conflicting versions to satisfy datasets
+RUN pip install --no-cache-dir dill==0.3.8 multiprocess==0.70.16 \
+ && pip install --no-cache-dir pathos==0.3.3 --no-deps \
+ && PATHOS_META=$(find /opt/conda/lib -type f -path "*pathos-0.3.3.dist-info/METADATA") \
+ && sed -i 's/dill.*/dill/' $PATHOS_META \
+ && sed -i 's/multiprocess.*/multiprocess/' $PATHOS_META
 
-# hf_transfer will be a built-in feature, remove the env variavle then
+
+# hf_transfer will be a built-in feature, remove the env variable then
 ENV HF_HUB_ENABLE_HF_TRANSFER="1"
 
 RUN apt-get update \
  # TODO: Remove upgrade statements once packages are updated in base image
  && apt-get -y upgrade --only-upgrade systemd openssl cryptsetup libkrb5-3 \
  && apt-get install -y git git-lfs wget tar \
- && wget https://go.dev/dl/go1.22.2.linux-amd64.tar.gz \
+ && wget https://go.dev/dl/go1.22.3.linux-amd64.tar.gz \
  && rm -rf /usr/local/go \
- && tar -C /usr/local -xzf go1.22.2.linux-amd64.tar.gz \
+ && tar -C /usr/local -xzf go1.22.3.linux-amd64.tar.gz \
  && ln -s /usr/local/go/bin/go /usr/bin/go \
- && rm go1.22.2.linux-amd64.tar.gz \
+ && rm go1.22.3.linux-amd64.tar.gz \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*
 
-
 COPY cuda-compatibility-lib.sh /usr/local/bin/cuda-compatibility-lib.sh
 RUN chmod +x /usr/local/bin/cuda-compatibility-lib.sh