AWS stuck on `Waiting for SSH access`

[mukundt]

Hey! I've been able to successfully deploy on GCP, but when I try AWS, SkyPilot times out after 600s:

I 10-10 16:33:21 provisioner.py:73] Launching on AWS us-west-2 (us-west-2a,us-west-2b,us-west-2c)
⠋ Launching - Waiting for SSH access
E 10-10 16:43:32 provisioner.py:491] *** Failed setting up cluster. ***
RuntimeError: Failed to SSH to xxx.xxx.xxx after timeout 600s.

Is something wrong with my AWS config? sky check returns AWS: enabled.

[Michaelvll]

Thanks for reporting the issue @mukundt! Could you share the skypilot version you are using as well as the yaml file used for launching the VM?

[mukundt]

@Michaelvll using skypilot==0.4.0.

Launch command: sky launch -c a10g finetune_skypilot_run.yaml -y --gpus A10G:1 --cloud aws

YML

resources:
  disk_size: 800
  ports:
    - 8000

file_mounts:
  /sky_data: .

setup: |
  conda activate train
  if [ $? -ne 0 ]; then
    conda create -n train python=3.9 -y
    conda activate train
  fi

  rm -rf axolotl
  git clone https://github.com/OpenAccess-AI-Collective/axolotl
  cd axolotl

  pip install wandb
  pip install vllm
  pip install fschat
  pip install packaging
  pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118
  pip install -e '.[deepspeed]'
  pip install flash-attn==2.3.0 --no-build-isolation

run: |
  conda activate train
  cd axolotl
  echo 'Starting training...'
  accelerate launch -m axolotl.cli.train /sky_data/refinetune_7b_axolotl_config.yml

[Michaelvll]

Hey @mukundt, thanks for sharing the yaml file. Would you like to try the latest SkyPilot master branch or pip uninstall skypilot; pip install -U skypilot-nightly to see if that solves the issue?

[PoornaSaiNagendra]

Hi @Michaelvll, I've encountered the same issue with SkyPilot on AWS and attempted the solution you mentioned by installing skypilot-nightly, but the problem persists. Any additional suggestions or insights would be greatly appreciated.

[Michaelvll]

Hi @Michaelvll, I've encountered the same issue with SkyPilot on AWS and attempted the solution you mentioned by installing skypilot-nightly, but the problem persists. Any additional suggestions or insights would be greatly appreciated.

Hi @PoornaSaiNagendra, thanks for the question! Could you share the provision.log as mentioned in the output of your sky launch?

Also, it would be nice to try to check the AWS console to find the public IP of the cluster, and ssh <public-ip> -i ~/.ssh/sky-key to see if you can connect to the cluster to understand where the error comes from. : )

[nshaposh]

Anything on this? Just experienced the same issue...

[Michaelvll]

Anything on this? Just experienced the same issue...

Thanks for asking @nshaposh! Could you share the yaml and the provision.log that causes the issue, so we can take a deeper look into the issue?

[nshaposh]

provision.log

There seems to be a problem propagating the ssh keys to the instance or something...
When I check from AWS console, the instance is there, although it is not associated with any ssh keys.

Here yaml (can't attach it). It is yaml for llm-llama-tuner, I just modified instance type and cloud:

# llm-tuner.yaml

resources:
  accelerators: A10G:1  # 1x NVIDIA A10 GPU, about US$ 0.6 / hr on Lambda Cloud. Run `sky show-gpus` for supported GPU types, and `sky show-gpus [GPU_NAME]` for the detailed information of a GPU type.
  cloud: aws  # Optional; if left out, SkyPilot will automatically pick the cheapest cloud.

file_mounts:
  # Mount a presisted cloud storage that will be used as the data directory.
  # (to store train datasets trained models)
  # See https://skypilot.readthedocs.io/en/latest/reference/storage.html for details.
  /data:
    name: epigen-llm-tuner-data  # Make sure this name is unique or you own this bucket. If it does not exists, SkyPilot will try to create a bucket with this name.
    store: s3  # Could be either of [s3, gcs]

    mode: MOUNT

# Clone the LLaMA-LoRA Tuner repo and install its dependencies.
setup: |
  conda create -q python=3.8 -n llm-tuner -y
  conda activate llm-tuner

  # Clone the LLaMA-LoRA Tuner repo and install its dependencies
  [ ! -d llm_tuner ] && git clone https://github.com/zetavg/LLaMA-LoRA-Tuner.git llm_tuner
  echo 'Installing dependencies...'
  pip install -r llm_tuner/requirements.lock.txt

  # Optional: install wandb to enable logging to Weights & Biases
  pip install wandb

  # Optional: patch bitsandbytes to workaround error "libbitsandbytes_cpu.so: undefined symbol: cget_col_row_stats"
  BITSANDBYTES_LOCATION="$(pip show bitsandbytes | grep 'Location' | awk '{print $2}')/bitsandbytes"
  [ -f "$BITSANDBYTES_LOCATION/libbitsandbytes_cpu.so" ] && [ ! -f "$BITSANDBYTES_LOCATION/libbitsandbytes_cpu.so.bak" ] && [ -f "$BITSANDBYTES_LOCATION/libbitsandbytes_cuda121.so" ] && echo 'Patching bitsandbytes for GPU support...' && mv "$BITSANDBYTES_LOCATION/libbitsandbytes_cpu.so" "$BITSANDBYTES_LOCATION/libbitsandbytes_cpu.so.bak" && cp "$BITSANDBYTES_LOCATION/libbitsandbytes_cuda121.so" "$BITSANDBYTES_LOCATION/libbitsandbytes_cpu.so"
  conda install -q cudatoolkit -y

  echo 'Dependencies installed.'

  # Optional: Install and setup Cloudflare Tunnel to expose the app to the internet with a custom domain name
  [ -f /data/secrets/cloudflared_tunnel_token.txt ] && echo "Installing Cloudflare" && curl -L --output cloudflared.deb https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb && sudo dpkg -i cloudflared.deb && sudo cloudflared service uninstall || : && sudo cloudflared service install "$(cat /data/secrets/cloudflared_tunnel_token.txt | tr -d '\n')"

  # Optional: pre-download models
  echo "Pre-downloading base models so that you won't have to wait for long once the app is ready..."
  python llm_tuner/download_base_model.py --base_model_names='decapoda-research/llama-7b-hf,nomic-ai/gpt4all-j'

# Start the app. `hf_access_token`, `wandb_api_key` and `wandb_project` are optional.
run: |
  conda activate llm-tuner
  python llm_tuner/app.py \
    --data_dir='/data' \
    --hf_access_token="$([ -f /data/secrets/hf_access_token.txt ] && cat /data/secrets/hf_access_token.txt | tr -d '\n')" \
    --wandb_api_key="$([ -f /data/secrets/wandb_api_key.txt ] && cat /data/secrets/wandb_api_key.txt | tr -d '\n')" \
    --wandb_project='llm-tuner' \
    --timezone='Atlantic/Reykjavik' \
    --base_model='decapoda-research/llama-7b-hf' \
    --base_model_choices='decapoda-research/llama-7b-hf,nomic-ai/gpt4all-j,databricks/dolly-v2-7b' \
    --share

[Michaelvll]

Thanks for sharing this @nshaposh! We don’t use the ssh key metadata on aws, but directly add the key to the ~/.ssh/authorized_keys on the instance created by SkyPilot using the cloud-init.

skypilot/sky/templates/aws-ray.yml.j2

Lines 103 to 109 in 57cfa7c

	users:
	- name: skypilot:ssh_user
	shell: /bin/bash
	sudo: ALL=(ALL) NOPASSWD:ALL
	ssh_authorized_keys:
	- |-
	skypilot:ssh_public_key_content

Did you recall that there was some modification to your ~/.ssh folder you have been made, e.g. something that can cause the private and public keys ~/.ssh/sky-key and ~/.ssh/sky-key.pub of SkyPilot fail to match? Is it possible to have a try with the ssh command shown in the provision.log directly in your terminal with a -vvv added to see what the problem is?

If it is permission denied, could you try moving the two files to another place and sky launch another cluster and see if the problem fixes?

[nshaposh]

@Michaelvll Here is ssh output:

root@82068e909dca:/app# ssh -vvv -i ~/.ssh/sky-key ec2-user@34.200.237.170 
OpenSSH_9.2p1 Debian-2+deb12u1, OpenSSL 3.0.11 19 Sep 2023
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 34.200.237.170 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/root/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/root/.ssh/known_hosts2'
debug3: ssh_connect_direct: entering
debug1: Connecting to 34.200.237.170 [34.200.237.170] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug1: connect to address 34.200.237.170 port 22: Connection refused
ssh: connect to host 34.200.237.170 port 22: Connection refused

[Michaelvll]

ssh -vvv -i /.ssh/sky-key ec2-user@34.200.237.170
OpenSSH_9.2p1 Debian-2+deb12u1, OpenSSL 3.0.11 19 Sep 2023
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 34.200.237.170 is address
debug3: expanded UserKnownHostsFile '/.ssh/known_hosts' -> '/root/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/root/.ssh/known_hosts2'
debug3: ssh_connect_direct: entering
debug1: Connecting to 34.200.237.170 [34.200.237.170] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug1: connect to address 34.200.237.170 port 22: Connection refused
ssh: connect to host 34.200.237.170 port 22: Connection refused

Hmm, this is weird. It seems to be some firewall issue. Could you check if the security group assigned to the VM has the port 22 open to public?
It would be also good to make sure that your AWS account have enough permission to create security groups that has the right rules: https://skypilot.readthedocs.io/en/latest/cloud-setup/cloud-permissions/aws.html

If the problem keeps happening, please feel free to join our slack channel: http://slack.skypilot.co so we can help in an interactive way or even start a debugging session with you : )

[nshaposh]

@Michaelvll I have noticed that the subnet, where this instance was placed, doesn't have a route table. May be this is the reason. How can I control, from the skypilot side, which VPC cluster should use?

[nshaposh]

@Michaelvll When I specified the correct VPC in ~/.sky/config.yaml, everything worked. Thank you!