Skip to content

refactor: remove container from list of run image types #2178

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 27 additions & 1 deletion docs/source/distributions/building_distro.md
Original file line number Diff line number Diff line change
Expand Up @@ -260,7 +260,33 @@ Containerfile created successfully in /tmp/tmp.viA3a3Rdsg/ContainerfileFROM pyth
You can now edit ~/meta-llama/llama-stack/tmp/configs/ollama-run.yaml and run `llama stack run ~/meta-llama/llama-stack/tmp/configs/ollama-run.yaml`
```

After this step is successful, you should be able to find the built container image and test it with `llama stack run <path/to/run.yaml>`.
After this step is successful, you should be able to find the built container image and test it with:
```
# This docker command will run the Llama Stack server based on the image built before
# The following is a list of docker flags and their use in running the server:
# -p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT: Maps the container port to the host port for accessing the server
# -v ~/.llama:/root/.llama: Mounts the local .llama directory to persist configurations and data
# -v <path/to/run.yaml>:/app/run.yaml: Mounts the run configuration file into the container
# --entrypoint python: Specifies the entry point to run Python directly
# localhost/distribution-ollama:dev: The name and tag of the container image to run
# -m llama_stack.distribution.server.server: The Python module to execute
# --config /app/run.yaml: Path to the configuration file inside the container
# --port $LLAMA_STACK_PORT: Port number for the server to listen on
# --env INFERENCE_MODEL=$INFERENCE_MODEL: Sets the model to use for inference
# --env OLLAMA_URL=http://host.docker.internal:11434: Configures the URL for the Ollama service

docker run -it \
-p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \
-v ~/.llama:/root/.llama \
-v <path/to/run.yaml>:/app/run.yaml \
--entrypoint python \
localhost/distribution-ollama:dev \
-m llama_stack.distribution.server.server \
--config /app/run.yaml \
--port $LLAMA_STACK_PORT \
--env INFERENCE_MODEL=$INFERENCE_MODEL \
--env OLLAMA_URL=http://host.docker.internal:11434
```
:::

::::
Expand Down
2 changes: 1 addition & 1 deletion llama_stack/cli/stack/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ def _add_arguments(self):
"--image-type",
type=str,
help="Image Type used during the build. This can be either conda or container or venv.",
choices=[e.value for e in ImageType],
choices=[e.value for e in ImageType if e.value != "container"],
)

# If neither image type nor image name is provided, but at the same time
Expand Down
82 changes: 25 additions & 57 deletions llama_stack/distribution/start_stack.sh
Original file line number Diff line number Diff line change
Expand Up @@ -132,63 +132,31 @@ if [[ "$env_type" == "venv" || "$env_type" == "conda" ]]; then
$env_vars \
$other_args
elif [[ "$env_type" == "container" ]]; then
set -x

# Check if container command is available
if ! is_command_available $CONTAINER_BINARY; then
printf "${RED}Error: ${CONTAINER_BINARY} command not found. Is ${CONTAINER_BINARY} installed and in your PATH?${NC}" >&2
exit 1
fi

if is_command_available selinuxenabled &> /dev/null && selinuxenabled; then
# Disable SELinux labels
CONTAINER_OPTS="$CONTAINER_OPTS --security-opt label=disable"
fi

mounts=""
if [ -n "$LLAMA_STACK_DIR" ]; then
mounts="$mounts -v $(readlink -f $LLAMA_STACK_DIR):/app/llama-stack-source"
fi
if [ -n "$LLAMA_CHECKPOINT_DIR" ]; then
mounts="$mounts -v $LLAMA_CHECKPOINT_DIR:/root/.llama"
CONTAINER_OPTS="$CONTAINER_OPTS --gpus=all"
fi

if [ -n "$PYPI_VERSION" ]; then
version_tag="$PYPI_VERSION"
elif [ -n "$LLAMA_STACK_DIR" ]; then
version_tag="dev"
elif [ -n "$TEST_PYPI_VERSION" ]; then
version_tag="test-$TEST_PYPI_VERSION"
# Determine the internal container address based on container runtime
if [ "$CONTAINER_BINARY" = "docker" ]; then
internal_host="host.docker.internal"
elif [ "$CONTAINER_BINARY" = "podman" ]; then
internal_host="host.containers.internal"
else
if ! is_command_available jq; then
echo -e "${RED}Error: jq not found" >&2
exit 1
fi
URL="https://pypi.org/pypi/llama-stack/json"
version_tag=$(curl -s $URL | jq -r '.info.version')
internal_host="localhost"
fi

# Build the command with optional yaml config
cmd="$CONTAINER_BINARY run $CONTAINER_OPTS -it \
-p $port:$port \
$env_vars \
$mounts \
--env LLAMA_STACK_PORT=$port \
--entrypoint python \
$container_image:$version_tag \
-m llama_stack.distribution.server.server"

# Add yaml config if provided, otherwise use default
if [ -n "$yaml_config" ]; then
cmd="$cmd -v $yaml_config:/app/run.yaml --config /app/run.yaml"
else
cmd="$cmd --config /app/run.yaml"
fi

# Add any other args
cmd="$cmd $other_args"

# Execute the command
eval $cmd
echo -e "${RED}Warning: Llama Stack no longer supports running Container.${NC}"
echo -e "Please use one of the following alternatives:"
echo -e "1. Use venv or conda environments"
echo -e "2. Run the container directly with Docker/Podman"
echo -e "\nExample $CONTAINER_BINARY command for ollama distribution:"
echo -e "$CONTAINER_BINARY run \\"
echo -e " -it \\"
echo -e " --network host \\"
echo -e " -p $port:$port \\"
echo -e " -v <path_to_yaml_config>:/app/run.yaml \\"
echo -e " --entrypoint python \\"
echo -e " localhost/distribution-ollama:<version> \\"
echo -e " -m llama_stack.distribution.server.server \\"
echo -e " --config /app/run.yaml \\"
echo -e " --env INFERENCE_MODEL=\"llama3.2:3b\" \\"
echo -e " --env LLAMA_STACK_PORT=<port> \\"
echo -e " --env OLLAMA_URL=\"http://$internal_host:11434\""
echo -e "\nExiting..."
exit 1
fi
7 changes: 2 additions & 5 deletions llama_stack/distribution/utils/exec.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,8 @@

def formulate_run_args(image_type, image_name, config, template_name) -> list:
env_name = ""
if image_type == LlamaStackImageType.CONTAINER.value:
env_name = (
f"distribution-{template_name}" if template_name else (config.container_image if config else image_name)
)
elif image_type == LlamaStackImageType.CONDA.value:

if image_type == LlamaStackImageType.CONDA.value:
current_conda_env = os.environ.get("CONDA_DEFAULT_ENV")
env_name = image_name or current_conda_env
if not env_name:
Expand Down
Loading