Skip to content

azure_vms.md

Latest commit

 

History

History
229 lines (164 loc) · 10.7 KB

azure_vms.md

File metadata and controls

229 lines (164 loc) · 10.7 KB
Raw

Azure Virtual Machines

The Azure Virtual Machines client of Lithops can provide a truly serverless user experience on top of Azure VMs where Lithops creates new Virtual Machines (VMs) dynamically at runtime and scales Lithops jobs against them (Create & Reuse modes). Alternatively, Lithops can start and stop an existing VM instance (Consume Mode).

Installation

  1. Install Microsoft Azure backend dependencies:
python3 -m pip install lithops[azure]

  1. Install Azure CLI

  2. Sign in with the Azure CLI:

az login

  1. Create a Resource Group and a Storage Account:

    Option 1:

    1. Access the Azure portal Resource Groups and create a new Resource group named LithopsResourceGroup (or similar) in your preferred region. If you already have a resource group, omit this step.

    2. Access the Azure portal Storage Accounts and create a new Storage Account with a unique name, for example: lithops0sa25s1. If you already have a storage account, omit this step.

    Option 2:

    1. Create a Resource Group in a specific location. If you already have a resource group, omit this step.
    az group create --name LithopsResourceGroup --location westeurope
    1. Create a Storage Account with a unique name. If you already have a storage account, omit this step.
    storage_account_name=lithops$(openssl rand -hex 3)
echo $storage_account_name
az storage account create --name $storage_account_name --location westeurope \
     --resource-group LithopsResourceGroup --sku Standard_LRS

Choose an operating system image for the VM

  • Option 1: By default, Lithops uses an Ubuntu 24.04 image. In this case, no further action is required and you can continue to the next step. Lithops will install all required dependencies in the VM by itself. Note that this can take about 3 minutes to complete all installations.

  • Option 2: Alternatively, you can use a pre-built custom image that will greatly improve VM creation time for Lithops jobs. To benefit from this approach, navigate to runtime/azure_vms, and follow the instructions.

Create and reuse modes

In the create mode, Lithops will automatically create new worker VM instances at runtime, scale Lithops jobs against the generated VMs, and automatically delete the VMs when the job is completed. Alternatively, you can set the reuse mode to keep the started worker VMs running and reuse them for further executions. In the reuse mode, Lithops checks all the available worker VMs and starts new workers only if necessary.

Lithops configuration for the create or reuse modes

Edit your Lithops config and add the relevant keys:

    lithops:
        backend: azure_vms

    azure:
        resource_group: <RESOURCE_GROUP_NAME>
        region: <LOCATION>
        subscription_id: <SUBSCRIPTION_ID>

    azure_vms:
        exec_mode: reuse

Azure

Group Key Default Mandatory Additional info
azure resource_group yes Name of a resource group, for example: LithopsResourceGroup
azure region yes Location of the resource group, for example: westeurope, westus2, etc
azure subscription_id yes Subscription ID from your account. Find it here

Azure VMs - Create and Reuse Modes

Group Key Default Mandatory Additional info
azure_vms region no Azure location for deploying the VMs. For example: westeurope, westus2, etc. Lithops will use the region set under the azure section if it is not set here
azure_vms image_id Canonical:ubuntu-24_04-lts:server:latest no Image ID. ARM resource identifier
azure_vms ssh_username ubuntu no Username to access the VM
azure_vms ssh_password no Password for accessing the worker VMs. If not provided, it is created randomly
azure_vms ssh_key_filename ~/.ssh/id_rsa no Path to the ssh key file provided to access the VPC. It will use the default path if not provided
azure_vms master_instance_type Standard_B1s no Profile name for the master VM
azure_vms worker_instance_type Standard_B2s no Profile name for the worker VMs
azure_vms delete_on_dismantle False no Delete the worker VMs when they are stopped. Master VM is never deleted when stopped. True is NOT YET SUPPORTED
azure_vms max_workers 100 no Max number of workers per FunctionExecutor()
azure_vms worker_processes AUTO no Number of parallel Lithops processes in a worker. This is used to parallelize function activations within the worker. By default it detects the amount of CPUs in the worker_instance_type VM
azure_vms runtime python3 no Runtime name to run the functions. Can be a container image name. If not set Lithops will use the default python3 interpreter of the VM
azure_vms auto_dismantle True no If False then the VM is not stopped automatically.
azure_vms soft_dismantle_timeout 300 no Time in seconds to stop the VM instance after a job completed its execution
azure_vms hard_dismantle_timeout 3600 no Time in seconds to stop the VM instance after a job started its execution
azure_vms exec_mode reuse no One of: consume, create or reuse. If set to create, Lithops will automatically create new VMs for each map() call based on the number of elements in iterdata. If set to reuse, Lithops will try to reuse running workers if they exist
azure_vms extra_apt_packages [] no Extra Debian/Ubuntu packages on master/worker VMs during setup (YAML list or space-separated string)
azure_vms extra_python_packages [] no Extra pip packages on master/worker VMs after Lithops (YAML list or space-separated string)

Consume mode

In this mode, Lithops can start and stop an existing VM, and deploy an entire job to that VM. The partition logic in this scenario is different from the create/reuse modes, since the entire job is executed in the same VM.

Lithops configuration for the consume mode

Edit your Lithops config and add the relevant keys:

    lithops:
        backend: azure_vms

    azure:
        resource_group: <RESOURCE_GROUP_NAME>
        region: <LOCATION>
        subscription_id: <SUBSCRIPTION_ID>

    azure_vms:
        exec_mode: consume
        instance_name: <VM_NAME>
        ssh_username: <SSH_USERNAME>
        ssh_key_filename: <SSH_KEY_PATH>

Azure

Group Key Default Mandatory Additional info
azure resource_group yes Name of a resource group, for example: LithopsResourceGroup
azure region yes Location of the resource group, for example: westeurope, westus2, etc
azure subscription_id yes Subscription ID from your account. Find it here

Azure VMs - Consume Mode

Group Key Default Mandatory Additional info
azure_vms instance_name yes Virtual server instance name. The instance must exist in your resource group
azure_vms ssh_username ubuntu yes Username to access the VM. It will use ubuntu if not provided
azure_vms ssh_key_filename ~/.ssh/id_rsa yes Path to the ssh key file provided to create the VM. It will use the default path if not provided
azure_vms region no Location of the resource group, for example: westeurope, westus2, etc. Lithops will use the region set under the azure section if it is not set here
azure_vms worker_processes AUTO no Number of parallel Lithops processes in a worker. This is used to parallelize function activations within the worker. By default it detects the amount of CPUs in the VM
azure_vms runtime python3 no Runtime name to run the functions. Can be a container image name. If not set, Lithops will use the default python3 interpreter of the VM
azure_vms auto_dismantle True no If False then the VM is not stopped automatically.
azure_vms soft_dismantle_timeout 300 no Time in seconds to stop the VM instance after a job completed its execution
azure_vms hard_dismantle_timeout 3600 no Time in seconds to stop the VM instance after a job started its execution

Test Lithops

Once you have your compute and storage backends configured, you can run a Hello World function with:

lithops hello -b azure_vms -s azure_storage

Viewing the execution logs

You can view the function executions logs in your local machine using the lithops client:

lithops logs poll

VM Management

Lithops for Azure VMs follows a Master-Worker architecture (1:N).

All the VMs, including the master VM, are automatically stopped after a configurable timeout (see hard/soft dismantle timeouts).

You can login to the master VM and get a live ssh connection with:

lithops attach -b azure_vms

The master and worker VMs contain the Lithops service logs in /tmp/lithops-root/*-service.log

To list all the available workers in the current moment, use the next command:

lithops worker list -b azure_vms

You can also list all the submitted jobs with:

lithops job list -b azure_vms

You can delete all the workers with:

lithops clean -b azure_vms -s azure_storage

You can delete all the workers including the Master VM with the --all flag:

lithops clean -b azure_vms -s azure_storage --all

Architecture diagram

Lithops creates a Virtual Network and subnet in your resource group (unless you provide existing ones). The master VM gets a static public IP; worker VMs use private IPs only and reach the internet through Azure's default subnet outbound routing.

flowchart TB
  subgraph azure [Azure resource group / region]
    VNET["Virtual Network lithops-vnet-XXXXXX\n10.0.0.0/16"]
    SUB["Subnet lithops-vnet-XXXXXX-subnet\n10.0.0.0/24"]
    NSG["Network Security Group lithops-security-group\nSSH :22 from internet\n:8080/:8081/:6379 within subnet"]
    PIP["Static Public IP lithops-vnet-XXXXXX-ip"]
    M["Master VM lithops-master-XXXXXX\nNIC + public IP"]
    W["Worker VMs lithops-worker-*\nNIC, private IP only"]
  end
  LAPTOP["Your laptop"] -->|SSH :22| PIP
  PIP --> M
  LAPTOP -->|HTTP :8080 via SSH tunnel| M
  M -->|Redis :6379 + HTTP :8081| W
  M --> SUB
  W --> SUB
  SUB --> VNET
  W -->|egress| INTERNET[Internet apt/docker/pip]
  NSG -.-> M
  NSG -.-> W
Loading