09_deployment___environment_setup

Chapter 9: Deployment & Environment Setup

In Chapter 8: Data Models (Persistence Layer), we learned how EMBArk meticulously stores and organizes all its data, from user accounts to complex analysis results. But having all these blueprints and data structures is just one part of the equation. How do we actually take all the different components of EMBArk – the web application, the database, the EMBA analysis tool, and all the background services – and get them up and running as a complete, functional system on a computer or server?

Imagine you have all the perfectly designed parts for a sophisticated machine: the engine, the electronics, the chassis, the tires. Each part is flawless. Now, you need a detailed instruction manual and all the right tools to assemble them into a working vehicle. Without this, your perfect parts are just a pile of components.

This is exactly the problem "Deployment & Environment Setup" solves for EMBArk. It's the blueprint and instruction manual for building and setting up EMBArk in different environments. Whether you want a quick setup on your local machine for development, or a robust, secure deployment for a team of users in production, this system handles everything. It ensures all necessary software (dependencies) is installed, databases are configured, web servers are set up, and all parts of EMBArk can communicate and run smoothly, making it a living, breathing application wherever it's deployed.

Understanding the Key Concepts

Getting EMBArk ready to run involves understanding a few important concepts:

1. Dependencies: "The Ingredients"

Just like a recipe needs specific ingredients, EMBArk needs other software to work. These are called dependencies. They include:

Python: The programming language EMBArk is built with.
Django: The web framework that powers EMBArk's interface.
MySQL: The database where EMBArk stores all its data.
Redis: A super-fast data store used for caching and managing background tasks (like Celery queues).
Docker: A tool to run applications in isolated "containers," making it easier to manage MySQL and Redis without messing up your main system.
EMBA: The core firmware analysis tool, which EMBArk controls.

2. Configuration: "The Recipe Adjustments"

Once you have the ingredients, you need to tell EMBArk how to use them. This is configuration. It involves setting up things like:

Database connection details: How EMBArk finds and talks to MySQL (username, password, host).
Secret keys: Unique codes for security purposes.
File paths: Where to store uploaded firmware, logs, and static web files.
Debug mode: Whether to show detailed error messages (good for development, bad for production).

3. Environment Setup: "The Kitchen Layout"

This is about setting up the overall environment where EMBArk will run. It includes:

Directories: Creating specific folders for logs, media, and the application code.
Users & Permissions: Setting up system users (like www-embark) and giving them the right to access files and run certain commands, especially EMBA securely.
Web Server Integration: Connecting the Django web application to a web server (like Apache or Daphne) so users can access it through a browser.

4. Startup Scripts: "The Automated Checklist"

Instead of manually doing all these steps, EMBArk provides startup scripts that automate the entire process. These scripts are like a detailed, step-by-step checklist that ensures everything is installed and configured correctly with just a few commands.

Solving Our Use Case: Getting EMBArk Up and Running

Let's imagine you've just downloaded EMBArk's code, and you want to get it running on your machine. You'll use the provided installer.sh script for the initial setup and then run-server.sh (or debug-server-start.sh for development) to start the application.

Step 1: Running the Installer Script

The installer.sh script is your starting point. It's designed to set up all the necessary system-level dependencies and basic configurations.

Here's how you'd typically run it for a default (production-like) installation:

sudo ./installer.sh -d

sudo: You need administrator (root) privileges because the installer installs system-level packages and creates system users.
./installer.sh: This is the script itself.
-d: This option tells the installer to perform a "default" installation, which includes EMBA and sets up EMBArk for production use. (For development, you might use -F instead).

What happens: The installer.sh script will perform a series of actions:

Install system dependencies: It checks and installs tools like Docker, Python, Git, MySQL client libraries, etc., based on your operating system.
Create system users and permissions: It sets up a dedicated user (e.g., www-embark) for the EMBArk web server and grants necessary sudo permissions for EMBA execution.
Prepare directories: It creates essential folders like /var/www/media for uploaded files and /var/www/emba_logs for analysis outputs.
Generate configuration: It creates a .env file (environment variables) with random passwords for the database and a Django secret key, ensuring a secure default setup.
Set up Docker containers: It uses docker-compose to start the MySQL database and Redis server in isolated containers.
Install Python dependencies: It installs all the Python libraries EMBArk needs using pipenv.
Generate SSL certificates: For secure communication, it creates self-signed SSL certificates.
Configure local DNS: It adds embark.local to your /etc/hosts file so you can access EMBArk easily by name.

A crucial part of the installer is the install_deps function, which makes sure all system-level tools are in place:

# Simplified snippet from installer.sh

install_deps(){
  # This function checks for and installs required system packages.
  echo -e "\n${GREEN}""${BOLD}""Install debian packages for EMBArk installation""${NC}"
  apt-get update -y
  
  if ! command -v git > /dev/null ; then
    apt-get install -y git
  fi
  # ... (checks for wget, python3, gcc, pip, pipenv, docker, etc.) ...
  
  if ! command -v docker > /dev/null || ! command -v docker compose > /dev/null ; then
    # Commands to install Docker and Docker Compose
    apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
  fi
  # ... more dependency checks and installations ...
}

This function is your helpful assistant, ensuring your system has all the foundational software EMBArk needs.

The installer also prepares the .env file, which is crucial for configuration:

# Simplified snippet from installer.sh

write_env(){
  local RANDOM_PW # Random password for the database
  local DJANGO_SECRET_KEY # Secret key for Django

  # Logic to generate or retrieve existing secure credentials
  DJANGO_SECRET_KEY=$(python3 -c 'from django.core.management.utils import get_random_secret_key; print(get_random_secret_key())')
  RANDOM_PW=$(openssl rand -base64 12)

  echo -e "${ORANGE}""${BOLD}""Creating a EMBArk configuration file .env""${NC}"
  {
    echo "DATABASE_NAME=embark"
    echo "DATABASE_USER=embark"
    echo "DATABASE_PASSWORD=${RANDOM_PW}" # Secure random password!
    echo "DATABASE_HOST=172.22.0.5" # IP of the Dockerized MySQL
    echo "DATABASE_PORT=3306"
    echo "REDIS_HOST=172.22.0.8"   # IP of the Dockerized Redis
    echo "REDIS_PORT=7777"
    echo "SECRET_KEY=${DJANGO_SECRET_KEY}" # Django's secret key
    # ... other environment variables like superuser credentials ...
  } > .env # Writes all these variables into the .env file
  chmod 600 .env # Make sure only the owner can read/write this file
}

This write_env function ensures that your EMBArk installation has a secure and unique configuration, with all sensitive details stored in the .env file.

Step 2: Running the Server Startup Script

After the installer.sh completes (which might ask you to run docker compose up -d manually for the first time), you'll use run-server.sh to start the actual EMBArk application.

sudo ./run-server.sh

sudo: Again, root privileges are often needed, especially for managing system services (like Apache or systemctl) and binding to standard ports (like 80).
./run-server.sh: This script brings up all parts of the EMBArk web application.

What happens: The run-server.sh script is much more involved, as it launches all the EMBArk components:

Starts Docker containers: Ensures MySQL and Redis are running via docker-compose.
Checks EMBA: Verifies that EMBA is correctly installed and configured.
Synchronizes code: Copies the latest EMBArk application code and EMBA's external modules to the /var/www/embark directory, which is where the production web server will run it.
Database setup: Runs Django database migrations to ensure the database schema matches the Chapter 8: Data Models (Persistence Layer) definitions. It also creates a default superuser and loads initial data (fixtures) like permission groups (Chapter 1: User Authentication & Authorization).
Collects static files: Gathers all CSS, JavaScript, and image files into a single directory for efficient serving by the web server.
Starts web servers:
- Launches mod_wsgi (an Apache module) to serve the main Django web application over HTTP (port 80).
- Launches daphne (an ASGI server) to handle WebSocket connections (port 8001), essential for Chapter 3: Real-time Progress Monitoring.
Starts background task processors:
- Launches celery worker to process asynchronous tasks, including those for Chapter 6: Worker Node Orchestration.
- Launches runapscheduler for scheduled tasks like system resource tracking and cleanup (Chapter 7: Background Task Execution).
Starts a systemctl daemon: If running as a system service.

After this script finishes, EMBArk will be accessible in your web browser, typically at http://embark.local (or http://YOUR_IP_ADDRESS).

The core of starting EMBArk is typically managed by runmodwsgi and daphne for the web interface, and celery and runapscheduler for background tasks:

# Simplified snippet from run-server.sh

# Start Apache with mod_wsgi for Django web application
"${PIPENV_COMMAND}" run ./manage.py runmodwsgi \
--host "${BIND_IP}" --port="${HTTP_PORT}" \
--url-alias /static/ /var/www/static/ \
--url-alias /media/ /var/www/media/ \
--processes 4 --threads 4 \
--server-name "embark.local" \
--include-file /var/www/conf/embark.conf & # Custom Apache config

# Start daphne (ASGI server) for WebSockets
cd /var/www/embark && sudo -u www-embark "${PIPENV_COMMAND}" run daphne \
-e ssl:8000:privateKey=/var/www/conf/cert/embark-ws.local.key:certKey=/var/www/conf/cert/embark-ws.local.crt \
-b "${BIND_IP}" -p 8001 -s embark-ws.local embark.asgi:application &

# Start celery worker for background tasks
sudo -u www-embark "${PIPENV_COMMAND}" run python -m celery -A embark worker --beat --scheduler django -l INFO --logfile=../logs/celery.log &
CELERY_PID=$! # Store PID to kill later

This part of the script meticulously launches all the necessary processes, ensuring EMBArk is fully operational.

Under the Hood: The Automated Assembly Line

Let's look at how all these pieces fit together behind the scenes, from initial setup to running application.

The Deployment Flow: A Simple Sequence

When you run the installer and then the server script, here's a simplified sequence of how EMBArk gets deployed:

sequenceDiagram
    participant User
    participant installer.sh
    participant Docker Compose
    participant OS System (Dependencies, Users, Services)
    participant run-server.sh
    participant EMBArk Web Server (WSGI/ASGI)
    participant Database (MySQL)
    participant Redis
    participant Background Tasks (Celery/APScheduler)
    participant EMBA Analysis Tool

    User->>installer.sh: sudo ./installer.sh -d
    installer.sh->>OS System (Dependencies, Users, Services): Installs Docker, Python, Git, creates www-embark user
    installer.sh->>Docker Compose: `docker compose pull` (downloads images)
    installer.sh->>Docker Compose: `docker compose up -d` (starts MySQL, Redis containers)
    installer.sh->>OS System (Dependencies, Users, Services): Sets up directories, SSL certs, local DNS, Python venv
    installer.sh-->>User: Installation complete. Run server script.

    User->>run-server.sh: sudo ./run-server.sh
    run-server.sh->>Docker Compose: `docker compose up -d` (ensures MySQL, Redis are running)
    run-server.sh->>Database (MySQL): Runs Django migrations
    run-server.sh->>EMBArk Web Server (WSGI/ASGI): Collects static files, starts Apache (mod_wsgi)
    run-server.sh->>EMBArk Web Server (WSGI/ASGI): Starts Daphne (ASGI) for WebSockets
    run-server.sh->>Background Tasks (Celery/APScheduler): Starts Celery worker, APScheduler
    Note over EMBA Analysis Tool: EMBA is assumed to be installed and ready.
    run-server.sh-->>User: EMBArk server running! Access via browser.

This diagram illustrates how the scripts orchestrate various components to bring EMBArk to life.

Key Components and Code Elements

Dockerfile - Container Blueprint (for Docker build): While installer.sh focuses on host-level setup, Dockerfile defines how EMBArk could be built inside a Docker container. Many of the installer.sh steps (like installing system deps and copying EMBA) are mirrored here if you were to deploy EMBArk itself in a container.

# Simplified snippet from Dockerfile
FROM kalilinux/kali-rolling:latest

# Install core system dependencies
RUN apt-get update && apt-get -y -q --no-install-recommends install wget \
    sudo \
    pipenv \
    # ... other build tools and python dev headers ...

# Copy EMBA (the analysis tool) into the container
COPY ./emba/ /var/www/emba/

# Create a user and grant sudo permissions for EMBA
RUN useradd www-embark -G sudo ... && \
    echo 'www-embark ALL=(ALL) NOPASSWD: /var/www/emba/emba' | EDITOR='tee -a' visudo

# Copy Python dependencies and install them
COPY ./Pipfile.lock /app/Pipfile.lock
RUN pipenv install

# Copy EMBArk application code
COPY ./embark /app/embark

EXPOSE 80
EXPOSE 8001
ENTRYPOINT  ["./entrypoint.sh"] # The script that starts the application inside the container

This Dockerfile is essentially a recipe for building a self-contained EMBArk environment within a Docker image. It shows the core steps of dependency installation, EMBA placement, user setup, Python environment creation, and application code copying.

docker-compose.yml - Service Orchestration: This file describes how to run and link multiple Docker containers together. It's used by both installer.sh and run-server.sh to manage the database and Redis server.

# Simplified snippet from docker-compose.yml
services:
  embark_db:
    container_name: embark_db
    image: mysql:latest
    env_file:
      - .env # Uses the .env file for database credentials
    expose:
      - "3306"
    networks:
      embark_backend:
        ipv4_address: "172.22.0.5" # Fixed IP address for easier configuration
    # ... volumes for persistent data ...

  embark_redis:
    container_name: embark_redis
    image: redis:5
    env_file:
      - .env
    command: --port 7777 # Custom port for Redis
    expose:
      - "7777"
    networks:
      embark_backend:
        ipv4_address: "172.22.0.8" # Fixed IP address
    # ... volumes for persistent data ...

networks:
  embark_backend:
    name: embark_backend
    driver: bridge
    ipam:
      config:
        - subnet: "172.22.0.0/16" # Defines the network for containers

docker-compose.yml sets up embark_db (MySQL) and embark_redis (Redis) as separate services. They share a custom Docker network (embark_backend) with static IP addresses, making it easy for EMBArk to find and connect to them using the DATABASE_HOST/REDIS_HOST variables from the .env file.

embark/embark/settings/deploy.py - Django's Brain: This is the main configuration file for the Django application in a production environment. It pulls sensitive details from the .env file created by installer.sh.

# Simplified snippet from embark/embark/settings/deploy.py
import os
from dotenv import load_dotenv

load_dotenv() # Load environment variables from .env file

DEBUG = False # IMPORTANT: Never True in production!
SECRET_KEY = os.environ.get('SECRET_KEY') # Loaded from .env

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': os.environ.get('DATABASE_NAME'),
        'USER': os.environ.get('DATABASE_USER'),
        'PASSWORD': os.environ.get("DATABASE_PASSWORD"),
        'HOST': os.environ.get('DATABASE_HOST'), # e.g., 172.22.0.5
        'PORT': os.environ.get('DATABASE_PORT'),
    },
}

EMBA_ROOT = os.path.join(BASE_DIR.parent, 'emba') # Path to EMBA tool
STATIC_ROOT = os.path.join(BASE_DIR.parent, 'static') # Where static files are collected
MEDIA_ROOT = os.path.join(BASE_DIR.parent, 'media') # Where uploaded files are stored

WSGI_APPLICATION = 'embark.wsgi.application' # Entry point for WSGI web servers (Apache)
ASGI_APPLICATION = 'embark.asgi.application' # Entry point for ASGI web servers (Daphne)

# Redis/Channel Layers for WebSockets and Celery
REDIS_HOST = os.environ.get('REDIS_HOST', '127.0.0.1')
REDIS_PORT = int(os.environ.get('REDIS_PORT', 6379))
CHANNEL_LAYERS = {
    'default': {
        'BACKEND': 'channels_redis.core.RedisChannelLayer',
        'CONFIG': { "hosts": [(REDIS_HOST, REDIS_PORT)], },
    },
}
CELERY_BROKER_URL = f"redis://{REDIS_HOST}:{REDIS_PORT}/0"

This settings.deploy.py file dictates how Django behaves in production. It defines the database connection using environment variables, specifies file paths for various assets, and configures the WSGI/ASGI application entry points and the Redis connections for Channels and Celery. There's also embark/embark/settings/dev.py for development, which sets DEBUG = True and uses different logging levels for easier debugging.

run-server.sh (or dev-tools/debug-server-start.sh) - The Startup Orchestrator: These scripts are the main entry points for starting the EMBArk application after installation. run-server.sh is for production, while debug-server-start.sh (which shares many helper functions) is for a local developer environment.

# Simplified snippet from run-server.sh (core logic)

# Import helper functions (e.g., check_db, sync_emba_forward)
import_helper

# Ensure Docker containers (MySQL, Redis) are up
check_db

# Sync EMBA files to the production directory
sync_emba_forward

# Navigate to the Django project directory
cd /var/www/embark/ || exit 1

# Activate Python virtual environment
source /var/www/.venv/bin/activate || exit 1

# Run Django database migrations
"${PIPENV_COMMAND}" run ./manage.py migrate

# Collect static files
"${PIPENV_COMMAND}" run ./manage.py collectstatic --no-input

# Start Apache with mod_wsgi for the main web application
"${PIPENV_COMMAND}" run ./manage.py runmodwsgi --host "${BIND_IP}" --port="${HTTP_PORT}" ... &

# Start Daphne for WebSocket connections
sudo -u www-embark "${PIPENV_COMMAND}" run daphne ... &

# Start Celery worker for background tasks
sudo -u www-embark "${PIPENV_COMMAND}" run python -m celery -A embark worker ... &

# Start APScheduler for scheduled tasks
"${PIPENV_COMMAND}" run ./manage.py runapscheduler ... &

This script encapsulates the entire sequence of operations needed to launch the EMBArk server, from checking dependencies and database health to starting all the web and background services. It makes sure everything is in its place and running correctly.

helper/helper_embark_general.sh - Shared Utility Functions: This file contains many reusable shell functions that both the installer.sh and run-server.sh scripts use.

# Simplified snippet from helper/helper_embark_general.sh

check_db() {
  # This function verifies that the MySQL and Redis Docker containers are running
  # and that EMBArk can successfully connect to the MySQL database.
  echo -e "\\n${ORANGE}""${BOLD}""checking database""${NC}"
  if docker compose -f ./docker-compose.yml up -d ; then
    echo -e "${GREEN}""${BOLD}""Finished setup mysql and redis docker images""${NC}"
  else
    echo -e "${ORANGE}""${BOLD}""Failed setup mysql and redis docker images""${NC}"
    exit 1
  fi
  # ... logic to test MySQL connection ...
}

sync_emba_forward() {
  # This function copies the EMBA analysis tool files from the EMBArk source
  # directory to the /var/www/emba/ directory for the production server.
  echo -e "\\n${ORANGE}""${BOLD}""Synchronising EMBA""${NC}"
  if [[ ! -d "/var/www/emba/" ]]; then
    git clone "${lEMBA_URL:='https://github.com/e-m-b-a/emba'}" /var/www/emba
  fi
  rsync -r -u --progress --chown=www-embark:sudo "${EMBARK_BASEDIR:-${PWD}}"/emba/external /var/www/emba/
}

These helper functions abstract away common tasks, ensuring consistency and reusability across different setup and startup scripts.

Conclusion

Deployment & Environment Setup is the crucial final step in bringing EMBArk to life. By leveraging a robust installer, docker-compose for service orchestration, detailed Django settings, and comprehensive startup scripts, EMBArk provides a streamlined process for getting its complex ecosystem up and running. This ensures that all components – from the user interface to the deep firmware analysis engine – are correctly configured and interconnected, providing a stable and efficient platform for security analysis.

With EMBArk now deployed and ready, you can fully utilize all the functionalities we've explored in previous chapters.

_{^{Generated by AI Codebase Knowledge Builder.}} _{^{References: [1], [2], [3], [4], [5], [6], [7], [8], [9]}}

EMBArk - firmware security scanning at its best

Sponsor EMBA and EMBArk:

The EMBA environment is free and open source!

We put a lot of time and energy into these tools and related research to make this happen. It's now possible for you to contribute as a sponsor!

If you like EMBArk you have the chance to support future development by becoming a Sponsor

Thank You ❤️ Get a Sponsor

EMBArk - firmware security scanning at its best

Home
- Publications
- Referring sites and talks
- Your Feedback
Feature overview
- Architecture
- Administration
- Tracking
- Web interface
- API
- Orchestrator
Installation
- Prerequisites
- Automated installation
- Updating_EMBA
- Manual installation
Usage
- Expert-Mode
EMBArk-book
Development
- Overview
- Dev-mode
- Contributing
- Tec-Doc(outdated!)
- Internals
- Testing
FAQ
Sponsoring EMBArk
AMOS project archive
EMBA firmware scanning backend

Uh oh!

09_deployment___environment_setup

Chapter 9: Deployment & Environment Setup

Understanding the Key Concepts

1. Dependencies: "The Ingredients"

2. Configuration: "The Recipe Adjustments"

3. Environment Setup: "The Kitchen Layout"

4. Startup Scripts: "The Automated Checklist"

Solving Our Use Case: Getting EMBArk Up and Running

Step 1: Running the Installer Script

Step 2: Running the Server Startup Script

Under the Hood: The Automated Assembly Line

The Deployment Flow: A Simple Sequence

Key Components and Code Elements

Conclusion

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally