Skip to content

development.md

Latest commit

 

History

History
274 lines (215 loc) · 6.98 KB

development.md

File metadata and controls

274 lines (215 loc) · 6.98 KB

Flatline Prototype Development

This document provides instructions to locally test, build and deploy individual Flatline components.

Requirements

To develop Flatline locally, ensure submodules are initialized and fetched:

git clone --recurse-submodules [email protected]:mollyim/flatline-platform.git
# Alternatively, for an existing repository:
# git submodule update --init --recursive

Testing and building this project relies on Docker.

The Java compoments are intended to be built with the Temurin 24 JDK.

For the Maven builds to succeed, ensure that JAVA_HOME points to a valid Temurin 24 JDK installation.

Testing

Whisper Service

Requires a FoundationDB client.

cd flatline-whisper-service
./mvnw clean verify -e \
-pl '!integration-tests' -Dsurefire.failIfNoSpecifiedTests=false -Dtest=\
\!org.whispersystems.textsecuregcm.controllers.VerificationControllerTest,\
\!org.whispersystems.textsecuregcm.controllers.SubscriptionControllerTest,\
\!org.whispersystems.textsecuregcm.registration.IdentityTokenCallCredentialsTest

Integration tests are excluded as they require an existing environment in which to run.

Tests for features that are disabled for the prototype are be excluded.

Storage Service

cd flatline-storage-service
./mvnw clean test

Registration Service

cd flatline-registration-service
./mvnw clean test

Contact Discovery Service

To test C dependencies:

cd flatline-contact-discovery-service
make -C c docker_tests
make -C c docker_valgrinds

To run minimal tests without Intel SGX:

cd flatline-contact-discovery-service
./mvnw verify -Dtest=\
\!org.signal.cdsi.enclave.**,\
\!org.signal.cdsi.IntegrationTest,\
\!org.signal.cdsi.JsonMapperInjectionIntegrationTest,\
\!org.signal.cdsi.limits.redis.RedisLeakyBucketRateLimiterIntegrationTest,\
\!org.signal.cdsi.util.ByteSizeValidatorTest

To run all tests with Intel SGX:

cd flatline-contact-discovery-service
# Set up Intel SGX on Ubuntu 22.04.
sudo ./c/docker/sgx_runtime_libraries.sh
./mvnw verify

Calling Service

This component is built in Rust and requires its toolchain.

cd flatline-calling-service
cargo test

Building

For Java components, this stage will build and locally store container images with Jib, in addition to the JAR artifacts.

For other components, the provided Dockerfiles will be used instead.

Whisper Service

cd flatline-whisper-service
./mvnw clean deploy \
  -Pexclude-spam-filter -Denv=dev -DskipTests \
  -Djib.to.image="flatline-whisper-service:dev"

Storage Service

cd flatline-storage-service
./mvnw clean package \
  -Pdocker-deploy -Denv=dev -DskipTests \
  -Djib.to.image="flatline-storage-service:dev"

The env property is used as a prefix to fetch the relevant configuration files from storage-service/config.

Registration Service

cd flatline-registration-service
./mvnw clean package \
  -Denv=dev -DskipTests \
  -Djib.to.image="flatline-registration-service:dev"

As configured for this prototype, the verification code is always the last six digits of the phone number.

Contact Discovery Service

cd flatline-contact-discovery-service
./mvnw package \
  -Dpackaging=docker -DskipTests \
  -Djib.to.image="flatline-contact-discovery-service:dev"

Calling Service

cd flatline-calling-service
TARGET_CPU=skylake # The target CPU family used to run the component.
docker build . -f frontend/Dockerfile \
  --build-arg rust_flags=-Ctarget-cpu=$TARGET_CPU \
  -t flatline-calling-service-frontend:dev
docker build . -f backend/Dockerfile \
  --build-arg rust_flags=-Ctarget-cpu=$TARGET_CPU \
  -t flatline-calling-service-backend:dev

The component should run on the target CPU family and others that support its features.

Deploying to Kubernetes

Kubernetes expects container images to be served from a container registry.

You can deploy a simple container registry with the Distribution Registry container image.

For example, to deploy an insecure registry for local testing:

docker run -d \
  -p 5000:5000 \
  --restart=always \
  --name registry \
  -v /tmp/registry:/var/lib/registry \
  registry:3

When building with Maven, push the resulting container images to a registry. For example:

# Whisper Service
( 
  cd flatline-whisper-service && \
  ./mvnw -e \
    deploy \
    -Pexclude-spam-filter \
    -Denv=dev \
    -DskipTests \
    -Djib.goal=build \
    -Djib.to.image=localhost:5000/flatline-whisper-service:dev \
    -Djib.allowInsecureRegistries=true
)

# Storage Service
( 
  cd flatline-storage-service && \
  ./mvnw -e \
    clean package \
    -Pdocker-deploy \
    -Denv=dev \
    -DskipTests \
    -Djib.goal=build \
    -Djib.to.image=localhost:5000/flatline-storage-service:dev \
    -Djib.allowInsecureRegistries=true
)

# Registration Service
(
  cd flatline-registration-service && \
  ./mvnw -e \
    clean package \
    -Denv=dev \
    -DskipTests \
    -Djib.goal=build \
    -Djib.to.image=localhost:5000/flatline-registration-service:dev \
    -Djib.allowInsecureRegistries=true
)

# Contact Discovery Service
(
  cd flatline-contact-discovery-service && \
  ./mvnw -e \
  deploy \
  -Dpackaging=docker \
  -DskipTests \
  -Djib.to.image=localhost:5000/flatline-contact-discovery-service:dev \
  -Djib.allowInsecureRegistries=true
)

# Calling Service
(
  cd flatline-calling-service
  docker build . -f frontend/Dockerfile \
    --build-arg rust_flags=-Ctarget-cpu=skylake \
    -t localhost:5000/flatline-calling-service-frontend:dev
  docker push localhost:5000/flatline-calling-service-frontend:dev
  docker build . -f backend/Dockerfile \
    --build-arg rust_flags=-Ctarget-cpu=skylake \
    -t localhost:5000/flatline-calling-service-backend:dev
  docker push localhost:5000/flatline-calling-service-backend:dev
)

Once the images are on the registry, override the Helm image values to reference these images.

For example, to do this for every core Flatline component, create local.yaml with the following:

whisperService:
  image:
    repository: localhost:5000/flatline-whisper-service
    tag: dev
storageService:
  image:
    repository: localhost:5000/flatline-storage-service
    tag: dev
registrationService:
  image:
    repository: localhost:5000/flatline-registration-service
    tag: dev
contactDiscoveryService:
  image:
    repository: localhost:5000/flatline-contact-discovery-service
    tag: dev
callingServiceFrontend:
  image:
    repository: localhost:5000/flatline-calling-service-frontend
    tag: dev
callingServiceBackend:
  image:
    repository: localhost:5000/flatline-calling-service-backend
    tag: dev

Finally, upgrade the Helm release to use these custom image values:

helm upgrade -f local.yaml $HELM_RELEASE ./charts/flatline