Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/notify_migrations_on_pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,4 @@ jobs:
uses: equinor/armada/.github/workflows/notify_migrations_on_pr.yml@main
with:
stage_one_workflow_run_id: ${{ github.event.workflow_run.id }}
migration_docs_url: https://github.com/equinor/flotilla/tree/main/backend#Database-model-and-EF-Core
migration_docs_url: https://github.com/equinor/flotilla/tree/main/backend#database-migrations-ef-core
2 changes: 0 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,13 @@ frontend/out/
*.log
*.vs
*.code-workspace
*.editorconfig

# Environment variables
*.env

# Backend
backend/*/bin
backend/*/obj
backend/api/Database/Maps/*
backend/backend.sln.DotSettings.user

# Broker
Expand Down
128 changes: 59 additions & 69 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,111 +4,101 @@
[![Frontend](https://github.com/equinor/flotilla/actions/workflows/lint_frontend_package.yml/badge.svg)](https://github.com/equinor/flotilla/actions/workflows/lint_frontend_package.yml)
[![Codacy Badge](https://app.codacy.com/project/badge/Grade/0b37a44f66044dbc81fff906344b476e)](https://www.codacy.com/gh/equinor/flotilla/dashboard?utm_source=github.com&utm_medium=referral&utm_content=equinor/flotilla&utm_campaign=Badge_Grade)

Flotilla is the main point of access for operators to interact with multiple robots in multiple facilities.
The application consists of a [frontend](frontend) in React, a [backend](backend) in ASP.NET, and a Mosquitto MQTT [Broker](broker).
Flotilla is the main point of access for operators to interact with multiple robots in multiple facilities.
The application consists of a [frontend](frontend) in React, a [backend](backend) in ASP.NET, and a Mosquitto MQTT [broker](broker).

## Deployments

We currently have 3 environments (Development, Staging, and Production) deployed to AKS under `robotics.equinor.com`.

| Environment | Deployment | Status |
| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Development | [Frontend](https://flotilla.dev.robotics.equinor.com/)<br>[Backend](https://flotilla.dev.robotics.equinor.com/api/swagger) | [![Dev](https://github.com/equinor/flotilla/actions/workflows/deploy_to_development.yml/badge.svg)](https://github.com/equinor/flotilla/actions/workflows/deploy_to_development.yml) |
| Staging | [Frontend](https://flotilla.staging.robotics.equinor.com/)<br>[Backend](https://flotilla.staging.robotics.equinor.com/api/swagger) | [![Staging](https://github.com/equinor/flotilla/actions/workflows/deploy_to_staging.yml/badge.svg)](https://github.com/equinor/flotilla/actions/workflows/deploy_to_staging.yml) |
| Production | [Frontend](https://flotilla.robotics.equinor.com/)<br>[Backend](https://flotilla.robotics.equinor.com/api/swagger) | [![Production](https://github.com/equinor/flotilla/actions/workflows/promote_to_production.yml/badge.svg)](https://github.com/equinor/flotilla/actions/workflows/promote_to_production.yml) |

## Setup

For development, please fork the repository. Then, clone the repository:

```
git clone https://github.com/equinor/flotilla
```

Please see separate installation guides for the [frontend](frontend), [backend](backend), and [Broker](broker). For the environment setup, either run the script as described below or set it up manually as described in each component's guide.
## Prerequisites

### Automatic environment setup

Run the [setup.sh](./setup.sh) to automatically set up your dev environment for the components.
This script will ask you for the `Client Secret` for the backend and the `MQTT broker server key` for the MQTT broker.

## Run with make
Common commands for the project are in Makefiles. See [frontend makefile](./frontend/Makefile), [backend makefile](./backend/Makefile) and [root makefile](./Makefile).
This requires the CLI program `make`.

Usage:
```bash
make <command-in-makefile> # for example: make run
```
| Tool | Version | Needed for |
| ------------------------------------------------------------------------------------------------------------ | ------- | -------------------------------- |
| [Docker](https://docs.docker.com/engine/install/) and [Docker Compose](https://docs.docker.com/compose/install/) | latest | Running the full stack |
| [.NET SDK](https://dotnet.microsoft.com/download) | 10.x | Running the backend directly |
| [Node.js](https://github.com/nodesource/distributions) | 24.x | Running the frontend directly |
| [pnpm](https://pnpm.io/installation) | latest | Frontend package management |
| `make` | any | The shorthand commands below |

<details>
<summary>Installation MacOS</summary>
<summary>Installing make on MacOS</summary>

```bash
brew install make
```

</details>
<details>
<summary>Installation Windows</summary>
<summary>Installing make on Windows</summary>

```bash
choco install make
```
</details>

## Run with Docker
</details>

Install [Docker](https://docs.docker.com/engine/install/ubuntu/) and [Docker Compose](https://docs.docker.com/compose/install/).
## Quick start

Build the Docker container:
For development, fork the repository first. Then clone it:

```
docker compose build
```bash
git clone https://github.com/equinor/flotilla
cd flotilla
```

Set up a .env file in the backend directory with the following environment variables:
Create the local configuration files:

```
ASPNETCORE_ENVIRONMENT=Local
Local__DevUserId=
SARA__BaseUrl=https://localhost:8100/
SARA__Scopes__0=
```bash
./setup.sh
```

Start Flotilla by running:
The script creates `frontend/.env`, `backend/api/.env`, and `broker/.env` from their `.env.example` files. It prompts you for the **MQTT broker server key**, which is found in our key vault. Everything else can be configured manually afterwards — see [Configuration](#configuration).

```
docker compose up
Start the full stack:

```bash
make compose # docker compose up --build
```

or
| Service | URL |
| ---------------- | --------------------------------------- |
| Frontend | <http://localhost:3001> |
| Backend Swagger | <http://localhost:8000/swagger> |
| Aspire dashboard | <http://localhost:18888> |

```
docker compose up --build
```
To run a single component instead, see the [frontend](frontend/README.md), [backend](backend/README.md), and [broker](broker/README.md) guides.

### Using the aspire dashboard
## Configuration

The docker-compose contains a container for the aspire dashboard to show telemetry and a opentelemetry collector to filter telemetry.
The aspire dashboard is available at http://localhost:18888 when run via docker compose.
Each component reads its configuration from a `.env` file. The matching `.env.example` file is the source of truth for the available variables.

To spin up only the broker, collector and aspire dashboard use
```bash
docker compose up broker aspire-dashboard otel-collector
```
| Component | File | Template | Notes |
| --------- | ------------------- | --------------------------- | --------------------------------------------------------------------- |
| Frontend | `frontend/.env` | `frontend/.env.example` | Variables must be prefixed with `VITE_` to reach the application. |
| Backend | `backend/api/.env` | `backend/api/.env.example` | Set `Local__DevUserId` to your own user id for local development. |
| Broker | `broker/.env` | `broker/.env.example` | `TLS_SERVER_KEY` is a secret and is found in our key vault. |

> **Note on Docker:** `backend/api/.env` is only read when the backend is run directly (`make run`). The backend *container* does not load it — [docker-compose.yml](./docker-compose.yml) sets `ASPNETCORE_ENVIRONMENT=Development` and reads `AZURE_CLIENT_SECRET` from a `.env` file in the repository root, which `setup.sh` does not create. To run the full stack against Azure AD you currently have to add `AZURE_CLIENT_SECRET=...` to a root `.env` yourself.

## Other make commands

## Connecting to the development database instead of initDb
Common commands are defined in the [root Makefile](./Makefile), the [backend Makefile](./backend/Makefile), and the [frontend Makefile](./frontend/Makefile):

Write in `backend/api/.env`:
```bash
Database__UseInMemoryDatabase=false
Database__PostgreSqlConnectionString=...
make compose # run the full stack in Docker
make broker # run only the MQTT broker
make broker-aspire # run the broker, OpenTelemetry collector, and Aspire dashboard
```
The database connection string can be found in the keyvault in the development resource group in Azure.
Remember to add your IP address to the accepted IPs for cennecting to the database.

## Deployments

We currently have 3 environments (Development, Staging, and Production) deployed to AKS under `robotics.equinor.com`.

| Environment | Deployment | Status |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Development | [Frontend](https://flotilla.dev.robotics.equinor.com/)<br>[Backend](https://flotilla.dev.robotics.equinor.com/api/swagger) | [![Dev](https://github.com/equinor/flotilla/actions/workflows/deploy_to_development.yml/badge.svg)](https://github.com/equinor/flotilla/actions/workflows/deploy_to_development.yml) |
| Staging | [Frontend](https://flotilla.staging.robotics.equinor.com/)<br>[Backend](https://flotilla.staging.robotics.equinor.com/api/swagger) | [![Staging](https://github.com/equinor/flotilla/actions/workflows/deploy_to_staging.yml/badge.svg)](https://github.com/equinor/flotilla/actions/workflows/deploy_to_staging.yml) |
| Production | [Frontend](https://flotilla.robotics.equinor.com/)<br>[Backend](https://flotilla.robotics.equinor.com/api/swagger) | [![Production](https://github.com/equinor/flotilla/actions/workflows/promote_to_production.yml/badge.svg)](https://github.com/equinor/flotilla/actions/workflows/promote_to_production.yml) |

## Contributions

Equinor welcomes all kinds of contributions, including code, bug reports, issues, feature requests, and documentation.
Please initiate your contribution by creating an [issue](https://github.com/equinor/isar/issues) or by forking the
Please initiate your contribution by creating an [issue](https://github.com/equinor/flotilla/issues) or by forking the
project and making a pull request. Commit messages shall be written according to [this guide](https://cbea.ms/git-commit/).
61 changes: 40 additions & 21 deletions backend/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,28 @@

The backend of Flotilla is an ASP.NET application. See the [.NET documentation](https://docs.microsoft.com/en-us/dotnet/fundamentals/) for framework concepts.

For general setup (environment, Docker, database connection), see the [root README](../README.md).
For development conventions and gotchas, see [best_practices.md](./best_practices.md).

## Common commands
## Prerequisites

- [.NET SDK 10.x](https://dotnet.microsoft.com/download)
- [`dotnet-ef`](https://docs.microsoft.com/en-us/ef/core/cli/dotnet), if you need to work with migrations:

```bash
dotnet tool install --global dotnet-ef
```

## Configuration

The backend reads environment variables from `backend/api/.env`. Create it by running [`setup.sh`](../setup.sh) from the repository root, or by copying the template manually:

```bash
cp api/.env.example api/.env
```

Set `Local__DevUserId` to your own user id for local development.

## Run

Common commands are defined in the [Makefile](./Makefile):

Expand All @@ -13,26 +32,34 @@ make run # dotnet run --project api
make build # dotnet build api
make test # dotnet test
make format # dotnet csharpier format .
make migration name=AddSomething # create a new EF Core migration
```

## Database migrations (EF Core)
Swagger is served at <http://localhost:8000/swagger>.

The database model lives in [`api/Database/Models`](./api/Database/Models) and we use [Entity Framework Core](https://docs.microsoft.com/en-us/ef/core/) as an O/RM. When changing the model, add a [migration](https://docs.microsoft.com/en-us/ef/core/managing-schemas/migrations/).
To run the backend in Docker together with the rest of the stack, see the [root README](../README.md#quick-start).

## Connecting to the development database

Install the EF Core CLI once:
By default the backend runs against an in-memory database. To use the development database instead, add the following to `backend/api/.env`:

```bash
dotnet tool install --global dotnet-ef
Database__UseInMemoryDatabase=false
Database__PostgreSqlConnectionString=...
```

The connection string is found in the key vault in the development resource group in Azure. Remember to add your IP address to the accepted IPs for connecting to the database.

## Database migrations (EF Core)

The database model lives in [`api/Database/Models`](./api/Database/Models) and we use [Entity Framework Core](https://docs.microsoft.com/en-us/ef/core/) as an O/RM. When changing the model, add a [migration](https://docs.microsoft.com/en-us/ef/core/managing-schemas/migrations/).

Create a new migration (make sure you have the latest `main` and that no one else is adding a migration at the same time):

```bash
make migration name=AddTableNamePropertyName
```

This adds files under `backend/api/Migrations` that must be committed. `add` does not modify any database — it only describes the changes.
This adds files under `backend/api/Migrations` that must be committed. Adding a migration does not modify any database — it only describes the changes.

To discard a migration you're not happy with:

Expand All @@ -45,20 +72,12 @@ dotnet ef migrations remove
- **Development**: after merging a PR that touches `backend/api/Migrations`, manually run the ["Run database migrations (Development)"](https://github.com/equinor/flotilla/actions/workflows/run_development_migrations.yml) workflow.
- **Staging / Production**: applied automatically by the [deploy_to_staging](https://github.com/equinor/flotilla/blob/main/.github/workflows/deploy_to_staging.yml) and [promote_to_production](https://github.com/equinor/flotilla/blob/main/.github/workflows/promote_to_production.yml) workflows.

## Formatting
## Monitoring

The backend is instrumented with [OpenTelemetry](https://opentelemetry.io/). Traces, metrics, and logs are exported via OTLP to a Grafana-compatible backend.

Formatting rules are defined in the [.editorconfig](../.editorconfig). We use [CSharpier](https://csharpier.com/) to auto-format on save (see [installation](https://csharpier.com/docs/About)). To check formatting locally:
Locally, telemetry can be inspected in the Aspire dashboard at <http://localhost:18888>. It is started as part of `make compose`, or on its own together with the broker and the OpenTelemetry collector:

```bash
make format
make broker-aspire
```

## SignalR

We use SignalR to push event updates to the frontend via `SignalRService`. Event names must match what the frontend expects.

Do **not** await SignalR sends — in the current library version, awaiting from an async thread can cause the thread to silently exit without an exception. Let SignalR run after the current thread completes and ignore the await warning.

## Monitoring

The backend is instrumented with [OpenTelemetry](https://opentelemetry.io/). Traces, metrics, and logs are exported via OTLP to a Grafana-compatible backend. Locally, telemetry can be inspected in the Aspire dashboard (see the [root README](../README.md#using-the-aspire-dashboard)).
57 changes: 0 additions & 57 deletions backend/api/Configurations/ConfigurationBuilderExtensions.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,63 +2,6 @@
{
public static class ConfigurationBuilderExtensions
{
/// <summary>
/// Creates the AZURE_CLIENT_ID and AZURE_TENANT_ID configuration values for the
/// <see href="https://docs.microsoft.com/en-us/dotnet/api/azure.identity.environmentcredential?view=azure-dotnet">Environment Credentials</see>
/// used by the application when dockerized.
/// </summary>
/// <param name="builder"></param>
/// <returns></returns>
public static void AddAppSettingsEnvironmentVariables(this WebApplicationBuilder builder)
{
string? clientId = builder
.Configuration.GetSection("AzureAd")
.GetValue<string?>("ClientId");
if (clientId is not null)
{
Environment.SetEnvironmentVariable("AZURE_CLIENT_ID", clientId);
Console.WriteLine("'AZURE_CLIENT_ID' set to " + clientId);
}

string? tenantId = builder
.Configuration.GetSection("AzureAd")
.GetValue<string?>("TenantId");
if (tenantId is not null)
{
Environment.SetEnvironmentVariable("AZURE_TENANT_ID", tenantId);
Console.WriteLine("'AZURE_TENANT_ID' set to " + tenantId);
}
}

/// <summary>
/// Creates if don't already exist/sets all the configuration variables present on the .env file for the
/// <see href="https://docs.microsoft.com/en-us/dotnet/api/azure.identity.environmentcredential?view=azure-dotnet">Environment Credentials</see>
/// used by the application when dockerized.
/// </summary>
/// <param name="builder"></param>
/// <param name="filePath"></param>
/// <returns></returns>
public static void AddDotEnvironmentVariables(
this WebApplicationBuilder builder,
string filePath
)
{
if (!File.Exists(filePath))
return;

foreach (string line in File.ReadAllLines(filePath))
{
string[] parts = line.Split('=', StringSplitOptions.RemoveEmptyEntries);

if (parts.Length == 0 || parts[0].StartsWith('#'))
continue;

Environment.SetEnvironmentVariable(parts[0], parts[1]);
}

builder.Configuration.AddEnvironmentVariables();
}

/// <summary>
/// Configures the logger used by the application
/// </summary>
Expand Down
Loading
Loading