DST Dashboard Backend

[mamoutou-diarra]

Context

This PR adds the DST Dashboard backend to 10ksim. The dashboard (FastAPI/uvicorn + MongoDB) exposes endpoints to the frontend to create/update/delete experiments, panels, and datasets. Datasources (VictoriaLogs, Prometheus) are provided via config.yaml. All write endpoints are JWT-protected. Tokens are generated from an admin page (accessible from the lab homepage, Authentik SSO required) that also provides an experiment CRUD UI for pasting JSON. Data is fetched using DataPuller and scrape_utils, and transform functions convert datasets into ECharts-ready panel specs.

Main changes

• Added experiment, dataset, and panel processors to fetch data, apply transformations, build ECharts-ready panels, and store them on MongoDB
• Added transform functions for boxplot, timeseries, and utilities like top-N filtering, groupBy, firstN, regex_match
• Implemented a MongoDB storage layer, using GridFS to store large experiment datasets/panels in chunks
• Added CRUD endpoints for experiments, datasets, and panels, exposed via Swagger, with writes gated behind JWT
• Added an admin page for token generation and experiment CRUD (paste JSON)
• Parallelized dataset and panel processing across a thread pool
• Added unit tests for panel processors
• Added script and example for adding new ewperiment json data (They can be added also by curl)

Deployment

• Added k8s manifests for the backend, frontend, and MongoDB (dev/prod)

App URLs

• Backend API: https://api.dashboard.lab.vac.dev
• Admin Page (token generation, Add/Update/Delete Experiments): https://api.dashboard.lab.vac.dev/admin/token (require Authentication via keycloak)

[AlbertoSoutullo]

I think this could have been splitted even more into smaller PRs. Consider that for next time please.
I would also appreciate an addition about how the flow of this works to better understand the modules.

[mamoutou-diarra]

I think this could have been splitted even more into smaller PRs. Consider that for next time please. I would also appreciate an addition about how the flow of this works to better understand the modules.

sorry for that, sure, I'll make sure to submit small PRs next time. Also I'll add the workflow in another small PR later