Long term Themes

Target Features

1. Use Beacon API to collect metrics especially from react client website. However, make it optional first at the admin level and second at the user level. User must have a choice for GDPR compliance.
2. Make DTaaS multi-architecture / platform
3. Replace traefik forward-auth with oauth-keeper
4. Keep support for relative paths in DTs. This means that all of user assets need to be cloned when a DT is executed in DevOps pipeline.
5. Create examples of job and service DTs using DevOps features
   1. Potentially using runner + nanomq + piston + Apache airflow
6. Config generation and validation from dtaas pip package.
7. Feature complete CLI for admin tasks. Tasks to be taken care of are: install, uninstall, backup and update. Python frameworks to be used for the CLI.
8. Feature complete DT Lifecycle manager and DT Configuration Validator microservices.
9. Support S3, git and seaweed modes in libms. Integrate with gitlab for closed loop DevOps execution.
10. Integrate git modes for library microservice. See discussion #951.
11. Runner server package
    1. Integrate async execution, reporting and clean up of orphan processes 1 Integrate MQTT protocol
    2. Make runner package cross-platform see issue #637
    3. JWT-based authentication
    4. Integrate nanomq
    5. Clean up all the child processes
12. Fully functional runner client package (use MQTT).
13. Complete code coverage for tests. Feature tests at the top-level of the application.
14. Gitlab compliant GraphQL API for library microservice
15. Kubernetes based execution manager. The execution manager should be able to manage DTs running inside docker containers.
16. Keep support for relative paths in DTs. This means that all of user assets need to be cloned when a DT is executed in DevOps pipeline.

Client Features

1. DT Execute Tab
   • Implement driver file for collecting performance measurements of concurrent execution of same and different DTs
2. Move the typescript code of DT Manage and DT Execute tabs into an npm package named @into-cps-association/dt-automation. Remember to place it inside servers/execution/automation.
3. DT Execute Tab
   • Provide option to execute a specific version of a DT
4. DT Create Tab
   • Clean up the interface of DT Assets (side bar)
5. DT Manage Tab
   • Add option to save DT version with Reconfigure button; the version information is to be saved as git tag.
   • Delete all versions (i.e. all related tags) when deleting a DT
6. DT Execute Tab
   • Concurrent execution on
     • different backends - Gitlab DevOps and Azure / remote docker daemon / kubernetes / onrender
   • Download option for execution logs
7. DT Configuration (business logic for DT Create and DT Manage pages)
   • Add yaml validator for all the .gitlab-ci.yml files
   • Validate yaml of DT config specific to DTaaS
   • Convert DT config of DTaaS to .gitlab-ci.yml files
8. File Explorer on Workbench
   • Uses gitlab backend
   • Reuse React components of the library page
   • Add View button for exploring directory of an asset
   • Use glb viewer for showing glb files

CLI Features

1. Add compose and dtaas.toml files in cli package. Seethis example. Use these for generating the required files.
2. Update help text with emoji and examples.
3. Put all the required compose files in templates/cli directory and add it to the installation package.
4. Add the tls=true field to select appropriate compose files. The server-dns=localhost to distinguish production or localhost installations. Users don't need to select a compose file. CLI can select the right one.
5. Use single compose file with different profiles. Use yaml anchors, aliases and extensions to remove duplication. This idea is applicable to compose files in other parts of the code base as well.
6. Put in all the required installation files (deploy/docker, deploy/config and deploy/services) in templates directory of the CLI. Use copier for managing template copies.
7. Manage config file with

   dtaas admin config generate [--output FILE]
   dtaas admin config validate [FILE]
   dtaas admin config show
   

   The client, compose, forward-auth, service configs need to be generated and validated.

Give a template config to use with the command dtaas admin [-c | --config] [install | uninstall | status] .

1. Use Gitlab API to create / delete new users.
2. Replace sub-process based user management with python on whales

Note: The CLI follows design principles of azure CLI, suggestions from CLI guidelines and 12 factor CLI apps.

Gitlab

• The logout on gitlab does not effect the logged in status on the react app. Gitlab does not have the capability to make a webhook call to oauth end points for logging them out.
• Two concurrent sessions at different instances of DTaaS do not work properly with OAuth. The OAuth for the instance launched next takes the OAuth configuration from the first application.

Security

• Secure folders for all user folders. Generate security keys and use them to encrypt user folders on the server. Only users with their keys can see the files but no one else can.
• Provide SSH access to user workspaces. Users can complete the development on their desktops and then push assets to user workspace (use TCP proxy servers - HA Proxy or Traefik). Perhaps a lightweight DTaaS (git) agent on user desktops can push the assets to gitlab which are then auto-downloaded into the user workspace.
• Migrate npm publish of GitHub Actions to trusted publishing

Documentation

• The markdown to html is good, but not markdown to html to pdf. The mathematical expressions are missing from the pdf documentation. The notices/warnings/tips etc. are not rendered correctly as well.

Installation

• Complete application released as Kubernetes Helm Chart
• All installation managed through the python package
• Upgrade Windows installation scripts and fix pre-commit hooks (see issue 635 and PR 635)

Other priorities

• Integrate Thingsboard and thingsboard-gateway into the DTaaS. Prefer these two over Eclipse Ditto / Hono.