AutoDev SJAD Agent

This doc is to record the test results of using a new SJAD agent based on AutoDev framework (see PR 611722).

Set up

Follow this README.md to set up the environment.

1. Install pnpm: npm install -g pnpm
2. Build: pnpm run clean:build

Prompts

https://dev.azure.com/devdiv/OnlineServices/_git/AutoDev/pullrequest/611722?_a=files&path=/node/packages/autodev-sjad/src/agents/javaToAzureYaml/prompts.ts

Model

Currently use o3-mini model, also we can switch to gpt-4o.

Test Results

Each test case will write the generated azure.yaml into the expected file, see above PR.

MySQL

Repo: https://github.com/haoozhang/spring-petclinic-mysql

# yaml-language-server: $schema=https://raw.githubusercontent.com/azure-javaee/azure-dev/feature/sjad/schemas/alpha/azure.yaml.json
name: spring-petclinic-mysql
services:
  petclinic-web:
    project: spring-petclinic-mysql
    host: containerapp
    language: java
resources:
  petclinic-web:
    type: host.containerapp
    uses:
      - mysql
    port: 8080
  mysql:
    type: db.mysql
    authType: password
    databaseName: petclinic

PostgreSQL

Repo: https://github.com/haoozhang/spring-petclinic-postgresql

# yaml-language-server: $schema=https://raw.githubusercontent.com/azure-javaee/azure-dev/feature/sjad/schemas/alpha/azure.yaml.json
name: spring-petclinic-postgresql
services:
  petclinic:
    project: spring-petclinic-postgresql
    host: containerapp
    language: java
    env:
      SPRING_DATASOURCE_URL: "@postgresql.connectionString"
resources:
  petclinic:
    type: host.containerapp
    uses:
      - postgresql
    port: 8080
  postgresql:
    type: db.postgres
    authType: password
    databaseName: petclinic

Cosmos

Repo: https://github.com/rujche/samples/tree/azd-enhancement-for-cosmos-db

# yaml-language-server: $schema=https://raw.githubusercontent.com/azure-javaee/azure-dev/feature/sjad/schemas/alpha/azure.yaml.json
name: samples-app
services:
  samples-web:
    project: samples
    host: containerapp
    language: java
resources:
  samples-web:
    type: host.containerapp
    uses:
      - mysql
      - eventhubs
    port: 8080
  mysql:
    type: db.mysql
    authType: password
    databaseName: samplesdb
  eventhubs:
    type: messaging.eventhubs
    authType: userAssignedManagedIdentity
    eventHubNames:
      - eventhub1

Kafka

Repo: https://github.com/haoozhang/Spring-Cloud-Stream-Kafka-Application

# yaml-language-server: $schema=https://raw.githubusercontent.com/azure-javaee/azure-dev/feature/sjad/schemas/alpha/azure.yaml.json
name: spring-cloud-stream-kafka-app
services:
  spring-cloud-stream-app:
    project: .
    host: containerapp
    language: java
    env:
      KAFKA_EVENTHUB_CONNECTION_STRING: "[kafka_connection_string]"
resources:
  spring-cloud-stream-app:
    type: host.containerapp
    uses:
      - kafka
  kafka:
    type: messaging.eventhubs
    authType: userAssignedManagedIdentity
    eventHubNames:
      - kafka-eventhub

Java backend + Vue Frontend

Repo: https://github.com/rujche/todo-java-mongo-aca/tree/azd-enhancement-for-mongo

# yaml-language-server: $schema=https://raw.githubusercontent.com/azure-javaee/azure-dev/feature/sjad/schemas/alpha/azure.yaml.json
name: todo-java-mongo-aca
services:
  todo-api:
    project: todo-java-mongo-aca
    host: containerapp
    language: java
    env:
      MONGO_CONNECTION: mongo
resources:
  todo-api:
    type: host.containerapp
    uses:
      - mongo
    port: 8080
  mongo:
    type: db.mongo
    databaseName: todo

Petclinic Microservices

Repo: https://github.com/azure-javaee/spring-petclinic-microservices/tree/sjad

# yaml-language-server: $schema=https://raw.githubusercontent.com/azure-javaee/azure-dev/feature/sjad/schemas/alpha/azure.yaml.json
name: spring-petclinic-microservices
services:
  config-server:
    project: config-server
    host: containerapp
    language: java
  discovery-server:
    project: discovery-server
    host: containerapp
    language: java
  api-gateway:
    project: api-gateway
    host: containerapp
    language: java
  vets-service:
    project: vets-service
    host: containerapp
    language: java
  visits-service:
    project: visits-service
    host: containerapp
    language: java
  customers-service:
    project: customers-service
    host: containerapp
    language: java
resources:
  config-server:
    type: host.containerapp
  discovery-server:
    type: host.containerapp
  api-gateway:
    type: host.containerapp
    uses:
      - config-server
      - discovery-server
      - mysql
    port: 8080
  vets-service:
    type: host.containerapp
    uses:
      - mysql
  visits-service:
    type: host.containerapp
    uses:
      - mysql
  customers-service:
    type: host.containerapp
    uses:
      - mysql
  mysql:
    type: db.mysql
    authType: password
    databaseName: petclinic

Conclusion

The result for Cosmos and Java backend and Vue frontend are not desired entirely.

This is the initial version, maybe we can improve its performance by:

1. Refine the prompts. I found that the prompt has a direct and obvious impact on the generated result, for example, the result for Cosmos is very similar as the example in the prompt.
2. Try other more advanced LLM model.
3. Currently the core logic of agent is to iterate the generated result until it follows the azure.yaml schema. Wish to further update the core logic of the agent by checking the content of azure.yaml.
4. Introduce our own domain knowledge and call LLM based on these knowledge, to validate if it can do more accurate azure.yaml generation.