Merge branch 'main' into feature/insights

Author: theunrepentantgeek

.github/instructions/copilot-instructions.md

@@ -0,0 +1,67 @@
+# Copilot Instructions for Azure Service Operator
+
+You are an expert Go developer and an AI assistant helping to maintain the Azure Service Operator (ASO) project. Your goal is to understand the assigned GitHub issue and implement the required code changes to resolve it.
+
+## Your Workflow
+
+Please follow these steps to ensure your contributions are effective and align with our project standards:
+
+1.  **Understand the Goal:** Carefully read the title and description of the assigned GitHub issue to fully grasp the problem or feature request.
+2.  **Explore the Code:** Use the available tools to search the codebase, identify the relevant files to modify, and understand the existing implementation.
+3.  **Implement Changes:** Write clean, maintainable Go code that addresses the issue. Please mimic the style of the existing code in the repository.
+4.  **Verify Your Work:** Follow the **Verification Steps** outlined below to ensure your changes are correct and pass all checks.
+
+## Project Context
+
+*   **Purpose:** ASO is a Go-based operator for creating and managing Azure resources from a Kubernetes cluster.
+
+*   **Key Files:**
+    *   `v2/azure-arm.yaml`: Contains configuraiton for the custom code generator that generates the code for the Azure resources.
+
+*   **Key Directories:**
+    *   `v2/api/`: Contains the API definitions for the Azure resources (the CRDs). 
+    *   `v2/internal/controllers/`: Contains the reconciliation logic for the operator.
+    *   `docs/hugo/content/`: Holds the reference documentation.
+    *   `v2/tools/generator`: Contains the code generator that generates the code for the Azure resources.
+
+## Development Environment
+
+Your environment is automatically configured by the `.github/workflows/copilot-setup-steps.yml` workflow. This ensures all necessary tools are installed and available.
+
+*   Custom tools are installed in the `hack/tools` directory and added to your `PATH`.
+*   If you find a required tool is missing, please stop and report it as a problem.
+*   Don't modify generated files by hand (e.g. `*.gen.go` and `*.gen_test.go` files) as those changes will be overwritten during the build.
+
+## Verification Steps
+
+Before completing your task, you **must** run these checks in the specified order, one at a time. If any step fails, fix the issues and **start again from the beginning of the list.**
+
+1.  **Format Code:** 
+    ```bash
+    task format-code
+    ```
+    
+
+2.  **Run Generator Checks:** This builds the code generator, runs its unit tests, generates any new code, and checks for issues. 
+    ```bash
+    task generator:quick-checks
+    ```
+    
+3.  **Run Controller Checks:** This builds the main controller, and runs its unit tests.
+    ```bash
+    task controller:quick-checks
+    ```
+
+If an issue seems too complex to fix, please stop and ask for clarification.
+
+## Key Guidelines
+
+1. Follow Go best practices and idiomatic patterns
+2. Maintain existing code structure and organization
+3. Write unit tests for new functionality. Use table-driven unit tests when possible.
+4. Document public APIs and complex logic. Suggest changes to the `docs/hugo/content` folder when appropriate
+
+## Reference Documentation
+
+We have extensive reference documentation available under the `docs/hugo/content` directory, including a detailed contributors guide in the `docs/hugo/content/contributing` directory. Please consult this as required.
+

.github/instructions/new-resource-instructions.md

@@ -0,0 +1,118 @@
+# Copilot instructions for adding a new resource to Azure Service Operator
+
+You are an expert Go developer and an AI assistant helping to maintain the Azure Service Operator (ASO) project. Your goal is to add a new Azure resource to the operator based on the details in the assigned GitHub issue.
+
+Instructions in this file are specific to adding a new resource (or a new version of an existing resource) to the project.
+
+## Your Workflow
+
+Follow these steps precisely. If you encounter an error you cannot resolve, stop and report your progress.
+
+### 1. Identify the Resource to Add
+
+From the GitHub issue, determine the resource's **Group**, **Version**, and **Kind** (GVK).
+
+*   **Group**: The Azure service name (e.g., `storage`, `network`, `compute`).
+*   **Version**: The API version, like `2021-06-01`, which becomes `v1api20210601`. Use the latest stable version unless the issue specifies a preview.
+*   **Kind**: The singular name of the resource (e.g., `storageAccount`, `virtualNetwork`).
+
+### 2. Configure the Code Generator
+
+You will now edit `v2/azure-arm.yaml` to tell the code generator about the new resource.
+
+1.  Locate the `objectModelConfiguration` section.
+2.  Find the resource's `group` (e.g., `synapse:`). If it doesn't exist, add it in alphabetical order.
+3.  Find the `version` (e.g., `2021-06-01:`). If it doesn't exist, add it in numerical order.
+4.  Add the resource `Kind` (e.g., `Workspace:`), ensuring it is nested correctly.
+5.  Add the `$exportAs` and `$supportedFrom` directives. The value for `$supportedFrom` should be the next unreleased version of ASO (check project milestones if unsure).
+
+**Example:**
+```yaml
+# ...
+  synapse:
+    2021-06-01:
+      Workspace:
+        $exportAs: Workspace
+        $supportedFrom: v2.10.0 # Check for the correct upcoming release
+# ...
+```
+
+### 3. Run the Code Generator
+
+Execute the following command from the root of the repository to generate the resource files.
+
+```bash
+task generator:quick-checks
+```
+
+This command will likely fail with errors about resource references. This is expected. Proceed to the next step.
+
+### 4. Fix Generator Errors
+
+The most common error is `"looks like a resource reference but was not labelled as one"`. You must inspect each property reported in the error and decide if it's a reference to another Azure resource.
+
+*   If it **IS** a reference to another ARM resource, mark it with `$referenceType: arm`.
+*   If it **IS NOT** a reference, mark it with `$referenceType: simple`.
+
+Update `v2/azure-arm.yaml` with the necessary configuration.
+
+**Example:**
+```yaml
+# ...
+  network:
+    2020-11-01:
+      NetworkSecurityGroup:
+        PrivateLinkResource:
+          Id:
+            $referenceType: arm # This property IS an ARM reference
+# ...
+```
+
+After editing the file, re-run `task generator:quick-checks`. Repeat this process until the generator succeeds without errors.
+
+### 5. Review the Generated Code
+
+Inspect the newly generated files in `v2/api/<group>/<version>/`. Pay close attention to the `_types_gen.go` file for your resource.
+
+*   **Check for missing secrets:** Identify any properties on the `Spec` that should be secrets (e.g., passwords, keys) but are currently strings. Mark them in `azure-arm.yaml` with `$isSecret: true`.
+*   **Check for Azure-generated secrets:** Look at the `Status` object. If Azure generates keys, connection strings, or other sensitive data upon creation, configure them to be exported to a Kubernetes secret using `$azureGeneratedSecrets`.
+*   **Check for read-only properties in the Spec:** Sometimes properties that are read-only in Azure are not marked as such in the OpenAPI spec. These must be removed from the `Spec` by adding a `remove: true` directive in `azure-arm.yaml`.
+
+After making any changes, re-run `task generator:quick-checks`.
+
+### 6. Write an Integration Test
+
+1.  Navigate to `v2/internal/controllers/`.
+2.  Find a test for a similar resource and copy it to a new file. The file should be named `<group>_<kind>_crud_test.go`.
+3.  Modify the test to create, verify, and delete your new resource. Ensure you are testing a representative set of properties.
+4.  Delete any existing recording file for your new test from the `recordings` sub-directory.
+5.  Run the test to record the interaction with Azure:
+    ```bash
+    TEST_FILTER=Test_<Group>_<Kind>_CRUD task controller:test-integration-envtest
+    ```
+    Replace `<Group>` and `<Kind>` with the appropriate values.
+
+### 7. Create a Sample
+
+1.  Create a new sample YAML file in `v2/samples/<group>/<version>/`.
+2.  The sample should demonstrate a simple, working configuration of the resource.
+3.  Run the samples test to record the new sample's creation and deletion:
+    ```bash
+    TEST_FILTER=Test_Samples_CreationAndDeletion task controller:test-integration-envtest
+    ```
+
+### 8. Final Verification
+
+Run all local checks to ensure your changes haven't introduced any regressions.
+
+```bash
+task ci
+```
+
+This command takes a long time but is the best way to ensure your pull request will pass CI. Once it completes successfully, your task is done.
+
+
+## Reference Documentation
+
+For more detailed instructions on adding a new resource, refer to the [Adding a new code generated resource to ASO v2 documentation](docs/hugo/content/contributing/add-a-new-code-generated-resource/_index.md). 
+

.github/workflows/copilot-setup-steps.yml

@@ -0,0 +1,32 @@
+name: "Copilot Setup Steps"
+
+# Automatically run the setup steps when they are changed to allow for easy validation, and
+# allow manual testing through the repository's "Actions" tab
+on:
+  workflow_dispatch:
+  push:
+    paths:
+      - .github/workflows/copilot-setup-steps.yml
+  pull_request:
+    paths:
+      - .github/workflows/copilot-setup-steps.yml
+
+jobs:
+  # The job MUST be called `copilot-setup-steps` or it will not be picked up by Copilot.
+  copilot-setup-steps:
+    runs-on: ubuntu-latest
+    permissions:
+      packages: read
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # pinned to 4.1.7
+        with:
+          fetch-depth: 0 # required to access tags
+          submodules: 'true'
+
+      - name: Install Dependencies
+        run: sudo .devcontainer/install-dependencies.sh --skip-installed
+
+      - name: Set custom PATH
+        run: echo "PATH=$PATH:$(realpath ./hack/tools)" >> $GITHUB_ENV

.gitignore

@@ -83,3 +83,4 @@ hack/tools
 .go-version
 v2/controller
 v2/cmd/asoctl/asoctl
+v2/tools/generator/aso-gen

Taskfile.yml

@@ -267,7 +267,9 @@ tasks:
     desc: Update {{.GENERATOR_APP}} golden test outputs.
     dir: '{{.GENERATOR_ROOT}}'
     cmds:
-      - go test ./... -run ^TestGolden -update
+      - go test ./...
+    env:
+      GOLDIE_UPDATE: 1
 
   generator:format-code:
     desc: Ensure all code for the code generator is formatted

docs/hugo/content/contributing/add-a-new-code-generated-resource/_index.md

@@ -6,6 +6,19 @@ layout: single
 
 Want to add a new resource to Azure Service Operator v2? You're in the right place - here's your step by step guide.
 
+## Quick Start
+
+Adding a new resource is a multi-step process driven by our code generator. At a high level, the process is:
+
+1.  **Configure**: Edit `v2/azure-arm.yaml` to add the new resource.
+2.  **Generate**: Run `task generator:quick-checks` to generate the Go code. This will likely fail.
+3.  **Fix & Repeat**: Fix errors reported by the generator by adding configuration to `v2/azure-arm.yaml`, then re-run the generator. Repeat until generation succeeds.
+4.  **Test**: Write an integration test to verify the resource can be created, updated, and deleted.
+5.  **Sample**: Create a sample YAML to demonstrate how to use the resource.
+6.  **Verify**: Run `task ci` to ensure all checks pass before creating a pull request.
+
+## Step by step guide
+
 By the end of this process, you'll have created a new resource, written a test to verify it works, and created a sample to demonstrate how to use it. This sounds like a lot of work, but most of it is done for you by our code generator.
 
 1. [**Before you begin**]({{< relref "before-you-begin" >}}) sets the context for the rest of the process. You begin by identifying the resource you want to add and preparing your development environment.

docs/hugo/content/contributing/aso-codegen-structure.svg

@@ -129,6 +129,15 @@ These resource(s) are available for use in the current release of ASO. Different
 
 To install the CRDs for these resources, your ASO configuration must include `cache.azure.com/*` as one of the configured CRD patterns. See [CRD Management in ASO](https://azure.github.io/azure-service-operator/guide/crd-management/) for details on doing this for both [Helm](https://azure.github.io/azure-service-operator/guide/crd-management/#helm) and [YAML](https://azure.github.io/azure-service-operator/guide/crd-management/#yaml) based installations.
 
+### Next Release
+
+Development of these new resources is complete and they will be available in the next release of ASO.
+
+| Resource                                                                                           | ARM Version | CRD Version   | Supported From | Sample                                                                                                                                      |
+|----------------------------------------------------------------------------------------------------|-------------|---------------|----------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+| [RedisEnterprise]({{< relref "/reference/cache/v1api20250401#RedisEnterprise" >}})                 | 2025-04-01  | v1api20250401 | v2.15.0        | [View](https://github.com/Azure/azure-service-operator/tree/main/v2/samples/cache/v1api20250401/v1api20250401_redisenterprise.yaml)         |
+| [RedisEnterpriseDatabase]({{< relref "/reference/cache/v1api20250401#RedisEnterpriseDatabase" >}}) | 2025-04-01  | v1api20250401 | v2.15.0        | [View](https://github.com/Azure/azure-service-operator/tree/main/v2/samples/cache/v1api20250401/v1api20250401_redisenterprisedatabase.yaml) |
+
 ### Latest Released Versions
 
 These resource(s) are the latest versions available for use in the current release of ASO. Different versions of a given resource reflect different versions of the Azure ARM API.