Improve link checking for our docs (#3022)

* Switch to using htmltest
* Use relref function for page links
* Fix image links
* Fix external references
* Fix charts reference
* Code Gardening
* Remove markdown-link-check
* Document use of private fork

Author: theunrepentantgeek

.devcontainer/install-dependencies.sh

@@ -150,7 +150,8 @@ go-install kustomize sigs.k8s.io/kustomize/kustomize/[email protected]
 
 # for docs site
 go-install hugo -tags extended github.com/gohugoio/[email protected]
-go-install htmltest github.com/wjdp/[email protected]
+# Restore this to github.com/wjdp/htmltest@v?? once PR#215 is merged with the feature we need
+go-install htmltest github.com/theunrepentantgeek/htmltest@latest
 
 # for api docs 
 # TODO: Replace this with the new release tag.
@@ -220,15 +221,7 @@ fi
 write-verbose "Checking for /usr/bin/postcss"
 if should-install "/usr/bin/postcss"; then 
     write-info "Installing postcss"
-    sudo npm install -g postcss postcss-cli autoprefixer
-fi
-
-# Ensure we can check links
-write-verbose "Checking for /usr/bin/markdown-link-check"
-if should-install "/usr/bin/markdown-link-check"; then 
-    write-info "Installing markdown-link-check"
-    # Pinned to 3.10 due to https://github.com/tcort/markdown-link-check/issues/246
-    sudo npm install -g [email protected]
+    npm install --global postcss postcss-cli autoprefixer
 fi
 
 if [ "$VERBOSE" == true ]; then 

.github/workflows/deploy-site.yml

@@ -54,7 +54,6 @@ jobs:
       - name: Build & validate docs site
         run: |
           container_id=${{ env.container_id }}
-          docker exec "$container_id" task check-docs-site-links
           docker exec "$container_id" task build-docs-site
 
       - name: Publish to gh-pages branch

.github/workflows/pr-validation-docs.yml

@@ -62,11 +62,6 @@ jobs:
           docker start "$container_id"
           echo "container_id=$container_id" >> $GITHUB_ENV
 
-      - name: Validate docs site
-        run: |
-          container_id=${{ env.container_id }}
-          docker exec "$container_id" task check-docs-site-links
-
       - name: Build & validate docs site
         run: |
           container_id=${{ env.container_id }}

Taskfile.yml

@@ -89,21 +89,8 @@ tasks:
   build-docs-site:
     dir: docs/hugo
     cmds:
-      - npm install -g postcss postcss-cli autoprefixer
       - hugo
-    # - htmltest
-    # disabled pending: https://github.com/wjdp/htmltest/issues/45
-    # , or us moving to a no-directory URL
-    env:
-      NODE_PATH:
-        sh: npm root -g
-
-  check-docs-site-links:
-    desc: Verify documentation site links
-    dir: docs/hugo
-    cmds:
-      # Excluding './content/reference/*' path as it contains all auto-generated API docs.
-      - find . -type f -name "*.md" -not -path "./content/reference/*" -not -path "./node_modules/*" -print0 | xargs -0 -n1 markdown-link-check -c link-checker.json
+      - htmltest
     env:
       NODE_PATH:
         sh: npm root -g

docs/hugo/.htmltest.yml

@@ -1,3 +1,18 @@
 DirectoryPath: public
-CheckExternal: false
+CheckExternal: true
 IgnoreAltMissing: true
+BaseURL: https://azure.github.io/azure-service-operator
+CheckSelfReferencesAsInternal: true
+IgnoreDirs:
+  - "reference" # Ignore errors in CRD docs, content is not under our control
+IgnoreURLs:
+  - /favicons/
+  - /scss/
+  - /js/
+  - example.com
+  - index.xml
+  - "https://armwiki.azurewebsites.net/api_contracts/guidelines/templatedeployment.html" # Returns 404 even though valid
+  - "https://marketplace.visualstudio.com/items" # Marketplace links return 401 even if valid
+  - "https://github.com/Azure/azure-resource-manager-rpc/blob/master/v1.0/async-api-reference.md" # Manually checked, not a 404
+  - "azure-workload-identity" # TODO: Work out why this fails
+LogLevel: 3

docs/hugo/content/contributing/_index.md

@@ -13,8 +13,8 @@ description: "How to contribute new resources to Azure Service Operator v2"
 
 ## Related pages
 
-* [Adding a new code-generator resource](./add-a-new-code-generated-resource).
-* [Generator code overview](./generator-overview)
+* [Adding a new code-generator resource]( {{< relref "add-a-new-code-generated-resource" >}} ).
+* [Generator code overview]( {{< relref "generator-overview" >}} )
 
 ## Developer setup (with VS Code)
 This is the recommended setup, especially if you are using Windows as your development platform.

docs/hugo/content/contributing/add-a-new-code-generated-resource.md

@@ -74,7 +74,7 @@ configuration around the existing values.
 
 ## Run the code generator
 
-Follow the steps in the [contributing guide](_index.md) to set up your development environment.
+Follow the steps in the [contributing guide]( {{< relref "." >}} ) to set up your development environment.
 Once you have a working development environment, run the `task` command to run the code generator.
 
 ## Fix any errors raised by the code generator

docs/hugo/content/design/ADR-2021-06-API-Version-Recovery.md

@@ -6,9 +6,9 @@ title: '2021-06: API Version Recovery'
 
 In addition to structural changes, there may be behaviour changes between ARM API versions. It is therefore important that we use the correct API version - the version requested by the user - when interacting with ARM, to ensure that we get the expected behaviour.
 
-Revisting the CRM example from the [Versioning](../versioning/) specification, consider what happens if we have two available versions of the resource `Person`, lets call them **v1** and **v2**. In **v2** the new properties `PostalAddress` and `ResidentialAddress` are mandatory, requiring that everyone have a both a mailing address and a home. 
+Revisting the CRM example from the [Versioning]( {{< relref "versioning" >}} ) specification, consider what happens if we have two available versions of the resource `Person`, lets call them **v1** and **v2**. In **v2** the new properties `PostalAddress` and `ResidentialAddress` are mandatory, requiring that everyone have a both a mailing address and a home. 
 
-![example](images/adr-2021-06-api-version-recovery-example.png)
+![example](../images/adr-2021-06-api-version-recovery-example.png)
 
 If we have a valid **v1** `Person`, trying to submit that through the **v2** ARM API will fail because it's missing these addresses.
 
@@ -53,7 +53,7 @@ type StorageAccounts_Spec struct {
 }
 ```
 
-![preservation](images/adr-2021-06-api-version-recovery-preservation.png)
+![preservation](../images/adr-2021-06-api-version-recovery-preservation.png)
 
 The generated `AssignProperties*()` methods will copy OriginalVersion as expected.
 

docs/hugo/content/design/ADR-2022-02-Backward-Compatibility.md

@@ -102,13 +102,13 @@ Introduce a new pipeline stage to create additional versions for backward compat
 
 For example, the API resource `v1beta20201201storage/VirtualMachine` will be cloned as `v1alpha1api20201201/VirtualMachine`. Subsequent pipeline stages will create matching storage versions and the necessary conversions.
 
-![image](images/adr-2022-02-backward-compatibility-single-version.png)
+![image](../images/adr-2022-02-backward-compatibility-single-version.png)
 
 For the trivial case where we only support a single ARM version of a resource, this gives us full compatibility to upgrade an older (alpha) installation of ASO to a new beta release.
 
 When we have multiple ARM versions in flight (as we do with `compute`), supporting all *potential* older resource versions gives us maximum compatibility without any need for configuration or to make any assumptions about which ARM versions were previously supported.
 
-![image](images/adr-2022-02-backward-compatibility-multiple-version.png)
+![image](../images/adr-2022-02-backward-compatibility-multiple-version.png)
 
 This approach also gives us backward compatibility when we have a mix of resources across multiple versions (as happens with both `compute` and `network`). We know from prior work (including introduction of the `PropertyBag` field) that conversions between storage versions are lossless.
 

docs/hugo/content/design/ADR-2022-09-Multiple-Credential-Operator.md

@@ -19,7 +19,7 @@ To support this, ASO will introduce support for multiple credentials, each assoc
 
 We'll be using a pattern to determine which of the above secrets we use for operation on a resource. As in the below flow chart, if a resource is applied to the operator and the resource exists, operator would fetch and apply credentials in a following manner. If it's a new resource, operator would have to go through the secret selection hierarchy. Where, operator would first check if Per-resource or per-resource-group secret exists, if not, then will check for the namespaced secret. If any of the above is provided, operator would perform actions according to the **options** below. If none is provided, we'll fallback to use the global credential for that resource, which is the default cluster scoped secret(aso-controller-settings) in ASO namespace used today. 
 
-![hierarchy](images/adr-2022-09-multiple-credential-operator.png) 
+![hierarchy](../images/adr-2022-09-multiple-credential-operator.png) 
 
 ## Options
 

docs/hugo/content/design/ADR-2022-12-Reconciliation-Extensions.md

@@ -46,11 +46,11 @@ The return will be one of three possibilities:
 
 For the initial implementation, we'll only do the GET if the extension exists, and the default `next` action will be hard coded to request a reconciliation.
 
-![Initial Sequence](images/adr-2022-12-reconciliation-initial-flow.png)
+![Initial Sequence](../images/adr-2022-12-reconciliation-initial-flow.png)
 
 Down the track when we switch to a GET/PUT workflow, we'll always do the GET, and the default 'next` action will do the comparison to see if the resource has changed.
 
-![GET/PUT Sequence](images/adr-2022-12-reconciliation-diffing-flow.png)
+![GET/PUT Sequence](../images/adr-2022-12-reconciliation-diffing-flow.png)
 
 ## Status
 

docs/hugo/content/design/ADR-2023-05-Installing-Only-Selected-CRDs.md

@@ -152,7 +152,7 @@ pre-install hook (or post-install) seems easier though.
 
 #### Answer 2c: Using a CRD
 
-We investigated doing this originally for [Helm chart size limitations](../ADR-2023-02-Helm-Chart-Size-Limitations)
+We investigated doing this originally for [Helm chart size limitations]( {{< relref "ADR-2023-02-Helm-Chart-Size-Limitations" >}} )
 
 Pros:
 * More structured than a `ConfigMap`.

docs/hugo/content/design/secrets.md

@@ -325,9 +325,7 @@ the right `ListKeys` or `GetKeys` call. To support rollover we would need a diff
 This is a relatively involved topic so not designing it all here. As a starting point, resources manually implementing the following interface would
 get us what we need. Issue [#1978](https://github.com/Azure/azure-service-operator/issues/1978) is tracking this request in more detail. 
 
-<!-- Link check fails because Hugo transforms the filename to lowercase. Manually checked by @theunrepentantgeek -->
-<!-- markdown-link-check-disable-next-line -->
-See also the design of [reconciler extensions](../adr-2022-01-reconciler-extensions).
+See also the design of [reconciler extensions]( {{< relref "adr-2022-01-reconciler-extensions" >}} ).
 
 ```go
 type ARMDetails struct {

docs/hugo/content/design/versioning/_index.md

@@ -35,11 +35,11 @@ Unlike the typical situation with a hand written service operator, we don't have
 
 There are three case studies that accompany this specification, each one walking through one possible solution and showing how it will perform as a synthetic ARM style API evolves over time.
 
-The [**Chained Versions**](../case-studies/chained-storage-versions/) case study shows how the preferred solution adapts to changes as the API is modified.
+The [**Chained Versions**]( {{< relref "case-studies/chained-storage-versions" >}} ) case study shows how the preferred solution adapts to changes as the API is modified.
 
-The [**Rolling Versions**](../case-studies/rolling-storage-versions/) case study shows an alternative that works well but falls down when hand coded conversions are introduced between versions.
+The [**Rolling Versions**]( {{< relref "case-studies/rolling-storage-versions" >}} ) case study shows an alternative that works well but falls down when hand coded conversions are introduced between versions.
 
-The [**Fixed Version**](../case-studies/fixed-storage-version/) case study shows how a popular alternative would fare, calling out some specific problems that will occur.
+The [**Fixed Version**]( {{< relref "case-studies/fixed-storage-version" >}} ) case study shows how a popular alternative would fare, calling out some specific problems that will occur.
 
 **TL;DR:** Using a *fixed storage version* appears simpler at first, and works well as long as the changes from version to version are simple. However, when the changes become complex (as they are bound to do over time), this approach starts to break down. While there is up front complexity to address with *chained storage versions*, the approach doesn't break down over time and we can generate useful automated tests for verification. The *rolling storage version* approach is viable, but requires additional ongoing maintenance when manual conversions are introduced between versions.
 

docs/hugo/content/guide/annotations.md

@@ -29,7 +29,7 @@ This credential supersedes any global or namespace scoped credentials the operat
 Allow values are:
 - The name of any secret in the same namespace as the resource. Secrets from other namespaces cannot be referenced.
 
-See [authentication](./authentication/_index.md#credential-scope) for more details.
+See [authentication]( {{< relref "authentication#credential-scope" >}} ) for more details.
 
 ## Annotations written by the operator
 

docs/hugo/content/guide/aso-controller-settings-options.md

@@ -13,7 +13,7 @@ The supported options are:
 ### AZURE_SUBSCRIPTION_ID
 
 Azure subscription the operator will use for ARM communication if 
-no [more specific](authentication/credential-scope)
+no [more specific]( {{< relref "authentication/credential-scope" >}} )
 credential is specified at the per-resource or per-namespace scope.
 
 **Format:** `GUID`
@@ -27,7 +27,7 @@ This may be set to empty string to configure no global credential.
 ### AZURE_TENANT_ID
 
 Azure tenantID the operator will use for ARM communication if
-no [more specific](authentication/credential-scope)
+no [more specific]( {{< relref "authentication/credential-scope" >}} )
 credential is specified at the per-resource or per-namespace scope.
 
 **Format:** `GUID`
@@ -41,7 +41,7 @@ This may be set to empty string to configure no global credential.
 ### AZURE_CLIENT_ID
 
 Azure clientID the operator will use for ARM communication if
-no [more specific](authentication/credential-scope)
+no [more specific]( {{< relref "authentication/credential-scope" >}} )
 credential is specified at the per-resource or per-namespace scope.
 
 **Format:** `GUID`
@@ -55,7 +55,7 @@ This may be set to empty string to configure no global credential.
 ### AZURE_CLIENT_SECRET
 
 The secret associated with the client to use if
-no [more specific](authentication/credential-scope)
+no [more specific]( {{< relref "authentication/credential-scope" >}} )
 credential is specified at the per-resource or per-namespace scope.
 
 **Format:** `String`

docs/hugo/content/guide/authentication/_index.md

@@ -13,18 +13,18 @@ There are two key topics surrounding authentication in Azure Service Operator: T
 
 Azure Service Operator supports four different styles of authentication today.
 
-1. [Service Principal using a Client Secret](./credential-format#service-principal-using-a-client-secret)
-2. [Service Principal using a Client Certificate](./credential-format#service-principal-using-a-client-certificate)
-3. [Azure-Workload-Identity authentication](./credential-format#azure-workload-identity) (OIDC + Managed Identity or Service Principal)
-4. [Deprecated] [aad-pod-identity authentication (Managed Identity)](./credential-format#deprecated-managed-identity--aad-pod-identity-)
+1. [Service Principal using a Client Secret]( {{< relref "credential-format#service-principal-using-a-client-secret" >}} )
+2. [Service Principal using a Client Certificate]( {{< relref "credential-format#service-principal-using-a-client-certificate" >}} )
+3. [Azure-Workload-Identity authentication]( {{< relref "credential-format#azure-workload-identity" >}} ) (OIDC + Managed Identity or Service Principal)
+4. [Deprecated] [aad-pod-identity authentication (Managed Identity)]( {{< relref "credential-format#deprecated-managed-identity-aad-pod-identity" >}} )
 
 ## Credential scope
 
 Each supported credential type can be specified at one of three supported scopes:
 
-1. [Global](./credential-scope#global-scope) - The credential applies to all resources managed by ASO.
-2. [Namespace](./credential-scope#namespace-scope) - The credential applies to all resources managed by ASO in that namespace.
-3. [Resource](./credential-scope#resource-scope) - The credential applies to only the specific resource it is referenced on.
+1. [Global]( {{< relref "credential-scope#global-scope" >}} ) - The credential applies to all resources managed by ASO.
+2. [Namespace]( {{< relref "credential-scope#namespace-scope" >}} ) - The credential applies to all resources managed by ASO in that namespace.
+3. [Resource]( {{< relref "credential-scope#resource-scope" >}} ) - The credential applies to only the specific resource it is referenced on.
 
 When presented with multiple credential choices, the operator chooses the most specific one:
 _resource scope_ takes precedence over _namespace scope_ which takes precedence over _global scope_.
@@ -53,4 +53,4 @@ Disadvantages:
 * Significantly harder to orchestrate ASO upgrades.
 * More kube-apiserver load, as there will be multiple operators running and watching/reconciling resources.
 
-For more details about this approach, see [multitenant deployment](./multitenant-deployment)
+For more details about this approach, see [multitenant deployment]( {{< relref "multitenant-deployment" >}} )