FORBIDDEN.MISSING_PROVIDER when submitting for notarization

[wagoodman]

We got a notarization failure during release:

notarization failed: unable to start submission: http status="403 Forbidden": 
body="{\n  \"errors\" : [ {\n    \"id\" : \"36LP4E6E7YLA7F23MAYGS3N7T4\",\n    \"status\" : \"403\",\n    \"code\" : \"FORBIDDEN.MISSING_PROVIDER\",\n    
\"title\" : \"This provider does not exist.\",\n    
\"detail\" : \"This provider does not exist.\",\n    
\"links\" : {\n      \"see\" : \"/landing\"\n    }\n  } ]\n}\n"

full logs

• building binaries
    • building                                       binary=dist/linux-build_linux_s390x/syft
    • building                                       binary=dist/linux-build_linux_ppc64le_power8/syft
    • building                                       binary=dist/linux-build_linux_amd64_v1/syft
    • building                                       binary=dist/linux-build_linux_arm64_v8.0/syft
    • building                                       binary=dist/darwin-build_darwin_amd64_v1/syft
    • building                                       binary=dist/darwin-build_darwin_arm64_v8.0/syft
    • building                                       binary=dist/windows-build_windows_amd64_v1/syft.exe
    • running hook                                   hook=.tool/quill sign-and-notarize "/home/runner/work/syft/syft/dist/darwin-build_darwin_amd64_v1/syft" --dry-run=false --ad-hoc=false -vv
    • running hook                                   hook=.tool/quill sign-and-notarize "/home/runner/work/syft/syft/dist/darwin-build_darwin_arm64_v8.0/syft" --dry-run=false --ad-hoc=false -vv
    • took: 5m50s
  ⨯ release failed after 5m49s              
    error=
    │ post hook failed: shell: '.tool/quill sign-and-notarize /home/runner/work/syft/syft/dist/darwin-build_darwin_amd64_v1/syft --dry-run=false --ad-hoc=false -vv': exit status 1: [0000]  INFO quill version: 0.5.1
    │ [0000] DEBUG config:
    │   log:
    │       quiet: false
    │       level: debug
    │       file: /tmp/quill-darwin_amd64_v1.log
    │   dev:
    │       profile: none
    │   path: /home/runner/work/syft/syft/dist/darwin-build_darwin_amd64_v1/syft
    │   sign:
    │       identity: ""
    │       p12: *******
    │       timestamp-server: http://timestamp.apple.com/ts01
    │       ad-hoc: false
    │       fail-without-full-chain: true
    │       password: *******
    │       entitlements: ""
    │   notary:
    │       issuer: ***
    │       key-id: ***
    │       key: *******
    │   status:
    │       wait: true
    │       poll-seconds: 10
    │       timeout-seconds: 900
    │   dry-run: false
    │ [0000] DEBUG root cert: CN=Apple Root CA,OU=Apple Certification Authority,O=Apple Inc.,C=US
    │ [0000] DEBUG intermediate cert: CN=Developer ID Certification Authority,OU=Apple Certification Authority,O=Apple Inc.,C=US
    │ [0000] DEBUG signing cert: CN=Developer ID Application: ANCHORE\, INC. (9MJHKYX5AT),OU=9MJHKYX5AT,O=ANCHORE\, INC.,C=US,0.9.2342.19[200](https://github.com/anchore/syft/actions/runs/15071814428/job/42369826607#step:7:201)300.100.1.1=#130a394d4a484b5958354154
    │ [0000]  INFO signing binary binary=/home/runner/work/syft/syft/dist/darwin-build_darwin_amd64_v1/syft
    │ [0000] DEBUG estimating signing material size
    │ [0000] DEBUG SpecialSlotHashWriter: 2 special slots
    │ [0000] DEBUG SpecialSlotHashWriter: writing slot 2
    │ [0000] DEBUG SpecialSlotHashWriter: writing slot 1
    │ [0000] DEBUG SpecialSlotHashWriter: slot 1 is empty
    │ [0001] DEBUG patching binary with updated superblob offsets
    │ [0001] DEBUG creating signature for binary
    │ [0001] DEBUG SpecialSlotHashWriter: 2 special slots
    │ [0001] DEBUG SpecialSlotHashWriter: writing slot 2
    │ [0001] DEBUG SpecialSlotHashWriter: writing slot 1
    │ [0001] DEBUG SpecialSlotHashWriter: slot 1 is empty
    │ [0002] DEBUG patching binary with signature
    │ [0002]  INFO notarizing binary binary=/home/runner/work/syft/syft/dist/darwin-build_darwin_amd64_v1/syft
    │ [0002] DEBUG loading private key for notary
    │ [0005] DEBUG starting submission name=syft-a4840cb9e25cc0d5227656c6d473d495eb4aa8691a62974042bdd16f7407dcda-77c025d7
    │ notarization failed: unable to start submission: http status="403 Forbidden": body="{\n  \"errors\" : [ {\n    \"id\" : \"36LP4E6E7YLA7F23MAYGS3N7T4\",\n    \"status\" : \"403\",\n    \"code\" : \"FORBIDDEN.MISSING_PROVIDER\",\n    \"title\" : \"This provider does not exist.\",\n    \"detail\" : \"This provider does not exist.\",\n    \"links\" : {\n      \"see\" : \"/landing\"\n    }\n  } ]\n}\n"
    target=darwin_amd64_v1
task: Failed to run task "ci-release": exit status 1
make: *** [Makefile:29: ci-release] Error [201](https://github.com/anchore/syft/actions/runs/15071814428/job/42369826607#step:7:202)

I do see evidence that of the two notarizations being submitted there is only 1 that succeeded. There could be a race condition against the submission API that we haven't run into before (unconfirmed).

[akostibas]

We are also experiencing this issue as an intermittent failure.

[macrael]

@wagoodman Have you continued to see this? we put a retry in our CI which has quieted it but I have no idea if it's still happening