'401 Client Error: Unauthorized for url: https://tenant.sharepoint.com/sites/site/_api/contextInfo' when uploading a file to SharePoint with cert auth

[nicklockhart-fullfibre]

Hi all,

Not sure if this is an issue with my environment/setup or a bug with the library. I'm currently trying to migrate a script using SAML auth (see #981) to certificate-based app-only auth. I have a cert which was generated as per the Microsoft documentation, which results in a PFX-format private key. I can see that this library needs a PEM-format private key.

I converted the PFX-format key to PEM-format using openssl:

openssl pkcs12 -in myapp.pfx -out myapp.pem -nodes

If I then check the fingerprint of the PEM-format key, it matches the PFK-format key:

openssl x509 -fingerprint -noout -in myapp.pem
SHA1 Fingerprint=8F:42:8E:60[...]

However, if I then try to upload a file to Sharepoint using certificate authentication...

    login_cert = {
        'tenant': 'tenant.onmicrosoft.com',
        'client_id': 'bc4b906b[...]',
        'thumbprint': '8f428e60[...]',
        'cert_path': '/usr/local/apps/move_to_sharepoint/myapp.pem',
    }
    ctx = ClientContext(site_url).with_client_certificate(**login_cert)

    # Specify destination folder
    folder = ctx.web.get_folder_by_server_relative_url("[path]")

    # Upload file
    local_file_path = args.filename
    with open(local_file_path, "rb") as f:
        file = folder.files.upload(f).execute_query()

...the following error is thrown:

Traceback (most recent call last):
  File "/usr/local/apps/move_to_sharepoint/venv/lib/python3.10/site-packages/office365/runtime/client_request.py", line 37, in execute_query
    response = self.execute_request_direct(request)
  File "/usr/local/apps/move_to_sharepoint/venv/lib/python3.10/site-packages/office365/runtime/client_request.py", line 46, in execute_request_direct
    self.beforeExecute.notify(request)
  File "/usr/local/apps/move_to_sharepoint/venv/lib/python3.10/site-packages/office365/runtime/types/event_handler.py", line 41, in notify
    listener(*args, **kwargs)
  File "/usr/local/apps/move_to_sharepoint/venv/lib/python3.10/site-packages/office365/sharepoint/client_context.py", line 278, in _build_modification_query
    self._ensure_form_digest(request)
  File "/usr/local/apps/move_to_sharepoint/venv/lib/python3.10/site-packages/office365/sharepoint/client_context.py", line 216, in _ensure_form_digest
    self._ctx_web_info = self._get_context_web_information()
  File "/usr/local/apps/move_to_sharepoint/venv/lib/python3.10/site-packages/office365/sharepoint/client_context.py", line 228, in _get_context_web_information
    response = client.execute_request_direct(request)
  File "/usr/local/apps/move_to_sharepoint/venv/lib/python3.10/site-packages/office365/runtime/client_request.py", line 101, in execute_request_direct
    response.raise_for_status()
  File "/usr/local/apps/move_to_sharepoint/venv/lib/python3.10/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://tenant.sharepoint.com/sites/site/_api/contextInfo

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/apps/move_to_sharepoint/move_to_sharepoint.py", line 37, in <module>
    file = folder.files.upload(f).execute_query()
  File "/usr/local/apps/move_to_sharepoint/venv/lib/python3.10/site-packages/office365/runtime/client_object.py", line 55, in execute_query
    self.context.execute_query()
  File "/usr/local/apps/move_to_sharepoint/venv/lib/python3.10/site-packages/office365/runtime/client_runtime_context.py", line 173, in execute_query
    self.pending_request().execute_query(qry)
  File "/usr/local/apps/move_to_sharepoint/venv/lib/python3.10/site-packages/office365/runtime/client_request.py", line 41, in execute_query
    raise ClientRequestException(*e.args, response=e.response)
office365.runtime.client_request_exception.ClientRequestException: (None, None, '401 Client Error: Unauthorized for url: https://tenant.sharepoint.com/sites/site/_api/contextInfo')

The app has been granted Sites.FullControl.All as per the library's documentation, so I don't think this is a permissions issue. The file upload code should be correct and previously worked OK when using username/password auth (sadly now deprecated).

Is this an issue with my environment (eg PKCS key converted to X509 via OpenSSL), a bug in my code (eg wrong params passed to with_client_certificate) or have I hit a bug in the library?

I'm using the latest version of the library (2.6.2) with Python 3.10.12 on Ubuntu 22.04 LTS, if that affects anything.

[sengatang]

From our communication with Microsoft, using Azure AD Certified Authentication requires the following steps:
Step 1: Register an App in Azure AD

1. Go to https://portal.azure.com/.
2. Navigate to Azure Active Directory > App registrations > New registration.
3. Enter a name (e.g., SharePointAccessScript).
4. Set Supported account types to "Accounts in this organizational directory only".
5. Click Register.

Step 2:Configure API Permissions

1. Grant Application Permission (Sites.Read.All or Sites.FullControl.All)
2. Grant admin consent for those permissions

Step 3: Create and configure a Client Secret

Step 4: Update Python Script

Maybe you can check your permissions?

And just a thought for the pfx and pem, from what GPT told me"office365-rest-python-client requires the private key to be in the standard PKCS#8 format"

You can use the following command to convert the private key to PKCS#8 format:

openssl pkcs12 -in your_cert.pfx -nocerts -nodes | openssl pkcs8 -topk8 -nocrypt -out private_key.pem

[nicklockhart-fullfibre]

I have already created an application in our Azure portal and configured it correctly. The application has Sites.FullControl.All permissions, as suggested by Microsoft.

from what GPT told me

Please don't use ChatGPT to respond to other people's tickets. If I wanted to hear AI's best guess at a solution, I would've already asked it myself.

I did try converting the cert to PKCS8 format using the snippet provided, but this did not resolve the issue. I receive the same 401 error from the same contextInfo endpoint as I did with my original cert (converted from PFX to PEM using the snippet in my original comment - openssl pkcs12 -in myapp.pfx -out myapp.pem -nodes)

[sengatang]

Fine, I did not just quote GPT.
This is what it responded to me and worked for me.

[samyalabheek]

We are facing the same issue while trying to fetch the list of folders in our sharepoint site.

[ch-fuldry]

AFAIK check your path, using a wrong relative or absolute path would result in a 401 error, which does not mean access denied, but maybe that the target file/folder is wrong i have been struggling with this exact issue for half a day today :)

[samyalabheek]

I have tried the same code with a separate tenant, and I do not see the 401 error for that tenant.

[ch-fuldry]

I have tried the same code with a separate tenant, and I do not see the 401 error for that tenant.

Hello, you probably have a permission issue, as I said, if the path is wrong, or the permissions are wrong, whatever the cause of the issue, you will get a 401 error, to avoid information leak. To test your user / certificate, use the PnP powershell library, that' show I validated my permissions and paths. You can also test using the Graph powershell module, but it has trouble handling uploads.
For reference
Connect-PnPOnline -Url "(sharepoint site)" -ClientId "(App ID)" -Tenant "(Tenant ID)" -CertificatePassword (Read-Host -AsSecureString) -CertificatePath "(pfx file)"
Once connected, you can then use Get-PnPFolderItem to see the top level containers.

[illutari]

@nicklockhart-fullfibre I know you mentioned initially that the correct permissions were applied

The app has been granted Sites.FullControl.All as per the library's documentation, so I don't think this is a permissions issue

But I too was running into the same error after switching from Client ID + Client Secret to Client ID + Cert.

Tried a bunch of small changes, was thinking it was an issue with the URL I was using or I needed to make a code change somewhere... but after coming across #973 I realized that our permissions were setup incorrectly for the App Registration.

I myself didn't setup the access (as I had to work with a different team in my org to do that, but gave them the normal instructions), so I'm not sure if he followed the instructions incorrectly, or if part of the MS docs have everyone setting this up wrong. But if following these:

• https://github.com/vgrem/office365-rest-python-client/wiki/How-to-connect-to-SharePoint-Online-with-certificate-credentials
• https://learn.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azuread

I would recommend double checking your App Registration's permissions after setting it up, if getting the 401 unauthorized error.

1. Load up entra: https://entra.microsoft.com/
2. Search for your App Registration, then click on it.
3. App Registration Homepage --> Click on API Permissions
4. Under Configured permissions, are you seeing Microsoft Graph listed as a dropdown option above the applied permissions, or do you see SharePoint? In my case, all I saw was Microsoft Graph (same as image in 401 Client Error: Unauthorized for url when using certificate #973), which is where I realized I needed to fix perms there.

Once fixing the permissions for our App Registration (so it specifically applied for SharePoint and not just Microsoft Graph), everything started working again.

Unrelated NOTE: You DO NOT have to apply perms for ALL sites, you can apply the perms for selected sites so you are not giving your App Registration access to every site on your Sharepoint tenant.