Error performing GET `_last_checkpoint` after 10 retries

[nicolo-barboni]

Environment

Delta-rs version: 1.1.4

Binding:

Environment:

• Cloud provider: Azure
• OS: Linux
• Other:

---

Bug

What happened:
I get this error very often and I don't know what causes it.

Generic MicrosoftAzure error
          �[31m↳�[0m Error performing GET https://datalakelandingstage.blob.core.windows.net/stream-data/_delta_log/_last_checkpoint in 59.932717576s, after 10 retries, max_retries
           �[31m↳�[0m 10, retry_timeout
            �[31m↳�[0m 180s  - HTTP error
             �[31m↳�[0m error sending request
              �[31m↳�[0m client error (Connect)
               �[31m↳�[0m operation timed out

Is this an issue related to delta-rs or ADLG2?

What you expected to happen:
It should be able to retrieve the _last_checkpoint.

How to reproduce it:
I haven't managed to reproduce it in local.

More details:

[roeap]

Its hard to say conclusively, but _last_checkpoint is an optional file, so we do fall back if it does not exist.

Since the error is raised, it seems we are failing to connect to the object store at all. For this there can be many reasons. Since we are trying it seems we have a credential.

Could there be maybe networking issues? Maybe cogestion. Do you have access to the logs for the object store to see if the requests are received there?

If this is running in k8s with many (or large) other workloads, could it be node networking restrictions?

In general the azure integration does work i believe.

[ion-elgreco]

Operation timed out clearly points to connection issue

[nicolo-barboni]

Thank you for you suggestions. I will into connection/networking issues.

[nicolo-barboni]

Since the error is raised, it seems we are failing to connect to the object store at all. For this there can be many reasons. Since we are trying it seems we have a credential.

Could there be maybe networking issues? Maybe cogestion. Do you have access to the logs for the object store to see if the requests are received there?

I checked the logs, and there was no request at all, not even for others log files. In that scenario I was trying to append to a delta lake.

If this is running in k8s with many (or large) other workloads, could it be node networking restrictions?

This is running on an Azure function app. I tried checking the logs there, but there wasn't anything useful there.

[ion-elgreco]

@nicolo-barboni that makes sense because it says connection timed out, so it never got established with your store. Perhaps outbound network is not working from the azure function app.

[niltecedu]

Heyo I think this is a rust object store issue , @nicolo-barboni im 80% sure that if go back to version 1.1.0 or before It will work.
For my everything apart from the abfss endpoints has been broken for azure storage and deltalake1.1.0

[nicolo-barboni]

@niltecedu I use abfss endpoints though and it works most of the time. We get 300 errors like this over more than 17000 append operations per day.

[niltecedu]

Whats your networking setup? I know that abfss endponts and az endpoints get messey sometimes with private endpoints in hybrid envrionments. Ive also had issues with containers with soft delete enabled.

Also can you try out on 1.1.0 and see if the issue persists?