[C++] Problem when used from "qlora", huggingface. Does not clean up, and dumps core.

[phalexo]

Describe the bug, including details regarding any error messages, version, and platform.

/arrow/cpp/src/arrow/filesystem/s3fs.cc:2598: arrow::fs::FinalizeS3 was not called even though S3 was initialized. This could lead to a segmentation fault at exit
Segmentation fault (core dumped)

Ubuntu 20.04

arrow==1.2.3
pyarrow==12.0.0

Component(s)

C, C++

[phalexo]

I rebuilt arrow and pyarrow from sources, and the problem still persists. It is not clear in s3fs.cc where arrow::fs::FinalizeS3 should be called. The experiment to call it just before AWS connection termination did NOT get rid of the core dumping.

[westonpace]

This requirement comes from S3 and is documented here

The AWS SDK for C++ must be initialized by calling Aws::InitAPI. Before the application terminates, the SDK must be shut down by calling Aws::ShutdownAPI. Each method accepts an argument of Aws::SDKOptions

All other calls to the SDK can be performed between these two method calls.

All AWS SDK for C++ calls performed between Aws::InitAPI and Aws::ShutdownAPI should either to be contained within a pair of curly braces or should be invoked by functions called between the two methods.

The FinalizeS3 method must be called once and it must be called before the application exits (e.g. before any objects with "static storage duration" are destroyed)

[phalexo]

This requirement comes from S3 and is documented here

The AWS SDK for C++ must be initialized by calling Aws::InitAPI. Before the application terminates, the SDK must be shut down by calling Aws::ShutdownAPI. Each method accepts an argument of Aws::SDKOptions
All other calls to the SDK can be performed between these two method calls.
All AWS SDK for C++ calls performed between Aws::InitAPI and Aws::ShutdownAPI should either to be contained within a pair of curly braces or should be invoked by functions called between the two methods.

The FinalizeS3 method must be called once and it must be called before the application exits (e.g. before any objects with "static storage duration" are destroyed)

Yes, so arrow, pyarrow should clean the stuff up properly. I looked at the arrow code, and it is not clear to me where this should happen. Putting the call before the connection is severed did not fix the problem.

[westonpace]

I looked at the arrow code, and it is not clear to me where this should happen.

As I said, you must call it once before your application exits.

Putting the call before the connection is severed did not fix the problem.

I'm sorry. I don't understand what this means.

For example, this is valid:

int main() {
  if (!arrow::fs::EnsureS3Initialized().ok()) {
    std::cerr << "Failed to initialize S3" << std::endl;
    return 1;
  }
  // Your program goes here...
  if (!arrow::fs::EnsureS3Finalized().ok()) {
    std::cerr << "Failed to finalize S3" << std::endl;
    return 1;
  }
  return 0;
}

This is not valid:

struct GlobalS3Filesystem {
  GlobalS3Filesystem() {
    if (!arrow::fs::EnsureS3Initialized().ok()) {
      // handle error
      return;
    }
    arrow::Result<std::shared_ptr<arrow::fs::S3FileSystem>> maybe_s3fs =
        arrow::fs::S3FileSystem::Make({});
    if (!maybe_s3fs.ok()) {
      // handle error
      return;
    }
    s3fs = maybe_s3fs.MoveValueUnsafe();
  }
  ~GlobalS3Filesystem() {
    if (!arrow::fs::EnsureS3Finalized().ok()) {
      // handle error
    }
  }
  std::shared_ptr<arrow::fs::FileSystem> s3fs;
};

std::shared_ptr<arrow::fs::FileSystem> GetGlobalS3Filesystem() {
  static GlobalS3Filesystem global_s3fs;
  return global_s3fs.s3fs;
}

[AcidChristLab]

Use pyarrow 11.0.0

[westonpace]

It's possible this was fixed in 13.0.0. You still need to call FinalizeS3 as I described but we did fix a bug where users could call FinalizeS3 correctly and still get a crash.

[amoeba]

Hi @phalexo, could you provide a minimal reproduction here for us to test? It's not clear to me whether you're just using PyArrow and are encountered an issue or if you're doing something more complicated.

[imatiach-msft]

running into this issue as well - we aren't even using amazon S3 (we are using Microsoft Azure) but for some reason we are still seeing this error which is causing our builds to crash after hitting the 6 hour time limit transiently (sometimes builds pass, sometimes they do not):
microsoft/responsible-ai-toolbox#2212
using latest pyarrow==12.0.1

[pitrou]

@phalexo @imatiach-msft This should have been fixed in the about-to-be-released version 13.0.0. Can you perhaps try a nightly build? See https://arrow.apache.org/docs/dev/developers/python.html#installing-nightly-packages for how to install them.

[tvalentyn]

Seeing this error in some unit tests that use pyarrow, which we run in Apache Beam. We run unit tests on multiple os. After upgrading to pyarrow==13.0.0 in the windows tests, we are seeing

C:\arrow\cpp\src\arrow\filesystem\s3fs.cc:2829:  arrow::fs::FinalizeS3 was not called even though S3 was initialized.  This could lead to a segmentation fault at exit

and tests appear to timeout.

https://github.com/apache/beam/actions/runs/6177998648/job/16833521908?pr=28437