stream not properly closed (NoBackendError | OSError: Too many open files)

[ne555]

import audioread

filename = 'some_audio_file.ogg'
try:
    for K in range(2048):
        with audioread.audio_open(filename) as audio:
            print(K, audio.duration)
except audioread.exceptions.NoBackendError:
    with open(filename, 'rb') as file: # OSError: [Errno 24] Too many open files
        pass

the audio is openned in a loop using with, but it seems that is not properly closed
in my system it will print about 500 lines, then raises the NoBackendError and the OSError when trying to open a new file

using lsof shows lines of the form (my limit is 1024)

COMMAND   PID    USER   FD      TYPE             DEVICE SIZE/OFF     NODE NAME
python  38124 ne555 1023u     unix 0x00000000d813f451      0t0  2420780 type=STREAM (CONNECTED)

I've observed this issue with audioread.gstdec.GstAudioFile and audioread.ffdec.FFmpegAudioFile backends, couldn't test with audioread.rawread.RawAudioFile

[sampsyo]

Thanks for the complete script for testing this! I wasn't able to reproduce this with some quick testing (macOS, FFmpeg backend, using lsof to check if things went out of control). I don't have access to something using Gstreamer at the moment, but I would very much believe that that backend could have some kind of a leak.

For FFmpeg in particular, is there any chance you can check whether the loopy script also leaves a bunch of ffmpeg processes running? Like, I can imagine ps | grep ffmpeg showing 1024 processes if we're not correctly cleaning things up there.

[ne555]

good morning, I didn't realise that before trying to open the file with FFmpegAudioFile it was using GstAudioFile
when limited the backends to only FFmpegAudioFile it worked fine.

so the issue seems to be only with the GstAudioFile backend (either if it opens or not the file correctly)

[sampsyo]

Ah, that makes sense! I unfortunately don't have a great way to test this out here… I can imagine that we may need to do some additional explicit resource cleanup here:

audioread/audioread/gstdec.py

Lines 378 to 406 in ff9535d

	def close(self, force=False):
	"""Close the file and clean up associated resources.
	Calling `close()` a second time has no effect.
	"""
	if self.running or force:
	self.running = False
	self.finished = True
	# Unregister for signals, which we registered for above with
	# `add_signal_watch`. (Without this, GStreamer leaks file
	# descriptors.)
	self.pipeline.get_bus().remove_signal_watch()
	# Stop reading the file.
	self.dec.set_property("uri", None)
	# Block spurious signals.
	self.sink.get_static_pad("sink").disconnect(self.caps_handler)
	# Make space in the output queue to let the decoder thread
	# finish. (Otherwise, the thread blocks on its enqueue and
	# the interpreter hangs.)
	try:
	self.queue.get_nowait()
	except queue.Empty:
	pass
	# Halt the pipeline (closing file).
	self.pipeline.set_state(Gst.State.NULL)

But it will require some real GStreamer expertise or trial and error to figure out exactly what to clean up.

[nikita-petrashen]

I have run into the same problem while using librosa.load.
Passing a file descriptor instead of the path helped:

with open(path, "rb") as fp:
    librosa.load(fp)