load binary folder - progress bar displaying incorrect number of files

[mathieudpnt]

Hi,

When I use a DateFilter with load_pamguard_binary_folder, the progress bar shows the total number of .pgdf files in the folder, rather than the number of files that actually match the date filter.

Example:

from pathlib import Path
from pandas import Timestamp

from pypamguard import load_pamguard_binary_folder
from pypamguard.core.filters import DateFilter, Filters
from pypamguard.logger import Verbosity, logger

audio_path = Path(r"Z:\Bioacoustique\DATASETS\CETIROISE\F_Sud_Liniou\Phase_10\Sylence")
binary = Path(r"L:\acoustock\Bioacoustique\DATASETS\CETIROISE\ANALYSE\PAMGUARD_threshold_7\PHASE_10_POINT_F")

begin = Timestamp("2025-01-20T12:25:00+00:00")
end = Timestamp("2025-01-25T09:00:00+00:00")

logger.set_verbosity(Verbosity.ERROR)

filter_obj = Filters(
    {
    "daterange": DateFilter(start_date=begin, end_date=end, ordered=True)
    }
)

data, _, _ = load_pamguard_binary_folder(binary, "*/*.pgdf", filters=filter_obj)

Output:

>>> data, _, _ = load_pamguard_binary_folder(binary, "*/*.pgdf", filters=filter_obj)
  4%|▎         | 111/3050 [00:10<04:35, 10.65it/s]
>>>

Even though only a subset of files is actually loaded based on the date filter, the progress bar shows the total number of files (3050) rather than the filtered number.

I’ve considered pre-filtering files by their datetime in the filename, but I’m afraid this might exclude detections at the boundaries (last file before the start date or first file after the end date).

Is there a recommended way to make the progress bar reflect the actual number of files processed after filtering?

Thanks!

[sullivan-james]

Maybe not the most elegant solution but perhaps this could work... I am not massively familiar with tqdm, but I think that closing the resource will complete the progress bar and flush the change to the console.

Without reading the index files, or some kind of pre-processing (as you mentioned), there is no way to predict the total number of files that will match the filter. Even with the change below, the progress bar will look quite strange, quickly jumping to the end once FILTER_POSITION.STOP is reached. Progress bar is quite low priority and it would be good to prevent unecessary code waffle - this solution might be the best. If you have any other suggestions though, I would love to hear them.

See docs here: https://tqdm.github.io/docs/tqdm/

progress_bar = tqdm(glob.glob(pathname=mask, root_dir=directory, recursive=True))
try:
   for file in progress_bar:
       filter_copy = copy.deepcopy(filters)
       res = load_pamguard_binary_file(os.path.join(directory,file), filters=filter_copy, report=report, clear_fields=clear_fields)
       res.add_file_info()
       data.extend(res.data)
       background.extend(res.background)
       if res.filters.position == FILTER_POSITION.STOP: break
finally:
   progress_bar.close()

I haven't tested this code. If you are happy with this solution, feel free to submit a PR.

[mathieudpnt]

I completely agree, sticking to a simple approach is best, especially since this is low-priority. PR #15 should work fine; the progress bar will show the number of processed files without displaying the total.