New ExoMiner Pipeline release on Podman 🤖

[migmartinho]

Hello! 👽

The Podman images under Packages just got updated!

Instructions to set ExoMiner Pipeline with Podman: here

Major Changes

• Option to use different ExoMiner++ models: Three models are based in the ExoMiner++ architecture and were developed as part of the experiment for the ExoMiner on TESS 2-min paper. They reflect a tradeoff in generalization and robustness vs computational cost. On one side, the single model from a single CV iteration exominer++_single is lightweight and will produce faster inference, while exominer++_cv-super-mean-ensemble is recommended for those users that have
  the computational resources to run a model that smooths out biases from individual CV iterations and tranining initializations. exominer++_cviter-mean-ensemble provides a balance between performance and complexity. Details on those models:
  • exominer++_single (~1M parameters): ExoMiner++ model that comes from the TESS+Kepler cross-validation (CV) experiment, more concretely model 0 in CV iteration 0.
  • exominer++_cviter-mean-ensemble (~10M parameters): ExoMiner++ model that comes from the TESS+Kepler CV experiment and corresponds to the mean ensemble of CV iteration 0.
  • exominer++_cv-super-mean-ensemble (~100M parameters): ExoMiner++ model that comes from the TESS+Kepler CV experiment and corresponds to the "super mean ensemble" of the ensembles of all CV iterations.

Minor Changes

• If the user decides to use target light curve TESS FITS files and TESS SPOC DV XML files that they have downloaded locally instead of using the pipeline to query the MAST for the queried targets and sector runs, the pipeline now only extracts the difference image data for the queried TICs, even if the user has DV XML files for other targets in the same directory.
• Validation of RUWE and stellar parameters catalogs: if the user chooses to use their own catalogs of stellar parameters and RUWE values for the queried TIC IDs, the pipeline checks the structure of those CSV files.

Updated shell script (until we update the GitHub repository) that works with the new ExoMiner Pipeline Podman packages:

#!/bin/bash

### Run ExoMiner Pipeline by running a container of a Podman image

### default values ## 

# directory where the inputs for the ExoMiner Pipeline are stored
inputs_dir="/data3/exoplnt_dl/experiments/exominer_pipeline/inputs"
# file path to the TICs table
tics_tbl_fn="test_tics_S72-72.csv"
tics_tbl_fp=$inputs_dir/$tics_tbl_fn
# name of the run
exominer_pipeline_run=test_exominer_pipeline_run_8-20-2025_1210
# directory where the ExoMiner Pipeline run is saved
exominer_pipeline_run_dir=/data3/exoplnt_dl/experiments/exominer_pipeline/runs/$exominer_pipeline_run
# data collection mode: either 2min or ffi
data_collection_mode="2min"
# number of processes
num_processes=1
# number of jobs to split the TIC IDs
num_jobs=1
# set to "true" or "false". If "true", it will create a CSV file with URLs to the SPOC DV reports for each TCE in the
# queried TICs
download_spoc_data_products=true
# path to a directory containing the light curve FITS files and DV XML files for the TIC IDs and sector runs that you
# want to query; set to "null" otherwise
external_data_repository=null
# define source of stellar parameters for TICs. If set to 'ticv8', TIC-8 is queried; if set to 'tess-spoc', it uses the
# parameters stored in the TICs DV XML files; if set to a filepath that points to an external catalog of stellar
# parameters, it will use those values.
stellar_parameters_source=ticv8
# define source of Gaia RUWE for TICs. If set to 'gaiadr2', Gaia DR2 is queried; if set to 'unavailable', it assumes the
# values are missing; if set to a filepath that points to an external catalog of RUWE parameters, it will use those
# values.
ruwe_source=gaiadr2
# which ExoMiner model to use for inference. Choose between "exominer++_single", "exominer++_cviter-mean-ensemble", and "exominer++_cv-super-mean-ensemble".
exominer_model="exominer++_single"

# Help message
show_help() {
    echo "Usage: $0 [options]"
    echo ""
    echo "Options:"
    echo "  --tics_tbl_fp FILE                   TIC IDs table filepath"
    echo "  --exominer_pipeline_run_dir DIR      Directory to store pipeline run output"
    echo "  --data_collection_mode MODE          Data collection mode (2min or ffi)"
    echo "  --num_processes N                    Number of processes"
    echo "  --num_jobs N                         Number of jobs"
    echo "  --download_spoc_data_products BOOL   Whether to download TESS SPOC DV reports for the detected TCEs: true or false"
    echo "  --external_data_repository DIR       Path to external data repository containing light curve FITS files and DV XML files for the TIC IDs in the TICs table"
    echo "  --stellar_parameters_source SOURCE   Source for TICs stellar parameters"
    echo "  --ruwe_source SOURCE                 Source for TICs Gaia RUWE parameters"
    echo "  --exominer_model MODEL               ExoMiner model to use for inference"
    echo "  --help                               Show ExoMiner Pipeline help"
    echo ""
    exit 
}

# Parse arguments
while [[ $# -gt 0 ]]; do
    case "$1" in
        --tics_tbl_fp) tics_tbl_fp="$2"; shift 2 ;;
        --exominer_pipeline_run_dir) exominer_pipeline_run_dir="$2"; shift 2 ;;
        --data_collection_mode) data_collection_mode="$2"; shift 2 ;;
        --num_processes) num_processes="$2"; shift 2 ;;
        --num_jobs) num_jobs="$2"; shift 2 ;;
        --download_spoc_data_products) download_spoc_data_products="$2"; shift 2 ;;
        --external_data_repository) external_data_repository="$2"; shift 2 ;;
        --stellar_parameters_source) stellar_parameters_source="$2"; shift 2 ;;
        --ruwe_source) ruwe_source="$2"; shift 2 ;;
        --exominer_model) exominer_model="$2"; shift 2 ;;
        --help)
            show_help
            exit 0
            ;;
        *)
            echo "Unknown option: $1"
            show_help
            ;;
    esac
done

mkdir -p $exominer_pipeline_run_dir

# set up volume mounts
volume_mounts="-v $tics_tbl_fp:/tics_tbl.csv:Z -v $exominer_pipeline_run_dir:/outputs:Z"

# conditionally add external_data_repository mount
if [ "$external_data_repository" != "null" ]; then
  volume_mounts="$volume_mounts -v $external_data_repository:/external_data_repository:Z"
  external_data_repository_arg="--external_data_repository=/external_data_repository"
else
  external_data_repository_arg=""
fi

# add mount to external TICs stellar parameters catalog if filepath provided
if [ -f "$stellar_parameters_source" ]; then
    volume_mounts="$volume_mounts -v $stellar_parameters_source:/tics_stellar_parameters.csv:Z"
    stellar_parameters_source_arg=/tics_stellar_parameters.csv
else
    stellar_parameters_source_arg=$stellar_parameters_source
fi

# add mount to external TICs RUWE catalog if filepath provided
if [ -f "$ruwe_source" ]; then
    volume_mounts="$volume_mounts -v $ruwe_source:/tics_ruwe.csv:Z"
    ruwe_source_arg=/tics_ruwe.csv
else
    ruwe_source_arg=$ruwe_source
fi

echo "Running ExoMiner Pipeline with the following parameters:"
echo "TICs table file: $tics_tbl_fp"
echo "ExoMiner Pipeline run directory: $exominer_pipeline_run_dir"

podman run \
  ${volume_mounts} \
  ghcr.io/nasa/exominer:latest \
  --tic_ids_fp=/tics_tbl.csv \
  --output_dir=/outputs \
  --data_collection_mode=$data_collection_mode \
  --num_processes=$num_processes \
  --num_jobs=$num_jobs \
  --download_spoc_data_products=$download_spoc_data_products \
  --stellar_parameters_source=$stellar_parameters_source_arg \
  --ruwe_source=$ruwe_source_arg \
  --exominer_model=$exominer_model \
  $external_data_repository_arg \

echo "Finished ExoMiner Pipeline run $exominer_pipeline_run_dir."

Happy hunting! 🛰️