Run pipeline for non-P1 and P5 samples

[AndreOKovacs]

Hello,

I'm trying to run the LESIA L1 pipeline from platonium, but I'm getting the following error:

platonium 1 1 1 1 --project toi-201 --sample TOI-201 --pipeline --varfile /Users/andre/plato/platosim_workdir/toi-201/input/varsource_wl.txt -v 3 -w --pipe_psf microscan

PLATOnium

Loading stellar catalogue..

Initialize and configure PlatoSim

Applying pointing errors     (PRE FromFile)
Applying camera misalignment (APE FromFile)
Applying camera drift        (TED FromFile)
Applying platform jitter     (ACS RedNoise)
Applying thermal transients  (GTT FromFile)

Information about stellar target
        gaiaDR3   ra [deg]  dec [deg]      mag CCD   xCCD [pix]  yCCD [pix]  rOA [deg]  xFP [mm]  yFP [mm]  Ncon
0  4.767548e+18  87.401784 -54.910414  8.74134   4  3061.915012  4468.40699  12.847645  2.048674  56.41447     0

Star catalog used in simulation
    RA [deg]  Dec [deg]  P [mag]  Dis [pix]
0  87.401784 -54.910414  8.74134        0.0 

Applying variable-source file (varsource_wl.txt)
Applying on-board photometry  (photometry.txt)

[PlatoSim]: Simulating N-CAM 1.1 Q1 for 312113 exposures

Simulation statistics
------------------------------------------------------------
Max RAM memory for PlatoSim      : 1.0 MB
Storage memory for PlatoSim      : 166.0 MB
Execution time for PlatoSim      : 0:02:06.560765 [h:mm:ss]
Traceback (most recent call last):
  File "/Users/andre/anaconda3/envs/platonium-pipeline/bin/platonium", line 2218, in <module>
    p.resources()
  File "/Users/andre/anaconda3/envs/platonium-pipeline/bin/platonium", line 2084, in resources
    print(f'Execution time for Microscanning : {self.tocMicroscan} [h:mm:ss]')
                                                ^^^^^^^^^^^^^^^^^
AttributeError: 'PLATOnium' object has no attribute 'tocMicroscan'. Did you mean: 'run_microscan'?

Unfortunately, after checking the source code, it seems that platonium supports only P1 and P5 samples for the pipeline execution using the microscan PSF method (it doesn't run the on-ground L0-L1 pipeline chain and, instead, runs only the PlatoSim time series)?

For non-P1 and P5 samples I would need to manually run the pipeline following the steps described here? https://github.com/IvS-KULeuven/PlatoSim3/issues/955

Thanks,
Andre

[AndreOKovacs]

Hello,

I believe I found a solution for the error above, by creating the input catalog file from SIMBAD using the following command:

picsim --release PIC210 --simbad TOI-201 --pipe_sample P1 --pipe_field LOPS2 --project toi-201 -p

Thanks,
Andre

[AndreOKovacs]

Hello,

The pipeline runs fine for the N-CAMs, but it fails with the following error message for the F-CAMs:

platonium 1 5 1 1 --project toi-201 --nexp 314496 --sample P1 --pipeline --varfile /Users/andre/plato/platosim_workdir/toi-201/input/varsource_blue.txt -v 3 -w --pipe_psf microscan

PLATOnium

Loading stellar catalogue..

Initialize and configure PlatoSim

Applying pointing errors     (PRE FromFile)
Applying camera misalignment (APE FromFile)
Applying platform jitter     (ACS RedNoise)
Applying thermal transients  (GTT FromFile)

Information about stellar target
        gaiaDR3   ra [deg]  dec [deg]      mag CCD   xCCD [pix]  yCCD [pix]  rOA [deg]   xFP [mm]   yFP [mm]  Ncon
0  4.767548e+18  87.401784 -54.910414  8.74134   1  1377.130335  3078.84597   8.634963 -26.088346  27.060773     0

Star catalog used in simulation
    RA [deg]  Dec [deg]  P [mag]  Dis [pix]
0  87.401784 -54.910414  8.74134        0.0 

Applying variable-source file (varsource_blue.txt)
Applying on-board photometry  (photometry.txt)

[PlatoSim]: Simulating F-CAM blue Q1 for 314496 exposures

Microscanning & PSF inversion

[PlatoSim]: Simulating 428 imagettes along Archimedean spiral

[psim2datastruc]: Pre-processing imagettes
Traceback (most recent call last):
  File "/Users/andre/anaconda3/envs/platonium-pipeline/bin/platonium", line 2192, in <module>
    p.run_L1_microscan()
  File "/Users/andre/anaconda3/envs/platonium-pipeline/bin/platonium", line 1587, in run_L1_microscan
    comm = f'psim2datastruc --cam-id {self.cameraID:02d} --prnu_err {self.pipePrnuError} --seed {self.seedTarget} --mag-error {mag_err} --centroid-err {self.pipeAbsCenError} --target_id 1 . {self.starID} {self.starID} 6'
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Unknown format code 'd' for object of type 'str'

Checking the psim2datastruct, it seems that platonium should pass the camera ID 25, instead of 1 (blue F-CAM), when the cadence is 25s:
"--cam-id : ID of the camera. Otherwise ID=1 when the cadence is 25s and ID=25 when the cadence is 2.5s"

So, I applied the following patch to the run_L1_microscan() and it seems to work just fine now:

        if self.groupID == 'Fast':
            comm = f'psim2datastruc --cam-id 25 --prnu_err {self.pipePrnuError} --seed {self.seedTarget} --mag-error {mag_err} --centroid-err {self.pipeAbsCenError} --target_id 1 . {self.starID} {self.starID} 6'
        else:
            comm = f'psim2datastruc --cam-id {self.cameraID:02d} --prnu_err {self.pipePrnuError} --seed {self.seedTarget} --mag-error {mag_err} --centroid-err {self.pipeAbsCenError} --target_id 1 . {self.starID} {self.starID} 6'

But, I'm now getting the following error message from psim2datastruct:

platonium 1 5 1 1 --project toi-201 --nexp 314496 --sample P1 --pipeline --varfile /Users/andre/plato/platosim_workdir/toi-201/input/varsource_blue.txt -v 3 -w --pipe_psf microscan

PLATOnium

Loading stellar catalogue..

Initialize and configure PlatoSim

Applying pointing errors     (PRE FromFile)
Applying camera misalignment (APE FromFile)
Applying platform jitter     (ACS RedNoise)
Applying thermal transients  (GTT FromFile)

Information about stellar target
        gaiaDR3   ra [deg]  dec [deg]      mag CCD   xCCD [pix]  yCCD [pix]  rOA [deg]   xFP [mm]   yFP [mm]  Ncon
0  4.767548e+18  87.401784 -54.910414  8.74134   1  1377.130335  3078.84597   8.634963 -26.088346  27.060773     0

Star catalog used in simulation
    RA [deg]  Dec [deg]  P [mag]  Dis [pix]
0  87.401784 -54.910414  8.74134        0.0 

Applying variable-source file (varsource_blue.txt)
Applying on-board photometry  (photometry.txt)

[PlatoSim]: Simulating F-CAM blue Q1 for 314496 exposures

Microscanning & PSF inversion

[PlatoSim]: Simulating 428 imagettes along Archimedean spiral

[psim2datastruc]: Pre-processing imagettes
/Users/andre/plato/platosim_workdir/toi-201/output/microscan/P1/Q1/Ncam51/000000001
psim2datastruc --cam-id 25 --prnu_err 0.1 --seed 123457794 --mag-error 0.010857362047581295 --centroid-err 0.03 --target_id 1 . 000000001 000000001 6
This is psim2datastruc.py 
Target star ID: 000000001
Skipped overscan rows number = 10
Contaminant kept if distance <= 8.5 pixels
seed value: 123457794
Imagette corrections (offset, smearing, prnu, gain, background): ON
Smearing Imagette correction (offset, gain): ON
Input directory: ./000000001
Output directory: 000000001
Using : ./000000001/000000001.hdf5
FCam mode: OFF
Traceback (most recent call last):
  File "/Users/andre/anaconda3/envs/platonium-pipeline/bin/psim2datastruc", line 2179, in <module>
    main()
  File "/Users/andre/anaconda3/envs/platonium-pipeline/bin/psim2datastruc", line 2148, in main
    generate_data(
  File "/Users/andre/anaconda3/envs/platonium-pipeline/bin/psim2datastruc", line 784, in generate_data
    assert camera_id >= CameraId.N_G1C1 and camera_id <= CameraId.N_G4C6, "camera ID not consistent with the 25s cadence"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: camera ID not consistent with the 25s cadence
[Error]: psim2datastruc failed due to the above error!

This issue seems to be related to the following check in init_sim() from platonium, that's not saving the 2.5s cadence into the sim['ObservingParameters/CycleTime'] attribute for the F-CAMs, unless the cadence is forced with the --cadence parameter:

        # Cadence of time series [s]
        # NOTE CCD offset is automatically set by setSubfieldAroundCoordinates()
        # NOTE sim.useNormalCameras() and sim.useFastCameras() overwrites cadenc!
        if self.cadence:
            sim['ObservingParameters/CycleTime'] = self.cadence
        else:
            if self.groupID == 'Fast':
                self.cadence = 2.5
            else:
                self.cadence = sim['ObservingParameters/CycleTime']

So, I included the --cadence 2.5 parameter to force the 2.5s cadence, but the error persisted.
Checking again the platonium code, I noticed that the run_microscan() is overriding the sim['ObservingParameters/CycleTime'] attribute with a hardcoded value of 25.
After applying the following fix to run_microscan(), I finally managed to make psim2datastruc to understand that the data is from the F-CAM:

        if self.groupID == 'Fast':
            sim['ObservingParameters/CycleTime']    = 2.5
        else:
            sim['ObservingParameters/CycleTime']    = 25

But psim2datastruct is now generating the following error:

platonium 1 5 1 1 --project toi-201 --cadence 2.5 --nexp 314496 --sample P1 --pipeline --varfile /Users/andre/plato/platosim_workdir/toi-201/input/varsource_blue.txt -v 3 -w --pipe_psf microscan

PLATOnium

Loading stellar catalogue..

Initialize and configure PlatoSim

Applying pointing errors     (PRE FromFile)
Applying camera misalignment (APE FromFile)
Applying platform jitter     (ACS RedNoise)
Applying thermal transients  (GTT FromFile)

Information about stellar target
        gaiaDR3   ra [deg]  dec [deg]      mag CCD   xCCD [pix]  yCCD [pix]  rOA [deg]   xFP [mm]   yFP [mm]  Ncon
0  4.767548e+18  87.401784 -54.910414  8.74134   1  1377.130335  3078.84597   8.634963 -26.088346  27.060773     0

Star catalog used in simulation
    RA [deg]  Dec [deg]  P [mag]  Dis [pix]
0  87.401784 -54.910414  8.74134        0.0 

Applying variable-source file (varsource_blue.txt)
Applying on-board photometry  (photometry.txt)

[PlatoSim]: Simulating F-CAM blue Q1 for 314496 exposures

Microscanning & PSF inversion

[PlatoSim]: Simulating 428 imagettes along Archimedean spiral

[psim2datastruc]: Pre-processing imagettes
/Users/andre/plato/platosim_workdir/toi-201/output/microscan/P1/Q1/Ncam51/000000001
psim2datastruc --cam-id 25 --prnu_err 0.1 --seed 123457794 --mag-error 0.010857362047581295 --centroid-err 0.03 --target_id 1 . 000000001 000000001 6
This is psim2datastruc.py 
Target star ID: 000000001
Skipped overscan rows number = 10
Contaminant kept if distance <= 8.5 pixels
seed value: 123457794
Imagette corrections (offset, smearing, prnu, gain, background): ON
Smearing Imagette correction (offset, gain): ON
Input directory: ./000000001
Output directory: 000000001
Using : ./000000001/000000001.hdf5
FCam mode: ON
Camera ID = 25
Dealing with a F-cam
PlatoSim Transfer Time time = 0.20339999999999997 seconds
PlatoSim Exposure time = 2.2966 seconds
IncludeAberrationCorrection = 1
On board time = 101489.20703
Analytical Non Gaussian PSF
Using diffusedPSF
PSF: barycenter x = 3.8945341916611147, by= 3.9872624396477945
PSF Chromatic aberration offset ca_dx = -0.1054658083388853, ca_dy = -0.012737560352205524
Assuming a B-spline of type 1
The PSF is decomposed using B-splines with a resolution of 20 nodes per pixel and it is centered at the middle of its window
size (ny,nx)=(1024, 1024)
sub-resolution (subres)=128.0
size in pixels (sizex,sizey)=(8, 8)
total number of elements=1048576
barycenter (bx,by)=(3.8945328950574707, 3.9872611121722206)
offset (x0,y0)=(0.0, 0.0)
sum =1.0
New PSF Chromatic aberration offset ca_dx = -1.2988357882548485e-06, ca_dy = -1.328664974487026e-06
StarCatalog target coordinates : 3.8331528953776797, 4.024934330624319
StarPosition Exposure 0 target coordinates (row,col) : 3.819168634687685, 4.010303450442507
                               CCD positions (X,Y) : 1376.9953773413497, 3078.8460017981715
                               FP positions (X,Y) : -26.088347202501335, 27.06077397179686
CCDI D: 1, time shift: 0.000000
OriginOffsetX=[-1.3 -1.3 -1.3 -1.3]
OriginOffsetY=[-1.3 -1.3 -1.3 -1.3]
Orientation=[  0.  90. 180. 270.]
writing 000000001/000000001_c25_igm.hdf5
writing 000000001/000000001_psf.hdf5
Traceback (most recent call last):
  File "/Users/andre/anaconda3/envs/platonium-pipeline/bin/psim2datastruc", line 2179, in <module>
    main()
  File "/Users/andre/anaconda3/envs/platonium-pipeline/bin/psim2datastruc", line 2148, in main
    generate_data(
  File "/Users/andre/anaconda3/envs/platonium-pipeline/bin/psim2datastruc", line 1146, in generate_data
    star_positions_time = np.array(psim_output_file["StarPositions/time"]) + exposure_time / 2
                                   ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/Users/andre/anaconda3/envs/platonium-pipeline/lib/python3.11/site-packages/h5py/_hl/group.py", line 367, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 255, in h5py.h5o.open
KeyError: "Unable to synchronously open object (object 'time' doesn't exist)"
[Error]: psim2datastruc failed due to the above error!

Thanks,
Andre

[AndreOKovacs]

Hello,

I was checking the code for the generate_data() from psim2datastruc and it seems that the issue might be related to some change to the calculation of the star_positions_time done only for the N-CAM case, and probably missing for the F-CAM case. So, I fixed it to also use the new calculation method for the F-CAM:

    # Be aware that timestamp of exposure from Platosim is the beginning of the integration time and not the end of
    # integration time
    if is_fcam:
        # star_positions_time = np.array(psim_output_file["StarPositions/time"]) + exposure_time / 2
        star_positions_time = np.arange(len(psim_output_file["Images"])) * 2.5 + exposure_time / 2
    else:
        star_positions_time = np.arange(len(psim_output_file["Images"])) * 25.0 - time_shift + (exposure_time + 6.25 * 3) / 2
        # star_positions_time = np.array(psim_output_file["StarPositions/Time"]) - time_shift + (exposure_time + 6.25 * 3) / 2

Then, a similar issue as above for the call of psim2datastruc in run_L1_microscan() from platonium happened also for run_L1_onground(), and I applied a similar fix:

        if self.groupID == 'Fast':
            comm = f'psim2datastruc --cam-id 25 --prnu_err {self.pipePrnuError} --seed {self.seedTarget} --mag-error {mag_err} --centroid-err {self.pipeAbsCenError} --target_id 1 . {self.starID} {self.starID} 6'
        else:
            comm = f'psim2datastruc --cam-id {self.cameraID:02d} --prnu_err {self.pipePrnuError} --seed {self.seedTarget} --mag-error {mag_err} --centroid-err {self.pipeAbsCenError} --target_id 1 . {self.starID} {self.starID} 6'

So, to be sure, I fixed also a similar logic for the call of psim2datastruc in run_L1_onboard() from platonium:

        if self.groupID == 'Fast':
            comm = f'psim2datastruc --cam-id 25 --prnu_err {self.pipePrnuError} --seed {self.seedTarget} --mag-error {mag_err} --centroid-err {self.pipeAbsCenError} --target_id 1 . {self.starID} {self.starID} 6'
        else:
            comm = f'psim2datastruc --cam-id {self.cameraID:02d} --prnu_err {self.pipePrnuError} --seed {self.seedTarget} --mag-error {mag_err} --centroid-err {self.pipeAbsCenError} --target_id 1 . {self.starID} {self.starID} 6'

Thanks,
Andre

[AndreOKovacs]

Hello,

Now I'm getting the following error, when platonium tries to retrieve the hdf5 files back from the psim2@datastruc output:

platonium 1 5 1 1 --project toi-201 --cadence 2.5 --nexp 314496 --sample P1 --pipeline --varfile /Users/andre/plato/platosim_workdir/toi-201/input/varsource_blue.txt -v 3 -w --pipe_psf microscan

PLATOnium

...

Prologue

Restructuring data output
L1 light curve is saved to /Users/andre/plato/platosim_workdir/toi-201/output/reduced/P1/000000001
prefixInversion /Users/andre/plato/platosim_workdir/toi-201/output/microscan/P1/Q1/Ncam51/000000001/inversion/000000001
prefixStarIDsim /Users/andre/plato/platosim_workdir/toi-201/output/P1/Q1/Ncam51/000000001/000000001
prefixStarIDnew /Users/andre/plato/platosim_workdir/toi-201/output/reduced/P1/000000001/000000001_Fcam1_Q1
create_sim_table(/Users/andre/plato/platosim_workdir/toi-201/output/reduced/P1/000000001)
Traceback (most recent call last):
  File "/Users/andre/anaconda3/envs/platonium-pipeline/bin/platonium", line 2208, in <module>
    p.sort_output_pipeline()
  File "/Users/andre/anaconda3/envs/platonium-pipeline/bin/platonium", line 1896, in sort_output_pipeline
    lc_file = f"{self.outputDirStarIDsim}/LIGHTCURVE_L1A_IMAGETTE_c{self.cameraID:02d}_p000000001.hdf5"
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Unknown format code 'd' for object of type 'str'

So, I had to adjust all references from {self.cameraID:02d} to 25 in the filenames, in platonium function sort_output_pipeline(), when the F-CAMs are used with the P1 sample:

       # Fetch P1 light curve
        if args.sample == 'P1':
            if self.groupID == 'Fast':
                lc_file = f"{self.outputDirStarIDsim}/LIGHTCURVE_L1A_IMAGETTE_c25_p000000001.hdf5"
                cob_file = f"{self.outputDirStarIDsim}/COB_OG_c25_p000000001.hdf5"
                skypos_file = f"{self.outputDirStarIDsim}/SKYPOS_L1A_IMAGETTE_c25_p000000001.hdf5"
            else:
                lc_file = f"{self.outputDirStarIDsim}/LIGHTCURVE_L1A_IMAGETTE_c{self.cameraID:02d}_p000000001.hdf5"
                cob_file = f"{self.outputDirStarIDsim}/COB_OG_c{self.cameraID:02d}_p000000001.hdf5"
                skypos_file = f"{self.outputDirStarIDsim}/SKYPOS_L1A_IMAGETTE_c{self.cameraID:02d}_p000000001.hdf5"

Something similar to this fix might be also necessary for the P5 sample!

And now the pipeline finishes executing for both N-CAMs and F-CAMs.

Thanks,
Andre