NP02 datasets

[ahiguera-mx]

There is a new request to generate datasets from NP02 from cosmics runs and HV scans.
However, when making the corresponding Rucio containers, Rucio datasets contain what look like TP files, e.g. np02vd_tp_run039350_2975_tp-stream-writer_tpw_0_20250913T094525.hdf5 see for more files vd-protodune:vd-protodune_39350

Suggested action plan:
Create a new MetaCat/Rucio dataset for the TP files as vd-protodune: vd-protodune_trigger_primitives_39350 and detach those files from vd-protodune:vd-protodune_39350, any comments @StevenCTimm

[StevenCTimm]

We should follow the convention that was used from hd-protodune which is
vd-protodune:vd-protodune_trigprim_

During the actual beam running of np02 most of the runs did not have trigger primitives but since then they
have turned them back on, and it is an issue. Had pleaded with DAQ to put trigger primitives in one directory
and raw data in another but they won't do it.

I did not see the actual description of the datasets come in from Elisabetta. I was waiting for that.
Do you have the information?

As far as doing the trigprim split I have some system to do it because I had to do it for all the hd-protodune runs.
Let me know which run numbers need to be split and I can do it.

[ahiguera-mx]

Hi Steve
The information about datasets can be found at /exp/dune/app/home/duneproshift/np02/all.tgz
The following runs need to be split

39350: vd-protodune:vd-protodune_39350
40222: vd-protodune:vd-protodune_40222
40168: vd-protodune:vd-protodune_40168
40097: vd-protodune:vd-protodune_40097
40140: vd-protodune:vd-protodune_40140
40266: vd-protodune:vd-protodune_40266
40267: vd-protodune:vd-protodune_40267
40270: vd-protodune:vd-protodune_40270
40285: vd-protodune:vd-protodune_40285

Additional runs will need to be split. I will post the other runs later in the week

[ahiguera-mx]

Additional runs that need to be split

vd-protodune:vd-protodune_40343
vd-protodune:vd-protodune_40344
vd-protodune:vd-protodune_40345
vd-protodune:vd-protodune_40346
vd-protodune:vd-protodune_40347
vd-protodune:vd-protodune_40348
vd-protodune:vd-protodune_40361
vd-protodune:vd-protodune_40362
vd-protodune:vd-protodune_40363
vd-protodune:vd-protodune_40379
vd-protodune:vd-protodune_40380
vd-protodune:vd-protodune_40381
vd-protodune:vd-protodune_40382
vd-protodune:vd-protodune_40385
vd-protodune:vd-protodune_40393
vd-protodune:vd-protodune_40400
vd-protodune:vd-protodune_40402
vd-protodune:vd-protodune_40406
vd-protodune:vd-protodune_40419
vd-protodune:vd-protodune_40420
vd-protodune:vd-protodune_40423
vd-protodune:vd-protodune_40424
vd-protodune:vd-protodune_40426
vd-protodune:vd-protodune_40430
vd-protodune:vd-protodune_40431
vd-protodune:vd-protodune_40433
vd-protodune:vd-protodune_40436
vd-protodune:vd-protodune_40439
vd-protodune:vd-protodune_40460
vd-protodune:vd-protodune_40471
vd-protodune:vd-protodune_40475
vd-protodune:vd-protodune_40499
vd-protodune:vd-protodune_40515
vd-protodune:vd-protodune_40538
vd-protodune:vd-protodune_40547
vd-protodune:vd-protodune_40558
vd-protodune:vd-protodune_40568
vd-protodune:vd-protodune_40576

[ahiguera-mx]

Here are the official datasets and containers for the raw data. Once the trigger primitives are detached the 1:1 correspondence will be there, these datasets are already with a non-expired rule at CERN_PDUNE_CASTOR, DUNE_CERN_EOS, and FNAL_DCACHE

vd-protodune:vd-protodune__raw__cosmics__39350_v1_official
vd-protodune:vd-protodune__raw__cosmics__hvscan_hv154_2025_v1_official
vd-protodune:vd-protodune__raw__cosmics__hvscan_hv157_2025_v1_official
vd-protodune:vd-protodune__raw__cosmics__hvscan_hv171_2025_v1_official
vd-protodune:vd-protodune__raw__cosmics__hvscan_hv188_period1_2025_v1_official
vd-protodune:vd-protodune__raw__cosmics__hvscan_hv188_period2_2025_v1_official
vd-protodune:vd-protodune__raw__cosmics__hvscan_hv200_2025_v1_official
vd-protodune:vd-protodune__raw__cosmics__hvscan_hv220_2025_v1_official
vd-protodune:vd-protodune__raw__cosmics__hvscan_hv230_2025_v1_official
vd-protodune:vd-protodune__raw__cosmics__hvscan_hv240_2025_v1_official

[StevenCTimm]

OK starting now to split:

[StevenCTimm]

The split metacat datasets are made.. now filling the corresponding rucio datasets.

[StevenCTimm]

The split is done but I believe I now have to adjust the containers.. checking

[ahiguera-mx]

| vd-protodune:vd-protodune_trigprim_39350 | DATASET |
| vd-protodune:vd-protodune_raw_39350 | DATASET |
| vd-protodune:vd-protodune_39350 | DATASET |

The containers use vd-protodune:vd-protodune_39350, are those now empty?

[ahiguera-mx]

vd-protodune:vd-protodune_raw_39350 is empty on rucio
vd-protodune:vd-protodune_39350, still contains TP files

[StevenCTimm]

yes I know
I did not remove the trigger primitives from vd-protodune:vd-protodune_39350
(and friends)

I made a new dataset vd-protodune:vd-protodune_raw_39350 and vd-protodune:vd-protodune_trigprim_39350
and am one by one attaching the raw datasets to the containers.
Discovered there was a problem with the first raw dataset when I did that, now fixed
moving on to the other containers

[StevenCTimm]

Done, now checking all

[StevenCTimm]

There are no rules yet associated with the vd-protodune_raw_ datasets but there will be.. we will do that in a bunch later. In the meantime the vd-protodune_ rules are holding them on tape and disk.

[StevenCTimm]

OK verified that metacat official datasets and rucio containers are all the same size
These sets are only raw data.. aren't there supposed to be some for reco as well?