Filtered Particles: Mark as Duplicate

[skuschel]

Some PIC Codes (such as EPOCH) have the ability to dump particle subsets based on a variable set of requirements. For Example: in LWFA dump only the electrons that have gamma >2. Is this planned?

Alternative Idea

@skuschel commented

Instead of marking duplicates we could mark entries as 'original' data, meaning: not downsampled, not only a part of the simulation grid: the grid is the actual full simulation grid and the data is the actual quantitiy, that was used for calculations. for a particle species: the particle species is complete, this is not a subset.

[ax3l]

thank you for the input!
yes, that is definitely already possible. just create a particle record with a distinct name and fill in the particles there (it does not matter if one stores the full particle set again in an other particle record). In case of a gamma filter, I would write an explanation in the comment attribute.

[skuschel]

Ah i see, so you would suggest that this is just handeled as another particle species. But that means that if I calculate charge density -- and therefore run over all species that are dumped -- I may count particles twice. Let me add an example. LWFA simulaton, particle species are -- lets ignore ions:
background_electrons, N6_elec (6th ionization level of N2), N7_elec
i also add a subset
trapped_electrons, where all particles are in which have a gamma > 10.
If I calculate charge density now, i run over all available species and thus counting electonns in trapped_electrons twice.
Using the sdf file format allows to destinguish what species are 'original' and which are subsets/replica.
Is there a mechnism in OpenPMD to destinguish those species from another?

And just for completeness, the same question for the fields. A Code could dump electric and magnetic fields on a downsampled grid as well to save disk space. How are those saved?

[ax3l]

Ah now I see your point.
One thing is, that in the first version we did not define what a "complete" set of records, e.g., for PIC simulations, shall be. This would be required to allow to characterize an output as "restart(able) file".

Adding an attribute such as filter or subsampling (1 or 0; "None", "Momentum", "Position", "Origin", "Stride", "Other") is a good idea! (I would like to avoid "subset" since it might be mismatched with "patches" which are actually not filtered but just ordered in smaller groups).

Since I can think of an infinite number of filtering methods, we could directly add a (string) attribute subsamplingComment were users can describe the filter they used for subsampling in a human-readable format (automated parsing might not be very important for that aspect, only the clear detection of "None" is important).

An other attribute that might be interesting is the original number of particles that was subsampled record was derived from, e.g., in an attribute subsamplingSourceSize and maybe optional the name of the record that actually contains the full data set, e.g., subsamplingSource.

For fields (meshes) the question is similar to #10: the description of applied down-sampling methods (needs specification).
Nevertheless, naming is easier since usually the two specific fields are E and B (so one can simply call the other down-sampled fields and auxiliary fields differently).

[ax3l]

Updated description above.

Should be added to STANDARD.md. Similar names should be used for subsampled meshes (#10).

[RemiLehe]

After discussing with @ax3l, I think we would leave this out for the first release and have a look at it in a future release. The reason is that:

• Filtered particle arrays can already be written in openPMD (there is no problem with writing only a subset of the particles)
• Although there is no specific attribute for tagging the dataset as being only a subset (for the first release), one can always add an (optional) comment attribute for this.

@skuschel : is that fine with you ?

[ax3l]

I think @skuschel will not be satisfied with the comment attribute since he wants to create an automated, added, total charge density from all particles that he can find.

nevertheless, I also think it is a bit too complicated for the first version if that is ok.

[skuschel]

I think this is kind of an advanced feature, so we can definitely shift it to some future release. Unfortnually a comment wont do the job, when you want to iterate over all particles without dublicates as ax3l mentioned (charge density, total number density, total kinetic energy on grid, you cant even work out the total particle count in the simulation....)