CMS: import the generator fragments

[katilp]

For generator fragments, we link directly to mcm.
Example: http://opendata-dev.web.cern.ch/record/808#
where Generator parameters link brings directly to
https://cms-pdmv.cern.ch/mcm/public/restapi/requests/get_fragment/FSQ-LowPU2010-00013

This is fine for the moment but for long-term preservation, we should import the contents.

[katilp]

The link has also changed, but for the moment, redirecting to the new link works.
For example, the one above opens now in
https://cms-pdmv-prod.web.cern.ch/mcm/public/restapi/requests/get_fragment/FSQ-LowPU2010-00013

Right now (before adding the 2016 MC), there are 2085 of these (in the "url" json field of the provenance section)

$ grep cms-pdmv.cern.ch/mcm/public/restapi cernopendata/modules/fixtures/data/records/cms*json | grep '"url":' |
 wc -l
2085

[katilp]

@tiborsimko @jmhogan @tpmccauley
We should have a subdirectory for these fragments and copy them over.
We have /eos/opendata/cms/configuration-files/ but that's not the best place.
Some of these fragments can be shared between MC of different years so I would have them in a new separate directory,
e.g. /eos/opendata/cms/fragments/ or /eos/opendata/cms/generator-fragments/
Your thoughts?

[jmhogan]

@katilp Agreed, that would be a good folder name, I like generator-fragments best

[d0leh]

I have created generator-fragments directory in /eos/opendata/cms/upload/mohammad/, where it has all the fragments.

It currently has 2058 Python scripts, as 27 (2085 - 2058) fragments have occurred more than once and were overwritten with the same code.