[rl] Revert generator initialization race condition fix (#3809)

[wwwjn]

Reverts the initialization reordering from #3809, which spawned the trainer first and moved TorchStore init + policy_version read before generator spawn. Restores the original ordering: spawn trainer and generators together, then init TorchStore and read policy_version.

[tianyu-l]

I didn't get the "must not" part. Non-blocking should work with and without cudagraph, so shouldn't force it here?

If you want different version between trainer and generator. We can do similar things as how we separate DeepEP v2 for trainer and generator.

[wwwjn]

Non-blocking should work with and without cudagraph
The "must not" is not accurate, basically I want to allow different capacity factor (either None or a float) which needs to be configurable in trainer and generator. This override is too strict for trainer.

I already tried override to have separate HybridEP for trainer and generator, but the difficulty here is the trainer and generator might want different capacity factor number, but override doesn't support passing the capacity factor as a parameter, plus trainer and generator will receive the same model_spec / same config tree when calling apply_override.

• For DeepEP, we set cudagraph=false in model_spec (align with trainer), and override generator to be "True" (always fixed).
• For hybridEP, say if we want capacity_factor = None in trainer, but want capacity_factor to be configurable in generator, it's impossible to do that with current override + unified model.

Basically there's only one capacity_factor field in shared model_spec config, which is not expressive enough for both trainer and generator, and we have to fix one side using prior knowledge (eg, set trainer always to be None), and using the only capacity_factor field for another side

[wwwjn]

Solution:

• Make override configurable: List[string | tuple(string, {"", ""})]

  • Probably check nested overrided config

• Converter: quantization, LoRA, batchInvariantFlex