Context Parallelism

[kd303]

Is Context Parallelism supported in Nanotron? It seems that most of the frameworks are supporting Context Parallelism for LongContext training & it is making almost defacto standard (Llama paper, Nvidia Megatron etc.) I understand that SmolLM team has been creating LongContext training reciepies with nontron, however there seems to be no observations on how CP is not a limiting factor in terms of throughput?

[kd303]

I think this is probably my oversight, but can someone pls confirm whether CP is supported?

I can see in the SmolLMv3 example the following snippet:

parallelism:
  context_parallel_size: 4
  dp: 12
  expert_parallel_size: 1
  moe_layer_recompute: false
  pp: 1
  pp_engine: 1f1b

Git Documentation probably is not updated?, Happy to submit a simple pull request. :)

[kiankyars]

The ParallelContext class (in src/nanotron/parallel/context.py) has a context_parallel_size parameter, which is used in the parallelism setup and asserts that TP * CP * DP * PP = WORLD_SIZE, but that may just be for future adoption.