Tensorize dynamic inference mixed sampling

[tdene]

What does this PR do ?

#1927 implemented mixed sampling in an unoptimized fashion. This led to a significant regression in the latency of the forward step.

This PR tensorizes the associated bookkeeping, eliminating the related slowdown.

A sketch of the design follows:

• SamplingParams now uses unsafe_hash=True. A future PR should make it immutable and use frozen=True.
• The dynamic context now has an extra request-indexed Tensor, which keeps track of the SamplingParams hash associated with each request in the context. Every time a request is added to the dynamic context, its SamplingParams are hashed and associated with the new request.
• When the engine steps, there is no longer a need to use a slow nested Python loop to break up requests into SamplingParams buckets. Instead, this now uses tensorized logic inside _dynamic_step_sample_bookkeeping.
• An optional static_sampling parameter can be used to bypass even this tensorized logic.
• The sampling code from sample_from_logits has been factored out into is own sub-method, but neither it nor any of the sample_from_logits code has been modified at all.

Contribution process

flowchart LR
    A[Pre-checks] --> B[PR Tests]
    subgraph Code Review/Approval
        C1[Expert Review] --> C2[Final Review]
    end
    B --> C1
    C2 --> D[Merge]

Loading

Pre-checks

• I want this PR in a versioned release and have added the appropriate Milestone (e.g., Core 0.8)
• I have added relevant unit tests
• I have added relevant functional tests
• I have added proper typing to my code Typing guidelines
• I have added relevant documentation
• I have run the autoformatter.sh on my PR

Code review

The following process is enforced via the CODEOWNERS file for changes into megatron/core. For changes outside of megatron/core, it is up to the PR author whether or not to tag the Final Reviewer team.

For MRs into `main` branch

(Step 1): Add PR label Expert Review

(Step 2): Collect the expert reviewers reviews

1. Attach the Expert Review label when your PR is ready for review.
2. GitHub auto-assigns expert reviewers based on your changes. They will get notified and pick up your PR soon.

⚠️ Only proceed to the next step once all reviewers have approved, merge-conflict are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

(Step 3): Final Review

1. Add Final Review label
2. GitHub auto-assigns final reviewers based on your changes. They will get notified and pick up your PR soon.

(Optional Step 4): Cherry-pick into release branch

If this PR also needs to be merged into core_r* release branches, after this PR has been merged, select Cherry-pick to open a new PR into the release branch.

For MRs into `dev` branch

The proposed review process for `dev` branch is under active discussion.

MRs are mergable after one approval by either [email protected] or [email protected].

Merging your PR

Any member of core-adlr and core-nemo will be able to merge your PR.

[tdene]

All tests pass.

Merge conflict is due to branches temporarily being reverted from main, and it will soon be resolved on its own.