[LFX] Enhanced cloud-edge-collaborative-inference-for-llm example

[AryanNanda17]

The example "Cloud-Edge Collaborative Inference for LLM" is well-structured. This PR improves few areas of the example to make it a fully functional quick-start guide with minimal errors.

The changes done does the following:-

1. Included a Resource-Sensitive Router
2. Added api_provider, api_base_url and api_key_env parameters to cloudmodel in test_queryrouting.yaml file.
3. Ease of Setting up the environment
4. Backward Compatibility
5. Updating the Threshold for Random Routing
6. Error handling in this example is improved
7. Correcting device = “cuda” assumption

Note:- This PR is an implementation of point 1 of #185. This PR comes under LFX Spring Term 2025 project "Enhancing Dependency Management and Documentation of ianvs".

Fixes #178

[AryanNanda17]

/kind enhancement

[kubeedge-bot]

@AryanNanda17: The label(s) kind/enhancement cannot be applied, because the repository doesn't have them

Details

In response to this:

/kind enhancement

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[AryanNanda17]

@FuryMartin,

Since groq doesn't support the usage options, I have done an approximation of prompt_tokens and completion_tokens :

                prompt_tokens = len(messages[0]['content'].split())  # Approximate
                completion_tokens = len(text.split())  # Approximate

For more accuracy we can use a tokenizer compatible with the model to estimate token counts accurately. A common approach is to use the tiktoken library for models compatible with OpenAI’s tokenization.

That is:-

self.tokenizer = tiktoken.get_encoding("cl100k_base")
prompt_text = "".join([msg["content"] for msg in messages if msg["content"]])
prompt_tokens = len(self.tokenizer.encode(prompt_text))
completion_tokens = len(self.tokenizer.encode(text))

What would you recommend to go with?

[MooreZheng]

Good to see that all suggestions from other reviewers are properly tackled.

Just some tiny logistics consideration:

1. It aims to build the cloud-edge-collaborative-inference-for-llm example. The purpose can be specified in the title and description to make it clear to community members. We also need to squash the commits into one.
2. The CI test for this PR is not yet past, and might not be wrong for this PR. But we should have the CI pass before merging this PR.
3. It would be appreciated to update the dataset link, as specified below.

[AryanNanda17]

To do:-

• Update the dataset link

[AryanNanda17]

@FuryMartin , could you please share with me the updated dataset link?

[AryanNanda17]

The CI workflow failure is described in #212 and has been corrected in #213.

[FuryMartin]

Nice work

[kubeedge-bot]

@FuryMartin: changing LGTM is restricted to collaborators

Details

In response to this:

Nice work

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[AryanNanda17]

@MooreZheng @FuryMartin @hsj576
I have resolved all the comments. This PR is good to go as well.
Thanks

[MooreZheng]

/lgtm

[hsj576]

/lgtm

[MooreZheng]

/approve

[kubeedge-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: FuryMartin, hsj576, MooreZheng

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [MooreZheng]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment