feat: add generation stats to TextGeneration trait

[erfanium]

tabby/crates/tabby-inference/src/lib.rs

Lines 23 to 31 in 99d49a9

	#[async_trait]
	pub trait TextGeneration: Sync + Send {
	async fn generate(&self, prompt: &str, options: TextGenerationOptions) -> String;
	async fn generate_stream(
	&self,
	prompt: &str,
	options: TextGenerationOptions,
	) -> BoxStream<String>;
	}

Current TextGeneration trait is simple, but it doesn't tell the statics we need for monitoring and optimizing the inference server.

for example, input_token_length and output_token_length stats are really important to measure the inference server throughput.

My initial idea would be something like this:

pub struct TextGenerationStat {
    pub prompt_tokens_length: u32,
    pub output_tokens_length: Option<u32>, // not available in stream responses
    pub time: Option<u32>, // not available in stream responses
}

pub struct TextGenerationResponse {
    pub text: String,
    pub stat: Option<TextGenerationStat>,
}

pub struct TextGenerationStreamResponse {
    pub stream: BoxStream<String>,
    pub stat: Option<TextGenerationStat>,
}

pub trait TextGeneration: Sync + Send {
    async fn generate(
        &self,
        prompt: &str,
        options: TextGenerationOptions,
    ) -> TextGenerationResponse;
    async fn generate_stream(
        &self,
        prompt: &str,
        options: TextGenerationOptions,
    ) -> TextGenerationStreamResponse;
}

[wsxiaoys]

I think we can already compute stats for input string length / output string length / time. Should be sufficient for tabby's optimization purpose?

[erfanium]

@wsxiaoys yeah but as you know, input string length != input token length. metrics based on string length is not standard.

[wsxiaoys]

Yes - that's why I emphasis that for tabby's optimization purpose :).

[sundaraa-deshaw]

Is there a way (besides tabby server logs) to get a sense of inference performance (i.e. tokens/s) as part of the response for benchmarking? Since I am new to this space, can you please suggest other better alternatives to benchmark the server's performance?