CAMEL-24559: Extend GenAI observability to OpenAI embeddings, moderation, and responses - #26106
Conversation
|
Links CAMEL-24559 — first follow-up slice for extending GenAI observability beyond chat/streaming. Review status: Bugbot (no bugs) and Grok review feedback addressed in commit AI-generated comment on behalf of @atiaomar1978-hub |
|
Links CAMEL-24559 — first follow-up slice. Bugbot + Grok review feedback addressed in 38b10e2. |
|
🌟 Thank you for your contribution to the Apache Camel project! 🌟 🐫 Apache Camel Committers, please review the following items:
|
|
🧪 CI tested the following changed modules:
🔬 Scalpel shadow comparison — Scalpel: 73 tested, 25 compile-only — current: 71 all testedMaveniverse Scalpel detected 98 affected modules (current approach: 71).
|
gnodet
left a comment
gnodet
left a comment
There was a problem hiding this comment.
Review summary: Solid and well-structured extension of GenAI observability to three additional OpenAI producers. The pattern follows the established createChatCompletion in OpenAIProducer closely. Two issues worth addressing before merge.
This review was generated by an automated reviewer (Hermès) on behalf of @gnodet.
| calculateSimilarityIfRequested(exchange, embeddings); | ||
| } | ||
|
|
||
| private static Integer toTokenCount(long tokens) { |
There was a problem hiding this comment.
Math.toIntExact(tokens) will throw ArithmeticException if the token count exceeds Integer.MAX_VALUE. Since this is called inside the try block that catches Exception, a hypothetical overflow would abort the entire exchange even though the API call itself succeeded.
More importantly, this narrowing is unnecessary. GenAiUsage accepts Long directly — the existing createChatCompletion in OpenAIProducer passes usage.promptTokens() (a long) straight to GenAiUsage.of(Long, Long, ...) without any conversion:
| private static Integer toTokenCount(long tokens) { | |
| private static Long toTokenCount(long tokens) { | |
| return tokens; | |
| } |
Or just inline the long values directly and drop toTokenCount entirely, matching the OpenAIProducer pattern.
There was a problem hiding this comment.
Fixed in d1a3fc2 — removed toTokenCount and pass usage.promptTokens() directly to GenAiUsage.of(Long, ...), matching OpenAIProducer.createChatCompletion.
| finishReason, | ||
| response.model().toString())), | ||
| () -> observation.recordSuccess(GenAiUsage.of(null, null, finishReason, response.model().toString()))); | ||
| } |
There was a problem hiding this comment.
Same issue as in OpenAIEmbeddingsProducer — Math.toIntExact is both unnecessary (the Long overload of GenAiUsage.of exists) and risky (throws ArithmeticException on overflow, which the catch block would treat as a failed operation).
| } | |
| private static Long toTokenCount(long tokens) { | |
| return tokens; | |
| } |
There was a problem hiding this comment.
Fixed in d1a3fc2 — removed toTokenCount; usage.inputTokens() and usage.outputTokens() are passed directly as long values.
| .system("openai") | ||
| .requestModel(model) | ||
| .componentScheme("openai") | ||
| .build(); |
There was a problem hiding this comment.
createResponse and createStructuredResponse are nearly identical — same observation context, same error handling, same finally. The only difference is the SDK call. Consider extracting a common helper, e.g.:
private Response observedCall(Exchange exchange, String model,
ThrowingSupplier<Response> call) throws Exception {
GenAiObservationContext ctx = GenAiObservationContext.builder()
.operationName(GenAiOperationName.CHAT)
.system("openai").requestModel(model)
.componentScheme("openai").build();
GenAiObservation observation = GenAiObservability.start(exchange, ctx);
try {
Response response = call.get();
recordResponseSuccess(observation, response);
return response;
} catch (Exception e) {
GenAiErrorSupport.apply(exchange, e);
observation.recordError(e);
throw e;
} finally {
observation.close();
}
}Then createResponse becomes observedCall(exchange, model, () -> getEndpoint().getClient().responses().create(params)) and createStructuredResponse becomes observedCall(exchange, model, () -> getEndpoint().getClient().responses().create(structuredParams).rawResponse()).
Not blocking, but reduces ~40 lines of duplication to ~2.
There was a problem hiding this comment.
Fixed in d1a3fc2 — extracted shared observedCall(exchange, model, ThrowingSupplier<Response, Exception>) helper used by both code paths.
| observation.recordError(e); | ||
| throw e; | ||
| } finally { | ||
| observation.close(); |
There was a problem hiding this comment.
The comment that was here ("this operation is used to gate untrusted content, so a missing verdict must fail the exchange...") explained a security invariant — why a mismatched result count throws rather than silently proceeding. Worth keeping; it's not redundant with the observability changes.
There was a problem hiding this comment.
Fixed in d1a3fc2 — restored both security comments (mismatch must fail the exchange; store full response only after validation).
…n, and responses Extend camel-openai producers beyond chat-completion with OpenTelemetry spans and Micrometer metrics using the existing GenAiObservability API. - Instrument OpenAIEmbeddingsProducer, OpenAIModerationProducer, and OpenAIResponsesProducer with try/recordSuccess/recordError/close - Add GenAiOperationName.MODERATION for content-policy operations - Add OpenAIEmbeddingsObservabilityTest, OpenAIModerationObservabilityTest, OpenAIResponsesObservabilityTest, and shared test support - Document OpenAI operation coverage in ai-observability.adoc and 4.23 upgrade guide Co-authored-by: Cursor Agent <noreply@cursor.com>
- Scope GenAI spans to SDK calls only (embeddings/moderation) - Map openai:responses to gen_ai.operation.name=chat per OTel OpenAI usage - Clarify MODERATION as Camel extension in enum javadoc and docs - Strengthen tests: body/header assertions, exact token tags, override properties Co-authored-by: Cursor Agent <noreply@cursor.com>
- Pass OpenAI token counts as Long without Math.toIntExact (includes CAMEL-24560 GenAiUsage Long token fields cherry-picked from follow-up) - Extract observedCall helper in OpenAIResponsesProducer to deduplicate observation boilerplate between createResponse and createStructuredResponse - Restore moderation security comments explaining why mismatched result counts must fail the exchange Co-authored-by: Cursor Agent <noreply@cursor.com>
067e3b1 to
d1a3fc2
Compare
|
This comment was generated by an AI agent on behalf of @atiaomar1978-hub. Review feedback from @gnodet has been addressed and the branch rebased onto latest
Rebased commits on
Tests run locally after rebase: |
3 participants
Summary
Follow-up to CAMEL-23861 / CAMEL-24559: extend GenAI observability to additional OpenAI producer operations beyond
chat-completion.This PR instruments OpenAI embeddings, moderation, and responses with the existing
GenAiObservabilityAPI.gen_ai.operation.nameopenai:embeddingsembeddingsopenai:moderationmoderation(Camel extension)openai:responseschatopenai:chat-completionchat(unchanged)Changes
OpenAIEmbeddingsProducer,OpenAIModerationProducer,OpenAIResponsesProducerGenAiOperationName.MODERATIONOpenAIEmbeddingsObservabilityTest,OpenAIModerationObservabilityTest,OpenAIResponsesObservabilityTestai-observability.adoc, catalog mirror, 4.23 upgrade guideTest plan
./mvnw -pl components/camel-ai/camel-openai -am test -Dtest=OpenAIEmbeddingsObservabilityTest,OpenAIModerationObservabilityTest,OpenAIResponsesObservabilityTestImages, audio, Spring AI modules, and cloud LLMs remain for future CAMEL-24559 follow-ups.
AI-generated PR description on behalf of @atiaomar1978-hub