|
1 | | -# Braintrust Java SDK |
| 1 | +# Braintrust Java Tracing & Eval SDK |
2 | 2 |
|
3 | | -An OpenTelemetry-based Braintrust SDK for Java 17 and up. Contains: |
| 3 | +[](https://javadoc.io/doc/dev.braintrust/braintrust-sdk-java) |
| 4 | +[](https://github.com/braintrustdata/braintrust-sdk-java/actions/workflows/ci.yml) |
4 | 5 |
|
5 | | -- Instrumentation for major AI vendors (OpenAI, Anthropic, etc) |
6 | | -- LLM Eval framework for sending Experiments to Braintrust |
7 | | -- Utils for sending Open Telemetry data to Braintrust |
| 6 | +## Overview |
8 | 7 |
|
9 | | -If you're simply looking to call the Braintrust API with Java code, see [https://github.com/braintrustdata/braintrust-java](https://github.com/braintrustdata/braintrust-java) |
| 8 | +This library provides tools for **evaluating** and **tracing** AI applications in [Braintrust](https://www.braintrust.dev). Use it to: |
10 | 9 |
|
11 | | -This SDK is currently is in BETA status and APIs may change |
| 10 | +- **Evaluate** your AI models with custom test cases and scoring functions |
| 11 | +- **Trace** LLM calls and monitor AI application performance with OpenTelemetry |
| 12 | +- **Integrate** seamlessly with OpenAI, Anthropic, and other LLM providers |
12 | 13 |
|
13 | | -<!-- # Using the SDK in your code --> |
| 14 | +This SDK is currently in BETA status and APIs may change. |
14 | 15 |
|
15 | | -<!-- *NOTE: The SDK has not published a release to maven yet. This section will not work until our first release is published* --> |
| 16 | +## See Also |
16 | 17 |
|
17 | | -<!-- build.gradle --> |
18 | | -<!-- ```gradle --> |
19 | | -<!-- dependencies { --> |
20 | | -<!-- implementation 'dev.braintrust:sdk:0.0.1' --> |
21 | | -<!-- } --> |
22 | | -<!-- ``` --> |
| 18 | +If you're looking to call the Braintrust REST API with Java code, see the [Braintrust API Client](https://github.com/braintrustdata/braintrust-java) |
23 | 19 |
|
24 | | -# Examples |
| 20 | +## Quick Start |
25 | 21 |
|
26 | | -All examples can be found here: [./examples/src/main/java/dev/braintrust/examples](./examples/src/main/java/dev/braintrust/examples) |
| 22 | +Add the SDK to your package manager. Latest version and full instructions can be found in [Maven Central](https://central.sonatype.com/artifact/dev.braintrust/braintrust-sdk-java/versions) |
27 | 23 |
|
28 | | -To run the examples from the command line you need: |
29 | | -- A Braintrust account and a valid BRAINTRUST_API_KEY |
30 | | -- Java 17 (or greater): |
31 | | - - macOS: `brew install openjdk@17` |
32 | | - - Ubuntu: `sudo apt install openjdk-17-jdk` |
33 | | -- (Optional to run the oai example) a valid OPENAI_API_KEY |
34 | | - |
35 | | -Then, from the repo root: |
| 24 | +build.gradle example: |
| 25 | +```gradle |
| 26 | +dependencies { |
| 27 | + implementation 'dev.braintrust:sdk:<version-goes-here>' |
| 28 | +} |
36 | 29 | ``` |
37 | | -./gradlew :examples:runSimpleOpenTelemetry |
| 30 | + |
| 31 | +### Evals |
| 32 | + |
| 33 | +```java |
| 34 | +var config = BraintrustConfig.fromEnvironment(); |
| 35 | +var openTelemetry = BraintrustTracing.of(config, true); |
| 36 | +var openAIClient = BraintrustOpenAI.wrapOpenAI(openTelemetry, OpenAIOkHttpClient.fromEnv()); |
| 37 | + |
| 38 | +Function<String, String> getFoodType = |
| 39 | + (String food) -> { |
| 40 | + var request = |
| 41 | + ChatCompletionCreateParams.builder() |
| 42 | + .model(ChatModel.GPT_4O_MINI) |
| 43 | + .addSystemMessage("Return a one word answer") |
| 44 | + .addUserMessage("What kind of food is " + food + "?") |
| 45 | + .maxTokens(50L) |
| 46 | + .temperature(0.0) |
| 47 | + .build(); |
| 48 | + var response = openAIClient.chat().completions().create(request); |
| 49 | + return response.choices().get(0).message().content().orElse("").toLowerCase(); |
| 50 | + }; |
| 51 | + |
| 52 | +var eval = |
| 53 | + Eval.<String, String>builder() |
| 54 | + .name("java-eval-x-" + System.currentTimeMillis()) |
| 55 | + .tracer(BraintrustTracing.getTracer(openTelemetry)) |
| 56 | + .config(config) |
| 57 | + .cases( |
| 58 | + EvalCase.of("asparagus", "vegetable"), |
| 59 | + EvalCase.of("banana", "fruit")) |
| 60 | + .task(getFoodType) |
| 61 | + .scorers( |
| 62 | + Scorer.of( |
| 63 | + "fruit_scorer", |
| 64 | + result -> "fruit".equals(result) ? 1.0 : 0.0), |
| 65 | + Scorer.of( |
| 66 | + "vegetable_scorer", |
| 67 | + result -> "vegetable".equals(result) ? 1.0 : 0.0)) |
| 68 | + .build(); |
| 69 | +var result = eval.run(); |
| 70 | +System.out.println("\n\n" + result.createReportString()); |
38 | 71 | ``` |
39 | 72 |
|
40 | | -If you wish to see all examples run: |
| 73 | +### OpenAI Tracing |
| 74 | + |
| 75 | +```java |
| 76 | +var braintrustConfig = BraintrustConfig.fromEnvironment(); |
| 77 | +var openTelemetry = BraintrustTracing.of(braintrustConfig, true); |
| 78 | +OpenAIClient openAIClient = BraintrustOpenAI.wrapOpenAI(openTelemetry, OpenAIOkHttpClient.fromEnv()); |
| 79 | + |
| 80 | +var request = |
| 81 | + ChatCompletionCreateParams.builder() |
| 82 | + .model(ChatModel.GPT_4O_MINI) |
| 83 | + .addSystemMessage("You are a helpful assistant") |
| 84 | + .addUserMessage("What is the capital of France?") |
| 85 | + .temperature(0.0) |
| 86 | + .build(); |
| 87 | +# openai calls will be automatically traced and reported to braintrust |
| 88 | +var response = openAIClient.chat().completions().create(request); |
41 | 89 | ``` |
| 90 | + |
| 91 | +## Running Examples |
| 92 | + |
| 93 | +Example source code can be found [here](./examples/src/main/java/dev/braintrust/examples) |
| 94 | + |
| 95 | +```bash |
| 96 | +export BRAINTRUST_API_KEY="your-braintrust-api-key" |
| 97 | +export OPENAI_API_KEY="your-oai-api-key" # to run oai examples |
| 98 | +export ANTHROPIC_API_KEY="your-anthropic-api-key" # to run anthropic examples |
| 99 | +# install java 17 or later |
| 100 | +brew install openjdk@17 # macOS |
| 101 | +sudo apt install openjdk-17-jdk # ubuntu |
| 102 | +# to run a specific example |
| 103 | +./gradlew :examples:runSimpleOpenTelemetry |
| 104 | +# to see all examples |
42 | 105 | ./gradlew :examples:tasks --group="Braintrust SDK Examples" |
43 | 106 | ``` |
44 | 107 |
|
45 | | -If you wish to hack around with the examples you can modify source then re-run the gradle task. |
46 | | - |
47 | 108 | ## Logging |
48 | 109 |
|
49 | 110 | The SDK uses a standard slf4j logger and will use the default log level (or not log at all if slf4j is not installed). |
|
0 commit comments