[WIP] vitest evals #1232
[WIP] vitest evals #1232
Conversation
|
|
||
| // indicate the project name the tests will be sent to | ||
| const bt = wrapVitest( | ||
| { test, expect, describe, afterAll }, |
There was a problem hiding this comment.
I'd recommend for users to:
import * as vitest from 'vitest';
const { describe, expect, test, ... } = wrapVitest(vitest);
simpler, less prone to error, and future proof if we add more functions to our support
| metadata: { category: "math" }, | ||
| tags: ["arithmetic"], | ||
| }, | ||
| async ({ input, expected }) => { |
There was a problem hiding this comment.
til.. so the context is re-inserted as an argument
| afterAll: vitestMethods.afterAll || (() => {}), | ||
| beforeEach: vitestMethods.beforeEach, | ||
| afterEach: vitestMethods.afterEach, | ||
| logOutputs: (outputs: Record<string, unknown>) => { |
There was a problem hiding this comment.
didn't expect this method 🤔
| score: number; | ||
| metadata?: Record<string, unknown>; | ||
| }) => { | ||
| const span = currentSpan(); |
There was a problem hiding this comment.
what if instead of these methods we just asked folks to use currentSpan?
| @@ -0,0 +1,46 @@ | |||
| import tsconfigPaths from "vite-tsconfig-paths"; | |||
There was a problem hiding this comment.
didn't expect to see this file in here 🤔
| return originalDescribe(suiteName, () => { | ||
| // Lazily initialize experiment context on first access | ||
| let context: ExperimentContext | null = null; | ||
| const getOrCreateContext = (): ExperimentContext => { |
There was a problem hiding this comment.
like we want to extract and use the same getOrCreate inside of test() calls
| "Braintrust: vitestMethods.describe is required. Please pass in the describe function from vitest.", | ||
| ); | ||
| } | ||
| if (!vitestMethods.expect) { |
There was a problem hiding this comment.
i didn't see a wrapExpect in wrapper.ts. I wonder if each expect is a scorer?
| }); | ||
|
|
||
| // If test function returns a value, log it as output | ||
| if (testResult !== undefined) { |
There was a problem hiding this comment.
i think if you traced(maybeFn || configOrFn, ...) you may have gotten this automatically?
| scores: { | ||
| pass: 0, | ||
| }, | ||
| metadata: { |
There was a problem hiding this comment.
you should probably just throw again. the traced() call should handle the error.
| datasetExamples: Map<string, string>; // test name -> example id | ||
| } | ||
|
|
||
| // Global context holder (one per describe block) |
There was a problem hiding this comment.
what happens with concurrent calls i.e.
it.concurrent(
describe(..., () => {
})
);
it.concurrent(
describe(..., () => {
})
);
did you give currentExperiment a try?
No description provided.