SUT Options defined at test level #897
Conversation
bkorycki
temporarily deployed
to
Scheduled Testing
GitHub Actions
Inactive
bkorycki
temporarily deployed
to
Scheduled Testing
GitHub Actions
Inactive
bkorycki
temporarily deployed
to
Scheduled Testing
GitHub Actions
Inactive
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
|
Conceptually, I like the idea of removing sut options from TestItems and passing them in where needed. |
There was a problem hiding this comment.
Seems good other than the disabled test.
As to how to make it so people are less likely to accidentally break things by using the wrong options, would it make sense for the TestItem to have a pointer to the Test? Then anything that needs them, like a runner, could either do test_item.test.sut_options() or just test_item.sut_options() which has a default implementation that passes the request along to the Test.
src/modelgauge/base_test.py
Outdated
| @@ -50,6 +51,12 @@ def get_annotators(cls) -> List[str]: | |||
| """ | |||
| pass | |||
|
|
|||
| @classmethod | |||
There was a problem hiding this comment.
Why is this a class method? It seems to me that it could happen at the instance level, although given our small number of tests, I suppose it's hard to say.
There was a problem hiding this comment.
I might also not make it an abstract method, come to think of it. Having a default that uses a default SUTOptions seems fine by me.
There was a problem hiding this comment.
I thought it might be useful for it to be a class method so that we could get information without having to instantiate a test object, which might be expensive. But it definitely doesn't have to be a class method.
And yeah you're right about getting rid of the abstract portion. I'll implement add a default.
There was a problem hiding this comment.
Oh, yeah, if that's something we're doing, that would be a good reason. But if we don't need it now, I'd say stick with an instance method and we can always rejigger later.
There was a problem hiding this comment.
Okay I think I did it.
| @classmethod | ||
| def sut_options(cls) -> SUTOptions: | ||
| """Returns the SUT options that are supplied in each test item.""" | ||
| return SUTOptions( |
There was a problem hiding this comment.
This makes a new one each time, which is fine if it's not called often, but "once per test item" is for us a pretty frequent call.
There was a problem hiding this comment.
Yeah that's true. I could make it a class property instead.
bkorycki
temporarily deployed
to
Scheduled Testing
GitHub Actions
Inactive
bkorycki
temporarily deployed
to
Scheduled Testing
GitHub Actions
Inactive
bkorycki
temporarily deployed
to
Scheduled Testing
GitHub Actions
Inactive
bkorycki
had a problem deploying
to
Scheduled Testing
GitHub Actions
Failure
bkorycki
had a problem deploying
to
Scheduled Testing
GitHub Actions
Error
bkorycki
had a problem deploying
to
Scheduled Testing
GitHub Actions
Failure
|
@wpietri @rogthefrog I ended up going with Roger's suggestion of removing sut_options from TestItem. I tried doing William's suggestion but you can't really point to non-pydantic models in a pydantic model. |
bkorycki
temporarily deployed
to
Scheduled Testing
GitHub Actions
Inactive
bkorycki
temporarily deployed
to
Scheduled Testing
GitHub Actions
Inactive
This is part of the effort to chisel away the need for the test wrapper class. More specifically, this PR makes it easy to access the SUT options used by a test object.
However, there is no guarantee that the SUT options returned by
test.sut_options()are actually used in all of the test items. This is because the tests are responsible to assembling their test own test items (i.e. bundling the prompts + contexts + sut options).I see 3 options here:
sut_options()is accurate (i.e. what this PR does). I don't like this weak link between the method and what it actually represents, but it is the easiest approach.sut.translate_text_prompt(prompt)becomessut.translate_text_prompt(prompt, sut_options).