Skip to content

feat(static tests) : do not compile test fillers source code while loading the models #1439

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 12 commits into from
Apr 23, 2025
Merged

feat(static tests) : do not compile test fillers source code while loading the models #1439

merged 12 commits into from
Apr 23, 2025

Conversation

winsvega
Copy link
Contributor

@winsvega winsvega commented Apr 11, 2025 •
edited by marioevz
Loading

@marioevz please don't merge there are bullet points to complete:

  • When reading Filler.json into pydantic do not compile test source right away. this speeds up test generation
  • Support lllc container docker image, so no need to have lllc compiled locally. can import from docker hub: winsvega/lllc:latest
  • After all tests generated or on exception fail stop, need to call stop_lllc_containers() function

- [x] Make sure exceptions are working correctly after mapper refactoring (too many things to fix) #1440

the test generation is supported with lllc from docker hub.
https://hub.docker.com/r/winsvega/lllc

it works fast using docker exec. all we need is to pull a docker image in our CI

@winsvega winsvega force-pushed the fill_static_tests branch 2 times, most recently from 65e4eee to 8a295d3 Compare April 11, 2025 13:34
@winsvega winsvega requested a review from marioevz April 11, 2025 13:37
@winsvega winsvega mentioned this pull request Apr 11, 2025
Merged
8 tasks
@winsvega winsvega changed the title do not compile test fillers source code while loading the models feat(static tests) : do not compile test fillers source code while loading the models Apr 11, 2025
@spencer-tb spencer-tb added type:feat type: Feature scope:fill Scope: fill command labels Apr 14, 2025
@winsvega
Copy link
Contributor Author

restored this logic
a092ecb

@winsvega
Copy link
Contributor Author

winsvega commented Apr 14, 2025 •
edited
Loading

@marioevz

execution-spec-tests/src/ethereum_test_specs/static_state/common/docker.py
def stop_lllc_containers():

needs to be called after filling of the fixtures has finished. or in case of exception or keyboard interrupt and we are exiting.
where this part of logic is defined in pytest plugin of static test filler?

marioevz
marioevz reviewed Apr 14, 2025
View reviewed changes
Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, just a few comments, let me know what you think about the docker alternative.

winsvega and others added 10 commits April 23, 2025 18:40
@marioevz marioevz force-pushed the fill_static_tests branch from 0123bd6 to fb9dc13 Compare April 23, 2025 21:56
marioevz
marioevz approved these changes Apr 23, 2025
View reviewed changes
Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I verified and all legacy tests are filling locally, I think we can go ahead and merge this plus #1442.

@marioevz
Copy link
Member

I marked the last checkbox in the description because we are now using locally built lllc, which doesn't require for us to stop any container.

@marioevz marioevz mentioned this pull request Apr 23, 2025
Merged
4 tasks
@marioevz marioevz merged commit b65fc7c into main Apr 23, 2025
21 checks passed
@marioevz marioevz deleted the fill_static_tests branch April 23, 2025 22:23
pacrob pushed a commit to pacrob/execution-spec-tests that referenced this pull request May 5, 2025
@winsvega @marioevz @pacrob
feat(plugins/filler/static): do not compile test fillers source code …
1a43361 
…while loading the models (ethereum#1439)

* do not compile test fillers source code while loading the models

* support docker lllc image

* export filled static state tests into test suite folder.
 Each legacy test filler has only 1 test per file if it's a !state test!
 So no need to create directory Add11/add11.json it can be plain add11.json

* Apply suggestions from code review

* fix(specs): Prefer local `lllc` binary

* feat(filler): Allow arbitrary markers for static tests

* feat(specs): Read pytest markers from static yml/json files

* fix(filler): Propagate parametrize marks to generated test

* fix(specs/static_state): Parametrization to add `exception_test` marker

* fix(plugins/filler): Use sorted intersection fork set

* fix: tox

* Update src/ethereum_test_specs/static_state/common/common.py

---------

Co-authored-by: Mario Vega <[email protected]>
felix314159 pushed a commit to felix314159/execution-spec-tests that referenced this pull request May 16, 2025
@winsvega @marioevz @felix314159
feat(plugins/filler/static): do not compile test fillers source code …
3ac1044 
…while loading the models (ethereum#1439)

* do not compile test fillers source code while loading the models

* support docker lllc image

* export filled static state tests into test suite folder.
 Each legacy test filler has only 1 test per file if it's a !state test!
 So no need to create directory Add11/add11.json it can be plain add11.json

* Apply suggestions from code review

* fix(specs): Prefer local `lllc` binary

* feat(filler): Allow arbitrary markers for static tests

* feat(specs): Read pytest markers from static yml/json files

* fix(filler): Propagate parametrize marks to generated test

* fix(specs/static_state): Parametrization to add `exception_test` marker

* fix(plugins/filler): Use sorted intersection fork set

* fix: tox

* Update src/ethereum_test_specs/static_state/common/common.py

---------

Co-authored-by: Mario Vega <[email protected]>
codeofcarson pushed a commit to codeofcarson/execution-spec-tests that referenced this pull request Jul 1, 2025
@winsvega @marioevz @Carsons-Eels
feat(plugins/filler/static): do not compile test fillers source code …
46efb41 
…while loading the models (ethereum#1439)

* do not compile test fillers source code while loading the models

* support docker lllc image

* export filled static state tests into test suite folder.
 Each legacy test filler has only 1 test per file if it's a !state test!
 So no need to create directory Add11/add11.json it can be plain add11.json

* Apply suggestions from code review

* fix(specs): Prefer local `lllc` binary

* feat(filler): Allow arbitrary markers for static tests

* feat(specs): Read pytest markers from static yml/json files

* fix(filler): Propagate parametrize marks to generated test

* fix(specs/static_state): Parametrization to add `exception_test` marker

* fix(plugins/filler): Use sorted intersection fork set

* fix: tox

* Update src/ethereum_test_specs/static_state/common/common.py

---------

Co-authored-by: Mario Vega <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@marioevz marioevz marioevz approved these changes

Assignees
No one assigned
Labels
scope:fill Scope: fill command type:feat type: Feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@winsvega @marioevz @spencer-tb