Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cli): Log and replay oracle transcript #7417

Open
wants to merge 86 commits into
base: master
Choose a base branch
from

Conversation

aakoshh
Copy link
Contributor

@aakoshh aakoshh commented Feb 18, 2025

Description

Problem*

Resolves #7379 and #7382

Summary*

  • Added NARGO_TEST_FOREIGN_CALL_LOG to nargo test which can be used to log the foreign calls made by a test either to stdout or a file. It only makes sense with a single test. The format of the log is JSON Line with a call and response as it appears in the JSON-RPC.
  • Added a LoggingForeignCallExecutor that prints foreign calls to a transcript log.
  • Added a ReplayForeignCallExecutor that deserialises the transcript, and implements the handler by popping off the next call, comparing the logged function name and inputs, rejecting any unexpected call to avoid confusion, then returns the logged response.
  • Added --oracle-file to noir-execute and nargo execute to be able to point at the transcript
  • Changed noir-execute to use the ReplayForeignCallExecutor as the final layer in the foreign call executor stack, if the file is present.
  • Added examples/oracle_transcript based on test_programs/noir_test_success/mock_oracle, which calls nargo test to capture a transcript and verify the expected input/output, then nargo execute with the transcript file to verify the expected return value in Prover.toml. The oracle file is checked in for reference.
  • Repeats the same call with noir-execute to demonstrate how it's used.
  • Build the noir-execute artifact on CI to be able to use it in the test script.

Additional Context

Here's a link to show the new example passing on CI: https://github.com/noir-lang/noir/actions/runs/13394747019/job/37411091682

The format in this PR, or something similar, will be the one used in AztecProtocol/aztec-packages#11855

Documentation*

Check one:

  • No documentation needed.
  • Documentation included in this PR.
  • [For Experimental Features] Documentation to be submitted in a separate PR.

PR Checklist*

  • I have tested the changes locally.
  • I have formatted the changes with Prettier and/or cargo fmt on default settings.

@asterite
Copy link
Collaborator

Added NARGO_TEST_FOREIGN_CALL_LOG to nargo test which can be used to log the foreign calls made by a test either to stdout or a file. It only makes sense with a single test.

I'm about to start reviewing this PR, but why does it only make sense with a single test? I know oracle calls from multiple tests might lead to different results, but I was wondering if we could key them under the test name.

Copy link
Collaborator

@asterite asterite left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! I guess, if it's possible, we could support running multiple tests in a future PR (is it an error right now if you run it with multiple tests?)

@aakoshh
Copy link
Contributor Author

aakoshh commented Feb 19, 2025

I'm about to start reviewing this PR, but why does it only make sense with a single test?

Only because I didn't spend any effort to delineate them:

  • if we set NARGO_TEST_FOREIGN_CALL_LOG=stdout then it will print things as it goes
  • if we set NARGO_TEST_FOREIGN_CALL_LOG=path/to/file.jsonl then the last test will overwrite the previous ones
  • there is no option to log to a string and print it to the stdout as one unit like you are doing it with the test result reporter for println collected during execution

@aakoshh
Copy link
Contributor Author

aakoshh commented Feb 19, 2025

I didn't want to change the signature of nargo::ops::run_test to add another PrintOutput for these, and also didn't want to have to copy paste from the terminal into a file, but if we think this is useful, or should be displayed along with the println output of the test, I'm happy to revisit. Currently it was just an idea for getting something to work with.

Base automatically changed from 7380-nargo-exec-fwd to master February 28, 2025 11:45
@aakoshh aakoshh requested a review from TomAFrench February 28, 2025 11:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add an example to test the artifact CLI on CI Execute artifact with oracle transcript
3 participants