Refactor active learning

[sgreenbury]

Closes #345.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 75.00000% with 43 lines in your changes missing coverage. Please review.

Project coverage is 90.17%. Comparing base (47be238) to head (8924390).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
autoemulate/experimental/data/validation.py	55.55%	28 Missing ⚠️
autoemulate/experimental/learners/stream.py	77.77%	6 Missing ⚠️
tests/experimental/test_learners.py	91.48%	4 Missing ⚠️
autoemulate/experimental/learners/base.py	85.00%	3 Missing ⚠️
autoemulate/experimental/learners/membership.py	0.00%	1 Missing ⚠️
autoemulate/experimental/learners/pool.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #411      +/-   ##
==========================================
- Coverage   90.53%   90.17%   -0.37%     
==========================================
  Files          96       97       +1     
  Lines        5983     5994      +11     
==========================================
- Hits         5417     5405      -12     
- Misses        566      589      +23     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
autoemulate/emulators
gaussian_process.py
gaussian_process_mt.py
autoemulate/experimental
types.py
autoemulate/experimental/data
utils.py
validation.py					15, 20, 42-46, 69, 71, 97-98, 123-130, 155, 158-160, 186, 189-191, 215-216, 219-221
autoemulate/experimental/emulators
base.py
autoemulate/experimental/emulators/gaussian_process
exact.py
autoemulate/experimental/learners
__init__.py
base.py					183, 188-192
membership.py					4
pool.py					4
stream.py					521-525, 528, 556-561, 564
tests
test_compare.py
tests/experimental
test_learners.py					47-68
Project Total

This report was generated by python-coverage-comment-action

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[sgreenbury]

Following c908238 and a few small changes, the notebook has been re-run now. There remain a couple of issues I have seen so far:

• For sin simulator, the Adaptive_D_optimal does not appear to converge with the other results as expected
• For the projectile simulator:
  • logdet does not decrease as expected
  • Adaptive_D_optimal does not run successfully

Since the change has been to the GP class, I think the explanation will be differences there. These might include:

• normalizing y (initial test of previous version without normalization might explain some of the difference)
• callbacks (early stopping and LR scheduler are present in non-experimental)

[radka-j]

We still have a GP implementation in the notebook in cell 7 (that inherits from GaussianProcessExact). We should remove this and where we instantiate the emulator we should call GaussianProcessExact() instead of GP().

[sgreenbury]

Adding this comparison we discussed:

The left-hand plot shows the unnormalized case where the increasing log determinant is due to the GP learning the loc/scale of the output while the right-hand plot the scale is loc=0 scale=1 so that the scale of the uncertainty captured by the log det is captured from the start.

[sgreenbury]

Thanks for the review and discussion @radka-j and @cisprague - merging this now as discussed.