Environment locking, bug fixes, stability improvements + various other changes.

[peterhessey]

What's Changed

• Correct path for ensemble models by @JonathanTripp in Correct path for ensemble models #472
• Fix for SDK regression bug, minor doc updates by @ant0nsc in Fix for SDK regression bug, minor doc updates #475
• Set l_rate correctly and ensure no overlap between subjects in dataset splits in CovidHierarchicalModel config by @Shruthi42 in Set l_rate correctly and ensure no overlap between subjects in dataset splits in CovidHierarchicalModel config #467
• Update documentation for submodules by @ant0nsc in Update documentation for submodules #481
• Multi-node checkpoint recovery fix by @melanibe in Multi-node checkpoint recovery fix #478
• Check bool parameter is either true or false (after lower) by @JonathanTripp in Check bool parameter is either true or false (after lower) #482
• Better handling of missing seriesId by @JonathanTripp in Better handling of missing seriesId #488
• Minor bug fixes in CovidHierarchicalModel by @Shruthi42 in Minor bug fixes in CovidHierarchicalModel #490
• Better handling of outliers by @JonathanTripp in Better handling of outliers #489
• Fix for stuck test set inference for LightningContainer models by @ant0nsc in Fix for stuck test set inference for LightningContainer models #494
• Regression test coverage for AzureML runs by @ant0nsc in Regression test coverage for AzureML runs #492
• Bug fix for regression test by @ant0nsc in Bug fix for regression test #496
• Remove 2 files that cause PR build to fail non-reproducibly by @ant0nsc in Remove 2 files that cause PR build to fail non-reproducibly #500
• Fix timeouts when downloading multiple checkpoint files by @ant0nsc in Fix timeouts when downloading multiple checkpoint files #498
• Reduce AML snapshot size by skipping test folders by @ant0nsc in Reduce AML snapshot size by skipping test folders #497
• Use best epoch for model comparison by @JonathanTripp in Use best epoch for model comparison #495
• Fix bug in all_azure_dataset_ids and all_dataset_mountpoints by @Shruthi42 in Fix bug in all_azure_dataset_ids and all_dataset_mountpoints #466
• Allow cross validation with 'bring your own' Lightning models (without ensemble building) by @dumbledad in Allow cross validation with 'bring your own' Lightning models (without ensemble building) #483
• Comment out glaucoma job by @JonathanTripp in Comment out glaucoma job #520
• Split validation and test infer config by @JonathanTripp in Split validation and test infer config #502
• Added ability to run segmentation inference module in the test data without or partial ground truth files. by @asantamariapang in Added ability to run segmentation inference module in the test data without or partial ground truth files.  #465
• Partial ground truth inference PR (465) missed two things by @dumbledad in Partial ground truth inference PR (465) missed two things #522
• Enable store_dataset_sample by @javier-alvarez in Enable store_dataset_sample #525
• Better Ensemble Child Inference Defaults by @JonathanTripp in Better Ensemble Child Inference Defaults #533
• Update PL to 1.3.8 by @melanibe in Update PL to 1.3.8 #531
• Update README.md by @ktakeda1 in Update README.md #511
• Fixes for mounting and matplotlib problems by @ant0nsc in Fixes for mounting and matplotlib problems #515
• Dropping windows tests, but keeping cred-scan by @dumbledad in Dropping windows tests, but keeping cred-scan #542
• Run inference using checkpoints from registered models by @Shruthi42 in Run inference using checkpoints from registered models #509
• Enable a disabled test by @ant0nsc in Enable a disabled test #536
• Print warning if inference is disabled but comparison requested by @JonathanTripp in Print warning if inference is disabled but comparison requested #537
• Update to fix issues in daily build by @ant0nsc in Update to fix issues in daily build #545
• Update Covid configs by @Shruthi42 in Update Covid configs #526
• Environment and hello_world_model documentation updated by @vale-salvatelli in Environment and hello_world_model documentation updated #546
• Fix: pl_find_unused_parameters was no longer used by @ant0nsc in Fix: pl_find_unused_parameters was no longer used #547
• Minor changes to CovidModel config parameters and updated report by @Shruthi42 in Minor changes to CovidModel config parameters and updated report #554
• Fix incomplete test data module setup in Lightning inference by @dccastro in Fix incomplete test data module setup in Lightning inference #553
• Moving InnerEye's Azure code to hi-ml package by @ant0nsc in Moving InnerEye's Azure code to hi-ml package #548
• Fix learning rate parameter in SSLContainer by @Shruthi42 in Fix learning rate parameter in SSLContainer #557
• Use SSLEncoder when building model in CovidModel config by @Shruthi42 in Use SSLEncoder when building model in CovidModel config #558
• Generalize SSL functionality to work on other datasets by @vale-salvatelli in Generalize SSL functionality to work on other datasets #555
• Adding Active Label Cleaning code by @melanibe in Adding Active Label Cleaning code #559
• Update environment.yml to hi-ml-azure>=0.1 by @dccastro in Update environment.yml to hi-ml-azure>=0.1 #566
• Updating pillow to fix component governance issue by @vale-salvatelli in Updating pillow to fix component governance issue #567
• Adding pre-commit hooks and black formatting by @vale-salvatelli in Adding pre-commit hooks and black formatting #560
• Adjust to new namespaces by @ant0nsc in Adjust to new namespaces #572
• Document Segmentation Model Evaluation by @JonathanTripp in Document Segmentation Model Evaluation #544
• Remove the stdout.txt file by @ant0nsc in Remove the stdout.txt file #576
• Switching batch time loading diagnostics to hi-ml by @ant0nsc in Switching batch time loading diagnostics to hi-ml #577
• Update hello_world_model.md by @maxilse in Update hello_world_model.md #580
• Fix for AzureML environment problem by @ant0nsc in Fix for AzureML environment problem #587
• Enable overriding AzureConfig parameters from a LightningContainer by @dccastro in Enable overriding AzureConfig parameters from a LightningContainer #589
• Fix for invalid path_on_compute problem by @ant0nsc in Fix for invalid path_on_compute problem #593
• Fixing SSL recovery, attempt 2 by @ant0nsc in Fixing SSL recovery, attempt 2 #584
• Use PIL.PngImagePlugin to load pngs by @JonathanTripp in Use PIL.PngImagePlugin to load pngs #588
• Make pytorch run non-deterministically by default, upgrade to AML SDK 1.36 by @ant0nsc in Make pytorch run non-deterministically by default, upgrade to AML SDK 1.36 #594
• Add cudatoolkit=11.1 to environment.yml by @dccastro in Add cudatoolkit=11.1 to environment.yml #596
• Add histopathology module and add hi-ml as submodule by @mebristo in Add histopathology module and add hi-ml as submodule  #603
• Add --pl_deterministic to build training jobs by @dccastro in Add --pl_deterministic to build training jobs #605
• Bug fix: deployed models and training code use different versions of hi-ml by @ant0nsc in Bug fix: deployed models and training code use different versions of hi-ml #606
• Bug fix: When using local folders, datasets are downloaded nevertheless by @ant0nsc in Bug fix: When using local folders, datasets are downloaded nevertheless #604
• Mebristo/move histopathology update by @mebristo in Mebristo/move histopathology update #613
• Downloading checkpoints from AML if not found on disk by @ant0nsc in Downloading checkpoints from AML if not found on disk #614
• Enabling distributed training for SSL online evaluator by @ant0nsc in Enabling distributed training for SSL online evaluator #612
• Add more histopathology configs by @mebristo in Add more histopathology configs #616
• Small cleanups for checkpoint loading by @ant0nsc in Small cleanups for checkpoint loading #615
• Simplify setting callbacks in LightningContainers by @ant0nsc in Simplify setting callbacks in LightningContainers #617
• Adding pipeline cache for conda environment by @ant0nsc in Adding pipeline cache for conda environment #618
• Vsalva/deepmil panda by @vale-salvatelli in Vsalva/deepmil panda #619
• Upgrade to Pytorch Lightning 1.5.5 by @ant0nsc in Upgrade to Pytorch Lightning 1.5.5 #591
• Bug fix: multi-GPU jobs on a VM use wrong folders by @ant0nsc in Bug fix: multi-GPU jobs on a VM use wrong folders #622
• Enable tiling non-PANDA WSI datasets by @dccastro in Enable tiling non-PANDA WSI datasets #621
• Bug fix: SSL on multiple nodes used wrong LR scheduler by @ant0nsc in Bug fix: SSL on multiple nodes used wrong LR scheduler #628
• SSL Online evaluator: save checkpoints without DDP wrapper by @ant0nsc in SSL Online evaluator: save checkpoints without DDP wrapper #623
• Upgrade to PyTorch 1.10 by @ant0nsc in Upgrade to PyTorch 1.10 #585
• Moving *.nii.gz from git lfs to git to simplify the HelloWorld test by @ant0nsc in Moving *.nii.gz from git lfs to git to simplify the HelloWorld test #632
• fixing PandaInnereyeSSLMIL by @vale-salvatelli in fixing PandaInnereyeSSLMIL #625
• Improve recovery of preempted jobs by @ant0nsc in Improve recovery of preempted jobs #633
• Plotting heatmap and thumbnails for test PANDA slides by @harshita-s in Plotting heatmap and thumbnails for test PANDA slides  #634
• Fix length problem in SSL Cosine LR by @ant0nsc in Fix length problem in SSL Cosine LR #638
• Panda SSL tile selection and binary evaluation by @harshita-s in Panda SSL tile selection and binary evaluation #635
• SimCLR recovery test by @maxilse in SimCLR recovery test #643
• Cancel queued AzureML jobs when starting a PR build by @ant0nsc in Cancel queued AzureML jobs when starting a PR build #640
• Enabling DeepSMILE on large encoded datasets by @vale-salvatelli in Enabling DeepSMILE on large encoded datasets #637
• Add class-wise metrics logging and confusion matrix to DeepMIL by @harshita-s in Add class-wise metrics logging and confusion matrix to DeepMIL #647
• Workaround for bug in PL 1.5.5: CombinedLoader cannot be used with DDP for training data by @ant0nsc in Workaround for bug in PL 1.5.5: CombinedLoader cannot be used with DDP for training data #646
• Fix in _convert_to_tensor_if_necessary so that also np.array get converted correctly by @maxilse in Fix in _convert_to_tensor_if_necessary so that also np.array get converted correctly #649
• Fix for stuck Linux build: Move pytest to Windows by @ant0nsc in Fix for stuck Linux build: Move pytest to Windows #652
• Re-enable Linux build by @ant0nsc in Re-enable Linux build #655
• Add dropout to DeepMIL and fix feature extractor setup by @dccastro in Add dropout to DeepMIL and fix feature extractor setup #653
• Enable fine-tuning in Deepmil by @harshita-s in Enable fine-tuning in Deepmil #650
• Fix DeepMIL for TCGA CRCK dataset by @maxilse in Fix DeepMIL for TCGA CRCK dataset #659
• Add subsampling transform and mean pooling by @dccastro in Add subsampling transform and mean pooling #656
• Replace RadIO with TorchIO for patch-based inference by @fepegar in Replace RadIO with TorchIO for patch-based inference #666
• Fix DeepMIL metrics bug by @dccastro in Fix DeepMIL metrics bug #674
• Updated README.md by @ktakeda1 in Updated README.md #680
• Add FP and TN to DeepMIL outputs and configure outputs for multi-class classification by @harshita-s in Add FP and TN to DeepMIL outputs and configure outputs for multi-class classification #679
• Print out name of log level, not only the number by @fepegar in Print out name of log level, not only the number #678
• Fix missing separator error in docs Makefile by @fepegar in Fix missing separator error in docs Makefile #683
• Fix Markdown lint errors [skip ci] by @fepegar in Fix Markdown lint errors [skip ci] #684
• Pad model outputs if they are smaller than the inputs by @fepegar in Pad model outputs if they are smaller than the inputs #681
• Clean up legacy code by @javier-alvarez in Clean up legacy code #671
• Replace data quality folder with a link to the commit by @javier-alvarez in Replace data quality folder with a link to the commit #692
• Ensure the shape of input patches is compatible with model constraints by @fepegar in Ensure the shape of input patches is compatible with model constraints #682
• Update TorchIO version by @fepegar in Update TorchIO version #677
• DOC: Improve setup for running the HelloWorld model in AzureML by @ant0nsc in DOC: Improve setup for running the HelloWorld model in AzureML #693
• Specify formatter_class=argparse.ArgumentDefaultsHelpFormatter in generic_parsing.py to show default values for arguments in help message by @JacopoTeneggi in Specify formatter_class=argparse.ArgumentDefaultsHelpFormatter in generic_parsing.py to show default values for arguments in help message #689
• Fix some Sphinx warnings by @fepegar in Fix some Sphinx warnings #699
• Automatically and linearly scale the learning rate of the SSL encoder to the number of GPUS by @maxilse in Automatically and linearly scale the learning rate of the SSL encoder to the number of GPUS #667
• Import submodules to sys.path to fix autodoc's warning by @JacopoTeneggi in Import submodules to sys.path to fix autodoc's warning #704
• Update hi-ml submodule by @fepegar in Update hi-ml submodule #709
• Deprecate manual changelog by @fepegar in Deprecate manual changelog #707
• Ignore Sphinx build folders by @fepegar in Ignore Sphinx build folders #698
• ENH: Enforce prefixes for pull request titles by @fepegar in ENH: Enforce prefixes for pull request titles #713
• STYLE: Trim trailing whitespaces by @fepegar in STYLE: Trim trailing whitespaces #715
• STYLE: Ensure line breaks at the end of all files by @fepegar in STYLE: Ensure line breaks at the end of all files #717
• DOC: Add a smoke-test model for AzureML setup by @ant0nsc in DOC: Add a smoke-test model for AzureML setup #708
• ENH: Customized VSCode Window Color by @ant0nsc in ENH: Customized VSCode Window Color #712
• DOC: Add part of the API to the Sphinx docs by @fepegar in DOC: Add part of the API to the Sphinx docs #718
• BUG: Fix broken build pipeline by updating package versions by @peterhessey in BUG: Fix broken build pipeline by updating package versions #725
• ENH: Add a structured template for issues by @ant0nsc in ENH: Add a structured template for issues #727
• ENH: Auto add issues to GitHub Project by @peterhessey in ENH: Auto add issues to GitHub Project #721
• ENH: Add Environment Locking by @peterhessey in ENH: Add Environment Locking #735
• DOC: Improve documentation for dataset creation by @ant0nsc in DOC: Improve documentation for dataset creation #723
• STYLE: Fix pre-commit errors by @fepegar in STYLE: Fix pre-commit errors #736

New Contributors

• @ktakeda1 made their first contribution in Update README.md #511
• @vale-salvatelli made their first contribution in Environment and hello_world_model documentation updated #546
• @maxilse made their first contribution in Update hello_world_model.md #580
• @mebristo made their first contribution in Add histopathology module and add hi-ml as submodule  #603
• @harshita-s made their first contribution in Plotting heatmap and thumbnails for test PANDA slides  #634
• @fepegar made their first contribution in Replace RadIO with TorchIO for patch-based inference #666
• @peterhessey made their first contribution in BUG: Fix broken build pipeline by updating package versions #725

Full Changelog: v0.3...v0.4

---

This discussion was created from the release Environment locking, bug fixes, stability improvements + various other changes..