Issues 74 and 149 msa spc spd#167
Merged
Merged
Conversation
…conversion run-through
…ntained via the pynxtools-microstructure plugin
…n studies, skeleton code for parsing, to be modified still from pynxtools-apm and pynxtools-microstructure, spellchecking
… dm2, dm3, dm4, iii) go to images
…ry as intermediate objects and jupyter objects are not deallocated and thus memory safely returned to the operating system
Closed
…nt memory consumption overhead of jupyter notebooks
…ompress these to compose a representative set for the use case images, i.e. tif, bmp, with and without sidecar files, the sampling generates a configuration file (ods spreadsheet automatically, is useful also for showing how to ingest per project extra metadata), sub-sampling is required as the total number of images in the EM dataset collection contains more than 300k images which we currently cannot sieve through manually and for which the adding of atom_types is plagued by file names which do not include chemical symbols of elements in the material or not even notes on the sample material, tested regex based expression of parsers but that gave too many false atom_types specifically single character ones like C, S, P, H, so not useful, tested also usage of AI technology to analyze the file names, was similarly challenged and using only the resources I had available locally does not allow me to processed as many tokens as those generated by the >300k file's names.
…so to all other tiff parsers, namely convert to grayscale and checking image mode
…so to all other tiff parsers, namely convert to grayscale and checking image mode
…er development tackles topics other than the here mentioned issues74 and 149
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
04_batch_process.py