Conversion/MIES_MassExperimentProcessing.ipf: Fix it #2569
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
t-b
force-pushed
the
bugfix/2569-mass-conversion-pxp-to-nwb
branch
from
72b2422 to
cd41b0b
Compare
7 tasks
t-b
force-pushed
the
bugfix/2569-mass-conversion-pxp-to-nwb
branch
4 times, most recently
from
4ccd555 to
416f622
Compare
4 tasks
MichaelHuth
requested changes
Nov 25, 2025
Collaborator
MichaelHuth
left a comment
MichaelHuth
left a comment
There was a problem hiding this comment.
Good description in commits what each specific problem was.
I think the handling of export errors can be improved.
This is about setting the dimension labels for the values waves and not the key wave. Bug introduced in 68c4087 (LNB: Fix empty column dimlabels in LNB and Results LNB keys waves, 2024-07-04).
This is required for a future commit where we want to call UpgradeLabNotebook from more places.
We introduced the retroactive computation of the NOTE_INDEX, which is the next free row, in 6535eba (UpgradeLabNotebook: Add NOTE_INDEX, 2024-10-25). The idea was that we take the first empty row starting from the end. But a real world file, Pvalb-IRES-Cre;Ai14(IVSCC)-165172.05.02.pxp, has an empty row as the first row (probably due to a bug) so the current approach breaks. Thinking about it again, what we really want to know is the row after the last filled row. So let's determine this. This is only an issue since 99ce799 (FindRange: Fix querying data from NaN sweep, 2024-10-25) as that added an optimization step to not look beyond NOTE_INDEX rows in FindRange used by GetLastSettingNoCache.
When loading experiments, possibly old experiments the data structures might be present in an old version. These need to be upgraded before used by the current code. The points where the LBN wave is upgraded are: - opening the Data Browser - Locking a device in the DAEphys panel - NWB Export At these locations the LBN upgrade for values and keys waves were added through the common function UpgradeLabNotebook.
This was broken in 2436f40 (GetAllFilesRecursivelyFromPath: Rework it completely, 2025-04-16). And also include the file from UTF_All_Includes.ipf so that this is compile tested.
This ensures that it always compiles and avoids regression on the fix in the previous commit.
The UpdateSweepPlot routine is called for every sweep change. But we only need to do the labnotebook upgrade once.
This matches the paths on the current developer machine of the author of this commit.
When we missed the labnotebook upgrade we get an invalid wave reference from GetLogbookKeysFromValues which is then passed into GetLogbookSettingsColumnFromSorted which calls BinarySearchText which fails with the below assertion. Let's add an explicit assertion as the below assertion is bogus. !!! Threadsafe assertion FAILED !!! Message: "Only works with 1D waves" Please provide the following information if you contact the MIES developers: ################################ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Stacktrace: AfterFileOpenHook(...)#L239 [MIES_MassExperimentProcessing.ipf] ProcessCurrentExperiment(...)#L110 [MIES_MassExperimentProcessing.ipf] PerformMiesTasks(...)#L154 [MIES_MassExperimentProcessing.ipf] NWB_ExportAllData(...)#L685 [MIES_NeuroDataWithoutBorders.ipf] NWB_GetFileForExport(...)#L190 [MIES_NeuroDataWithoutBorders.ipf] NWB_FirstStartTimeOfAllSweeps(...)#L89 [MIES_NeuroDataWithoutBorders.ipf] NWB_GetStartTimeOfSweep(...)#L49 [MIES_NeuroDataWithoutBorders.ipf] GetLastSettingTextIndep(...)#L308 [MIES_MiesUtilities_Logbook.ipf] GetLastSetting(...)#L590 [MIES_MiesUtilities_Logbook.ipf] GetLogbookSettingsColumn(...)#L558 [MIES_MiesUtilities_Logbook.ipf] GetLogbookSettingsColumnFromSorted(...)#L563 [MIES_MiesUtilities_Logbook.ipf] BinarySearchText(...)#L974 [MIES_Utilities_Algorithm.ipf]
The very first version of the labnotebook (prior to wave versioning) only had two columns. But some upgrade paths always assumed that it had more. Let's fix that by inserting the columns on non-matching name or if the column size is too small.
Due to IP bug #7699 global strings with the same name are picked up if the function reference returns a string. Let's just rename the function references to func to make it less likely to be a problem.
When the hardware folder root:MIES:HardwareDevices contains folders which don't follow our device naming scheme, we currently bug out. Avoid that by skipping the folder if it is not a device.
This is a code path which is triggered on NWB export. And we don't want to recreate the GUI state wave as that might fail for old versions. So let's just query the GUI control itself. A similiar fix was already applied in 4e62d8c (DAP: Change comment getter for user comment to retrieve string directly from control, 2025-04-02).
t-b
force-pushed
the
bugfix/2569-mass-conversion-pxp-to-nwb
branch
from
c703697 to
b9ce241
Compare
Then we can use it in TestMe() as well.
Since 72cb45a (NWB: Create separate NWB file for each device, 2024-08-13) we create separate NWB files, one for each device. This also means that NWB_ExportAllData asserts out if data from multiple devices is present and we have supplied a NWB path via outputFilePath. Instead of iterating over all devices and call NWB_ExportAllData for each device, which an earlier version of this patch did, we can also leverage overrideFileTemplate and let the file be internally derived.
A lot of the old existing experiments found on the machine of this developer had various file includes in the builtin procedure window. In order to make mass experiment processing and conversion possible let's create them empty so that compilation succeeds.
As we are doing the labnotebook upgrade now for every labnotebook we find, we want to be sure that we don't destroy anything. An easy test for that is to have the same initial labnotebook after throwing away the version. This already fails as we don't set the row dimension labels. Since forever we have set those only in ED_FindIndizesAndRedimension when adding entries. But it is actually cleaner to set them initially already.
This avoids having us to deal with the HistoryCarbonCopy notebook.
We now do all error handling inside the catch part, clear lingering RTEs before as usual and also cleanup left-over NWB files.
The DFREF getter does the upgrade from the old ITCDevices to HardwareDevices. So in case this exists, we need to upgrade the path before we use its string representation. While the current code is correct, it is less confusing in this way.
When we export data from multiple devices, we use NWB_FirstStartTimeOfAllSweeps() which in turn query the labnotebook. So we must be sure that we upgraded all labnotebooks from all devices already. But for upgrading the labnotebooks we also need to have finished datafolder renaming, so let's do that before as well.
…G returns nothing Some broken experiments have differing sweep data in the datafolder and the labnotebook. Let's add assertions in case we could not get the required single channel data.
t-b
force-pushed
the
bugfix/2569-mass-conversion-pxp-to-nwb
branch
from
b9ce241 to
7c44b0f
Compare
Collaborator
Author
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.