Skip to content

Commit 6cb513a

Browse files
ehennestadCopilotbendichter
authored
Documentation: Restructure documentation based on Diátaxis framework (#729)
* Update file_read.rst * Update nwbfile.rst * Create schemas_and_generation.rst * Update schemas_and_generation.rst Fixed subsections about working with multiple schema versions * Fix typos * Update conf.py - Fill in current year for copyright - Detect version from Contents.m file - Change settings for navigation buttons * Reorganize docs based on diataxis framework * Fix links * smaller rewordings * Update overview.rst Fixed references/links * Removed troubleshooting section llm generated, too generic and verbose * Minor rewording * Update installation.rst Fixed uninstall instruction * Update quickstart.rst Add small introduction * Update file_create.rst Make the file creation overview page more of an explanation page * Update file_create.rst Update rst formatting * Update nwbfile.rst Make it into an explanation page * Add neurodata types page * Minor reformulations * Update hdf5_considerations.rst * Update performance_optimization.rst * Update overview.rst Remove todos and final note which was too meta * Update performance_optimization.rst * Update docs/source/pages/getting_started/quickstart.rst Co-authored-by: Copilot <[email protected]> * Update nwbfile.rst Hide todo as comment * Update docs/source/pages/concepts/file_create.rst * Update docs/source/pages/concepts/file_create/hdf5_considerations.rst * Update docs/source/pages/concepts/file_create/hdf5_considerations.rst * Update docs/source/pages/concepts/file_create/nwbfile.rst * Update docs/source/pages/concepts/file_create/performance_optimization.rst * Update docs/source/pages/getting_started/overview.rst * Update docs/source/pages/getting_started/overview.rst * Rename considerations.rst to dimension_ordering.rst * Rename hdf5_considerations.rst to about_hdf5.rst * Update file_create.rst Change wording * Update overview.rst Smaller edits to be more direct Combine Learn Mode and Related Resources sections * Updating the file_create concept pages * Update editing_nwb_files.rst minor formatting changes * Change performance page and add how-to for using config profiles * Update performance_optimization.rst Fix formatting * Update compression_profiles.rst * Simplify config-profile how-to guide, add to main index * Update compression_profiles.rst * Update neurodata_types.rst * Update compression_profiles.rst * Update neurodata_types.rst Added section on time intervals * Update index.rst * Rename performance_optimization to storage_optimization * Update compression_profiles.rst Fixed reference * Update storage_optimization.rst More clearly explain the rationale for chunking * Update storage_optimization.rst Add sentence explaining the overhead of HTTP requests and why that favours larger chunk size for cloud optimization * Improve api for applying dataset configuration profiles to file before or on export (#756) * Add ConfigurationProfile enum and enhance config loading Introduces the ConfigurationProfile enumeration class for dataset configuration profiles. Updates readDatasetConfiguration to use the new enum, adds an options argument for specifying a custom JSON file path, and improves input validation for file paths. * Add resolveDatasetConfiguration utility Introduces resolveDatasetConfiguration.m to handle NWB dataset configuration resolution from file paths, profile names, or direct struct input. Provides input validation and defaults to the standard configuration profile if no input is given. * Add dataset settings configuration to NWB export Introduces methods to apply dataset settings profiles in NwbFile and adds support for dataset configuration options in nwbExport. This enables users to specify dataset settings or profiles (e.g., for cloud or archive storage) prior to exporting NWB files, with an option to override existing settings. * Add tests for dataset settings application in NWB export Introduces unit tests to verify that dataset settings profiles are correctly applied via NWBFile and nwbExport functions, including checks for DataPipe configuration and output file creation when using the 'cloud' profile. * Update NwbFile.m * Update dataset settings argument documentation Clarified and expanded documentation for DatasetSettings and added DatasetSettingsProfile argument in nwbExport.m. Updated usage examples to reflect new argument names and options. * Update compression profile usage documentation Revised instructions to reflect new methods for applying dataset settings, including use of NwbFile.applyDatasetSettings and updated export workflow. Clarified steps for customizing and loading configuration profiles, and improved code examples for better guidance. * Improve docstring for applying dataset config in nwbExport/NwbFile Enhanced documentation for dataset configuration methods in NwbFile and nwbExport, clarifying usage of profiles and custom settings. Renamed argument in NwbFile.applyDatasetSettings for clarity and updated method comments to better describe input options and behavior. * Update compression_profiles.rst Fix indentation, spaces instead of tab * Update compression_profiles.rst * Update NwbFile.m * Add ConfigurationProfile enum to docs * Add documentation for ConfigurationProfile enum Added detailed class-level documentation to ConfigurationProfile describing available dataset configuration profiles and their intended use. Also fixed missing newline at end of matnwb_generateRstFilesFromCode.m. * Add function tests for io.config namespace * Fix negated expression in assertion * Remove default values from method arguments Default values for 'profile' and 'settingsReference' arguments were removed in two methods of NwbFile. This change enforces explicit argument passing and may help prevent unintended behavior due to implicit defaults. * Update ApplyDatasetConfigurationTest.m * Update nwbExport.m * Update nwbExport.m * Update compression_profiles.rst Reorder sections, presenting the simple nwbExport first * Update compression profiles documentation Minor improvements * Update compression_profiles.rst * Update compression_profiles.rst * Update compression_profiles.rst * Clarify chunk size options in compression profiles doc Expanded descriptions for 'flex', 'max', and integer options in the chunk size configuration section to improve clarity for users customizing compression profiles. * Fix advice for TargetSizeExceeded warning in docs Corrects the troubleshooting guidance for the TargetSizeExceeded warning by suggesting to increase target_chunk_size instead of lowering it. * Document limitations on editing NWB datasets in MatNWB Added clarification that in-place editing of dataset data is not supported in MatNWB and referenced the relevant GitHub issue for users seeking this functionality. * Update editing_nwb_files.rst * Fix links and formatting issues * Update documentation with citation info and improved links Added a 'Cite MatNWB' section to the index with a link to citation instructions. Improved references in the getting started overview by formatting class names as code and linking to the configuration profiles guide. * Remove concepts page on neurodata types * Clarify NWB schema usage and class regeneration docs Improved explanations and updated links regarding NWB schemas, clarified when class regeneration can be skipped, and refined language for better accuracy. Removed unrealistic use case examples for generating classes in separate directories. * Add note on lazy loading with DataStub in MatNWB Added an 'important' section explaining MatNWB's lazy reading mechanism using DataStub objects. This clarifies how large datasets are handled efficiently and provides guidance on accessing and loading data. * Update storage_backends.rst * Apply suggestion from @bendichter Co-authored-by: Ben Dichter <[email protected]> * Update overview.rst * Revise documentation on editing NWB files in MatNWB Reorganized and clarified the documentation for editing NWB files with MatNWB. Added sections on supported operations (adding and appending data), detailed current limitations (editing in-place, appending to non-extendable datasets, and removing data), and provided guidance on using PyNWB for advanced editing. * Adjust introductions for index and overview pages * Remove page on editing NWB files * Update nwbfile.rst * Update storage_optimization.rst --------- Co-authored-by: Copilot <[email protected]> Co-authored-by: Ben Dichter <[email protected]>
1 parent 8d5173c commit 6cb513a

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

61 files changed

+1933
-125
lines changed

‎+io/+config/+enum/ConfigurationProfile.m‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
classdef ConfigurationProfile < handle
2+
%CONFIGURATIONPROFILE Dataset configuration profiles recognised by MatNWB.
3+
%
4+
% Use these enumeration members when selecting chunking/compression presets
5+
% via NwbFile.applyDatasetSettingsProfile, or nwbExport. Profiles map to
6+
% JSON files in the ``configuration`` folder:
7+
%
8+
% * ``default`` – general-purpose balance of size and performance.
9+
% * ``cloud`` – tuned for object storage and remote streaming access.
10+
% * ``archive`` – favors compact, long-term storage.
11+
% * ``none`` – opt out of applying a profile entirely.
12+
13+
enumeration
14+
none
15+
default
16+
cloud
17+
archive
18+
end
19+
end

‎+io/+config/readDatasetConfiguration.m‎

Lines changed: 47 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
function datasetConfig = readDatasetConfiguration(profile)
1+
function datasetConfig = readDatasetConfiguration(profile, options)
22
% READDATASETCONFIGURATION Reads the default dataset configuration from a JSON file.
33
%
44
% Syntax:
@@ -20,22 +20,61 @@
2020
%
2121
% % Load the default dataset configuration
2222
% datasetConfig = io.config.readDatasetConfiguration();
23+
% disp(datasetConfig);
24+
%
25+
% Example 2 - Load dataset configurations from a specific file ::
26+
%
27+
% datasetConfig = io.config.readDatasetConfiguration("FilePath", "configuration_file.json");
2328
% disp(datasetConfig);
2429

2530
arguments
26-
profile (1,1) string {mustBeMember(profile, [ ...
27-
"default", ...
28-
"cloud", ...
29-
"archive"
30-
])} = "default"
31+
profile (1,1) io.config.enum.ConfigurationProfile = "default"
32+
options.FilePath string {mustBeJsonFileOrEmpty} = string.empty
3133
end
3234

33-
filename = sprintf('%s_dataset_configuration.json', profile);
35+
if profile == io.config.enum.ConfigurationProfile.none && isempty(options.FilePath)
36+
datasetConfig = [];
37+
return
38+
end
3439

35-
configFilePath = fullfile(misc.getMatnwbDir, 'configuration', filename);
40+
% If FilePath is specified, we use that file
41+
if ~isempty(options.FilePath)
42+
configFilePath = options.FilePath;
43+
else
44+
filename = sprintf('%s_dataset_configuration.json', profile);
45+
configFilePath = fullfile(misc.getMatnwbDir, 'configuration', filename);
46+
end
47+
3648
datasetConfig = jsondecode(fileread(configFilePath));
3749
datasetConfig = datasetConfig.datasetSpecifications;
3850

3951
datasetConfig = io.config.internal.applyCustomMatNWBPropertyNames(datasetConfig);
4052
datasetConfig = io.config.internal.flipChunkDimensions(datasetConfig);
4153
end
54+
55+
function mustBeJsonFileOrEmpty(value)
56+
%MUSTBEJSONFILEOREMPTY Validate that input is a JSON file path or empty
57+
%
58+
% mustBeJsonFileOrEmpty(VALUE) throws an error if VALUE is not empty and
59+
% not a character vector or string scalar ending with '.json' (case-insensitive).
60+
61+
arguments
62+
value string
63+
end
64+
65+
if isempty(value)
66+
return
67+
end
68+
69+
assert(isscalar(value), ...
70+
"NWB:validator:mustBeJsonFileOrEmpty:InvalidInput", ...
71+
"Value must be a string scalar, character vector, or empty.");
72+
73+
assert(endsWith(value, ".json", "IgnoreCase", true), ...
74+
"NWB:validator:mustBeJsonFileOrEmpty:InvalidFileType", ...
75+
"Value must end with '.json'.");
76+
77+
assert(exist(value, "file") == 2, ...
78+
"NWB:validator:mustBeJsonFileOrEmpty:FileMustExist", ...
79+
"Value must be the name of an existing json file.")
80+
end

‎+io/+config/resolveDatasetConfiguration.m‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
function datasetConfig = resolveDatasetConfiguration(input)
2+
% resolveDatasetConfiguration - Resolves the dataset configuration based on the input.
3+
%
4+
% Syntax:
5+
% datasetConfig = io.config.resolveDatasetConfiguration(input)
6+
% This function resolves NWB dataset configurations from the specified input,
7+
% which can be a file path or a structure. If no input is provided, it
8+
% uses the default NWB dataset configuration profile.
9+
%
10+
% Input Arguments:
11+
% input {mustBeStringOrStruct} - A value to resolve configurations for,
12+
% which can either be a string representing the file path to the
13+
% configurations or a struct containing the configurations directly.
14+
%
15+
% Output Arguments:
16+
% datasetConfig - The NWB dataset configurations, returned as a structure.
17+
18+
arguments
19+
input {mustBeStringOrStruct} = struct.empty
20+
end
21+
22+
if isempty(input)
23+
disp('No dataset settings provided, using default dataset settings profile.')
24+
datasetConfig = io.config.readDatasetConfiguration();
25+
26+
elseif ischar(input) || (isstring(input) && isscalar(input))
27+
input = string(input);
28+
if isfile(input)
29+
datasetConfig = io.config.readDatasetConfiguration("FilePath", input);
30+
else
31+
datasetConfig = io.config.readDatasetConfiguration(input);
32+
end
33+
34+
elseif isstruct(input)
35+
datasetConfig = input;
36+
end
37+
end
38+
39+
function mustBeStringOrStruct(value)
40+
isValid = isempty(value) || ...
41+
ischar(value) || (isstring(value) && isscalar(value)) || ...
42+
isstruct(value);
43+
44+
assert(isValid, ...
45+
'NWB:ResolveDatasetSettings:InvalidInput', ...
46+
['Expected datasetSettings to be a string (profile name or filename) ' ...
47+
'or a struct (already loaded settings).'])
48+
end

‎+tests/+unit/+io/+config/ApplyDatasetConfigurationTest.m‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -274,5 +274,24 @@ function testApplyCustomMatNWBPropertyNames(testCase)
274274
testCase.verifyTrue( isfield(updatedConfig, 'VectorData_data') );
275275
testCase.verifyTrue( isfield(updatedConfig, 'GrayscaleImage_data') );
276276
end
277+
278+
function testNwbFileApplyDatasetSettingsProfile(testCase)
279+
nwbFile = tests.factory.NWBFile();
280+
281+
largeSeries = types.core.TimeSeries( ...
282+
'data', rand(64, 100000), ...
283+
'data_unit', 'n/a', ...
284+
'timestamps', 1:100000);
285+
286+
nwbFile.acquisition.set('data', largeSeries);
287+
288+
datasetConfig = nwbFile.applyDatasetSettingsProfile('cloud');
289+
290+
resultPipe = nwbFile.acquisition.get('data').data;
291+
testCase.verifyTrue(isa(resultPipe, 'types.untyped.DataPipe'), ...
292+
'applyDatasetSettings should configure datasets using named profile');
293+
testCase.verifyTrue(isstruct(datasetConfig), ...
294+
'applyDatasetSettings should return the dataset configuration that was applied');
295+
end
277296
end
278297
end

‎+tests/+unit/+io/+config/FunctionsTest.m‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
classdef FunctionsTest < matlab.unittest.TestCase
2+
% FunctionsTest - Test inputs and outputs of functions in io.config namespace
3+
4+
methods (Test)
5+
function testReadDatasetConfiguration(testCase)
6+
% Test with no inputs:
7+
defaultDatasetConfig = io.config.readDatasetConfiguration();
8+
testCase.verifyClass(defaultDatasetConfig, 'struct')
9+
10+
% Test with configuration profile name
11+
cloudDatasetConfig = io.config.readDatasetConfiguration('cloud');
12+
testCase.verifyClass(cloudDatasetConfig, 'struct')
13+
testCase.verifyNotEqual(cloudDatasetConfig, defaultDatasetConfig)
14+
15+
% Test with configuration profile name "none"
16+
noConfig = io.config.readDatasetConfiguration('none');
17+
testCase.verifyEmpty(noConfig')
18+
19+
% Test with filepath input
20+
filename = 'default_dataset_configuration.json';
21+
configFilePath = fullfile(misc.getMatnwbDir, 'configuration', filename);
22+
defaultDatasetConfigFromFile = io.config.readDatasetConfiguration('FilePath', configFilePath);
23+
testCase.verifyEqual(defaultDatasetConfigFromFile, defaultDatasetConfig)
24+
end
25+
26+
function testResolveDatasetConfiguration(testCase)
27+
% Test with no inputs (capture command window output):
28+
C = evalc("defaultDatasetConfigA = io.config.resolveDatasetConfiguration()"); %#ok<NASGU>
29+
testCase.verifyClass(defaultDatasetConfigA, 'struct')
30+
31+
% Test with structure input, i.e already loaded configuration
32+
defaultDatasetConfigB = io.config.resolveDatasetConfiguration(defaultDatasetConfigA);
33+
testCase.verifyEqual(defaultDatasetConfigB, defaultDatasetConfigA)
34+
35+
% Test with profile name
36+
defaultDatasetConfigC = io.config.resolveDatasetConfiguration("default");
37+
testCase.verifyEqual(defaultDatasetConfigC, defaultDatasetConfigA)
38+
39+
% Test with filepath input
40+
filename = 'default_dataset_configuration.json';
41+
configFilePath = fullfile(misc.getMatnwbDir, 'configuration', filename);
42+
defaultDatasetConfigD = io.config.resolveDatasetConfiguration(configFilePath);
43+
testCase.verifyEqual(defaultDatasetConfigD, defaultDatasetConfigA)
44+
end
45+
end
46+
end

‎+tests/+unit/nwbExportTest.m‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -156,6 +156,25 @@ function testExportTimeseriesWithoutStartingTimeRate(testCase)
156156
'NWB:CustomConstraintUnfulfilled')
157157
end
158158

159+
function testExportAppliesDatasetSettingsOption(testCase)
160+
nwb = tests.factory.NWBFile();
161+
largeSeries = types.core.TimeSeries( ...
162+
'data', rand(64, 100000), ...
163+
'data_unit', 'n/a', ...
164+
'timestamps', 1:100000);
165+
166+
nwb.acquisition.set('export_data', largeSeries);
167+
168+
nwbFilePath = testCase.getRandomFilename();
169+
nwbExport(nwb, nwbFilePath, 'DatasetSettingsProfile', 'cloud');
170+
171+
configuredData = nwb.acquisition.get('export_data').data;
172+
testCase.verifyTrue(isa(configuredData, 'types.untyped.DataPipe'), ...
173+
'nwbExport should configure datasets when DatasetSettings option is provided');
174+
testCase.verifyTrue(isfile(nwbFilePath), ...
175+
'nwbExport should still write the requested file');
176+
end
177+
159178
function testEmbeddedSpecs(testCase)
160179

161180
% Install extensions, one will be used, the other will not.

‎NwbFile.m‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,99 @@ function export(obj, filename, mode)
9090
end
9191
end
9292

93+
function datasetConfig = applyDatasetSettingsProfile(obj, profile, options)
94+
% APPLYDATASETSETTINGSPROFILE - Configure datasets using predefined settings profile
95+
%
96+
% Syntax:
97+
% nwb.applyDatasetSettingsProfile(profile) applies a dataset
98+
% configuration profile to the nwb-file ``nwb``. Available profiles:
99+
% "default", "cloud", "archive". This will configure datasets in
100+
% the NwbFile object for chunking and compression.
101+
%
102+
% Input Arguments:
103+
% - obj (NwbFile) -
104+
% An instance of the NwbFile class.
105+
%
106+
% - profile (ConfigurationProfile) -
107+
% Specifies the settings profile to use. Default is "none".
108+
%
109+
% Name-Value Arguments:
110+
% - OverrideExisting (logical) -
111+
% This boolean determines if existing DataPipe objects in the
112+
% file will be reconfigured with the provided options. Default is
113+
% false. **Important**: This does not work for DataPipes that has
114+
% previously been exported to file.
115+
%
116+
% Output Arguments:
117+
% - datasetConfig -
118+
% (Optional) The configuration settings applied to the dataset.
119+
%
120+
% See also:
121+
% io.config.enum.ConfigurationProfile
122+
% NwbFile.applyDatasetSettings
123+
124+
arguments
125+
obj (1,1) NwbFile
126+
profile (1,1) io.config.enum.ConfigurationProfile
127+
options.OverrideExisting (1,1) logical = false
128+
end
129+
130+
datasetConfig = io.config.readDatasetConfiguration(profile);
131+
nvPairs = namedargs2cell(options);
132+
obj.applyDatasetSettings(datasetConfig, nvPairs{:});
133+
if ~nargout
134+
clear datasetConfig
135+
end
136+
end
137+
138+
139+
function datasetConfig = applyDatasetSettings(obj, settingsReference, options)
140+
% APPLYDATASETSETTINGS - Configure datasets using NWB dataset settings
141+
%
142+
% Syntax:
143+
% nwb.applyDatasetSettings(settingsReference) applies a dataset
144+
% configuration profile to the nwb-file ``nwb``. This method
145+
% accepts the filename of a custom configuration profile or a
146+
% structure representing a configuration profile.
147+
%
148+
% Input Arguments:
149+
% - obj (NwbFile) -
150+
% An instance of the NwbFile class.
151+
%
152+
% - settingsReference (string | struct) -
153+
% The filename of a custom configuration profile or an in-memory
154+
% structure representing a configuration profile.
155+
%
156+
% Name-Value Arguments:
157+
% - OverrideExisting (logical) -
158+
% This boolean determines if existing DataPipe objects in the
159+
% file will be reconfigured with the provided options. Default is
160+
% false. **Important**: This does not work for DataPipes that has
161+
% previously been exported to file.
162+
%
163+
% Output Arguments:
164+
% - datasetConfig -
165+
% (Optional) The configuration settings applied to the dataset.
166+
%
167+
% See also:
168+
% io.config.enum.ConfigurationProfile
169+
% NwbFile.applyDatasetSettingsProfile
170+
171+
arguments
172+
obj (1,1) NwbFile
173+
settingsReference
174+
options.OverrideExisting (1,1) logical = false
175+
end
176+
177+
datasetConfig = io.config.resolveDatasetConfiguration(settingsReference);
178+
179+
nvPairs = namedargs2cell(options);
180+
io.config.applyDatasetConfiguration(obj, datasetConfig, nvPairs{:});
181+
if ~nargout
182+
clear datasetConfig
183+
end
184+
end
185+
93186
function o = resolve(obj, path)
94187
if ischar(path)
95188
path = {path};

0 commit comments

Comments
 (0)