Skip to content

Conversation

@ehennestad
Copy link
Collaborator

@ehennestad ehennestad commented Jun 30, 2025

Motivation

Restructure documentation inspired by Diátaxis framework

Docs Preview

Todo

Points for improvements:

  • DataPipe currently does not have its own reference page. Will be added in a future PR
  • DataStub currently does not have its own reference page. Will be added in a future PR

Copilot summary

This pull request restructures the MatNWB documentation by adopting the Diátaxis framework, which organizes documentation into four types: tutorials (learning-oriented), how-to guides (problem-oriented), explanations/concepts (understanding-oriented), and reference (information-oriented).

  • Adds reference anchors to all tutorial files for cross-referencing
  • Reorganizes documentation structure with new sections for "Get Started," "How-tos," and "Concepts"
  • Creates new content including quickstart guide, overview, installation guide, and concept explanations
  • Added Create NWB File concept page with subpages.

Reviewed Changes

Copilot reviewed 42 out of 47 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
Tutorial files (.rst) Add reference anchors for cross-referencing
docs/source/index.rst Complete restructure to follow Diátaxis framework with new sections
New getting_started files Add quickstart, overview, and installation guides
New concepts files Add explanations for file reading, creation, and NWB concepts
Configuration files Update Sphinx configuration and add documentation README

Checklist

  • Have you ensured the PR description clearly describes the problem and solutions?
  • Have you checked to ensure that there aren't other open or previously closed Pull Requests for the same change?
  • If this PR fixes an issue, is the first line of the PR description fix #XX where XX is the issue number?

Fixed subsections about working with multiple schema versions
- Fill in current year for copyright
- Detect version from Contents.m file
- Change settings for navigation buttons
@ehennestad ehennestad changed the title Update pages on "Read with MatNWB" in the documentation Documentation: Restructure documentation based on Diátaxis framework Sep 4, 2025
@codecov
Copy link

codecov bot commented Sep 4, 2025

Codecov Report

❌ Patch coverage is 95.16129% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.59%. Comparing base (8d5173c) to head (33881e6).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
NwbFile.m 83.33% 2 Missing ⚠️
nwbExport.m 91.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #729      +/-   ##
==========================================
- Coverage   95.60%   95.59%   -0.01%     
==========================================
  Files         183      184       +1     
  Lines        6388     6443      +55     
==========================================
+ Hits         6107     6159      +52     
- Misses        281      284       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ehennestad ehennestad requested a review from Copilot September 25, 2025 16:49
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This pull request restructures the MatNWB documentation by adopting the Diátaxis framework, which organizes documentation into four types: tutorials (learning-oriented), how-to guides (problem-oriented), explanations/concepts (understanding-oriented), and reference (information-oriented).

  • Adds reference anchors to all tutorial files for cross-referencing
  • Reorganizes documentation structure with new sections for "Get Started," "How-tos," and "Concepts"
  • Creates comprehensive new content including quickstart guide, overview, installation guide, and concept explanations

Reviewed Changes

Copilot reviewed 42 out of 47 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
Tutorial files (.rst) Add reference anchors for cross-referencing
docs/source/index.rst Complete restructure to follow Diátaxis framework with new sections
New getting_started files Add quickstart, overview, and installation guides
New concepts files Add explanations for file reading, creation, and NWB concepts
Configuration files Update Sphinx configuration and add documentation README

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

ehennestad and others added 14 commits October 7, 2025 21:13
More clearly explain the rationale for chunking
Add sentence explaining the overhead of HTTP requests and why that favours larger chunk size for cloud optimization
…e or on export (#756)

* Add ConfigurationProfile enum and enhance config loading

Introduces the ConfigurationProfile enumeration class for dataset configuration profiles. Updates readDatasetConfiguration to use the new enum, adds an options argument for specifying a custom JSON file path, and improves input validation for file paths.

* Add resolveDatasetConfiguration utility

Introduces resolveDatasetConfiguration.m to handle NWB dataset configuration resolution from file paths, profile names, or direct struct input. Provides input validation and defaults to the standard configuration profile if no input is given.

* Add dataset settings configuration to NWB export

Introduces methods to apply dataset settings profiles in NwbFile and adds support for dataset configuration options in nwbExport. This enables users to specify dataset settings or profiles (e.g., for cloud or archive storage) prior to exporting NWB files, with an option to override existing settings.

* Add tests for dataset settings application in NWB export

Introduces unit tests to verify that dataset settings profiles are correctly applied via NWBFile and nwbExport functions, including checks for DataPipe configuration and output file creation when using the 'cloud' profile.

* Update NwbFile.m

* Update dataset settings argument documentation

Clarified and expanded documentation for DatasetSettings and added DatasetSettingsProfile argument in nwbExport.m. Updated usage examples to reflect new argument names and options.

* Update compression profile usage documentation

Revised instructions to reflect new methods for applying dataset settings, including use of NwbFile.applyDatasetSettings and updated export workflow. Clarified steps for customizing and loading configuration profiles, and improved code examples for better guidance.

* Improve docstring for applying dataset config in nwbExport/NwbFile

Enhanced documentation for dataset configuration methods in NwbFile and nwbExport, clarifying usage of profiles and custom settings. Renamed argument in NwbFile.applyDatasetSettings for clarity and updated method comments to better describe input options and behavior.

* Update compression_profiles.rst

Fix indentation, spaces instead of tab

* Update compression_profiles.rst

* Update NwbFile.m

* Add ConfigurationProfile enum to docs

* Add documentation for ConfigurationProfile enum

Added detailed class-level documentation to ConfigurationProfile describing available dataset configuration profiles and their intended use. Also fixed missing newline at end of matnwb_generateRstFilesFromCode.m.

* Add function tests for io.config namespace

* Fix negated expression in assertion

* Remove default values from method arguments

Default values for 'profile' and 'settingsReference' arguments were removed in two methods of NwbFile. This change enforces explicit argument passing and may help prevent unintended behavior due to implicit defaults.

* Update ApplyDatasetConfigurationTest.m

* Update nwbExport.m

* Update nwbExport.m

* Update compression_profiles.rst

Reorder sections, presenting the simple nwbExport first

* Update compression profiles documentation

Minor improvements

* Update compression_profiles.rst

* Update compression_profiles.rst

* Update compression_profiles.rst

* Clarify chunk size options in compression profiles doc

Expanded descriptions for 'flex', 'max', and integer options in the chunk size configuration section to improve clarity for users customizing compression profiles.

* Fix advice for TargetSizeExceeded warning in docs

Corrects the troubleshooting guidance for the TargetSizeExceeded warning by suggesting to increase target_chunk_size instead of lowering it.
Added clarification that in-place editing of dataset data is not supported in MatNWB and referenced the relevant GitHub issue for users seeking this functionality.
Added a 'Cite MatNWB' section to the index with a link to citation instructions. Improved references in the getting started overview by formatting class names as code and linking to the configuration profiles guide.
Improved explanations and updated links regarding NWB schemas, clarified when class regeneration can be skipped, and refined language for better accuracy. Removed unrealistic use case examples for generating classes in separate directories.
Added an 'important' section explaining MatNWB's lazy reading mechanism using DataStub objects. This clarifies how large datasets are handled efficiently and provides guidance on accessing and loading data.
@ehennestad ehennestad marked this pull request as ready for review October 23, 2025 18:32
@ehennestad ehennestad requested a review from bendichter October 23, 2025 18:32
Comment on lines 7 to 13

1. **Editing** data of datasets in place is currently not supported.

2. **Appending** data to a dataset requires the dataset to have been created as extendable. This is typically done when initially creating a dataset, using the :class:`~types.untyped.DataPipe` class. If the dataset was not created as extendable, it cannot be resized or appended to.

3. **Removing** property values or neurodata objects from the file object does not free up space in the file itself. If you need to significantly restructure a file, the standard approach is to create a new NWB file and copy the desired data into it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems odd to start a tutorial with what cannot be done

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a How-to for this really more than a concept guide

Copy link
Collaborator Author

@ehennestad ehennestad Oct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems odd to start a tutorial with what cannot be done

I think it's important to be upfront about it though. Also opened issues for following up and potentially fix these limitations

I think we need a How-to for this really more than a concept guide

Agree. This PR is already huge so I would prefer to do that as a separate task

Reorganized and clarified the documentation for editing NWB files with MatNWB.
Added sections on supported operations (adding and appending data), detailed current limitations (editing in-place, appending to non-extendable datasets, and removing data), and provided guidance on using PyNWB for advanced editing.
@ehennestad ehennestad requested a review from bendichter October 27, 2025 15:06
@bendichter bendichter merged commit 6cb513a into main Nov 4, 2025
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants