[REVISION] Chapter: 7 - Large Data Sets

[njlyon0]

What guide are you proposing revisions for?

• EML Best Practices guide
• Dataset Design For Special Cases
• Domain Specific Guides

Chapter and Revision Information

Chapter Number: 7
Chapter Title: Large Data Sets
Current Version/Commit: Version 2 prerelease
File Path: guide-special-cases/large-offline.qmd
Reviewer(s): Jason Downing, Nick Lyon, Kate Morkeski, Sage Lichtenwalner
Review Date: 2025-09-08

Revision overview

Broadly, this group was happy with the chapter as-is except for some minor fixes and a perhaps less concrete but useful philosophical conversation about what kinds of large data are worth preserving and how a prospective data provider might make that judgement call. The conversation is recorded on a more granular level in the comment below this one so hopefully that puts this ethos translates well to those making this issue actionable.

Detailed Content Feedback

Minor Issues

• Section [9.2.1]: There are no commas in Corinna's address
  • Consensus that including specific personnel at all is less maintainable than a shared email address

Missing Content

• Section [9.1]: Need to define what constitutes "large" in the context of this document
• Section [9.2.1]: Need to clarify workflow (i.e., what goes through ezEML versus what needs to be mailed in a hard drive). Also this should take into account EDI move to AWS

Additional Recommendations and Comments

See conversation below for other elements that might prove useful context.

[njlyon0]

Misc. Group Feedback / Thoughts (9/8/25)

• General consensus was that this chapter is already pretty polished
• May need to define "large" (and/or revisit original definition if that has evolved in recent years)
• Possible duplicate/related issue is (Clarify data access expectations in 'Large Datasets' chapter #84 )
• Zenodo has a data limit of 50 GB by default (but upgradeable to 200 GB)
  • EDI can support 100 GB but how do you get those data to EDI to put into AWS? Send the hard drive physically?
• Confusion about 'what constitutes large?' results in IMs saying they would reach out to EDI directly to ask follow-up questions to see where their specific data fall
• Consensus that some vagueness around this topic is due to waiting to see where 100s of GB / TB of drone footage (for example) would live
• For machine-collected data that are raw and huge, do we want to archive the raw-est form? Or is some tidied (smaller) version of the data okay / preferable?
• Balance between 'good enough' and affordable
• SL wishlist for chapter:
  • Add references to sections 6-8 in the 9.1 saying 'if your data meet X criteria, go back to that earlier chapter'
  • Want confirmation from EDI about workflow for accepting large data
• Becoming more common to have / need really large datasets
  • Maybe acknowledge that this is part of the rationale for the chapter?
  • Despite limitations of offline storage, having the data be theoretically accessible is preferable to not having the data archived anywhere
• Institutions sometimes provide DOIs for very limited metadata (e.g., just title and abstract) which makes them less useful
• Perceived CSV / tabular focus of LTER and EDI is a limitation for this chapter