Skip to content

database-models.md

Latest commit

 

History

History
97 lines (71 loc) · 2.92 KB

database-models.md

File metadata and controls

97 lines (71 loc) · 2.92 KB

Database and Data Models Rules

Architecture

  • MongoDB document database for flexible data storage
  • Flexible schemas for various JWST data types
  • Support for metadata and processing results
  • Proper indexing for performance

Data Models

JwstDataModel (Main Entity)

Primary document model for all JWST data records.

Core Fields:

  • id: MongoDB ObjectId
  • fileName, filePath, fileSize, fileFormat
  • dataType: image, sensor, spectral, metadata, calibration, raw
  • processingStatus: pending, processing, completed, failed
  • uploadDate, tags, description

Lineage Fields:

  • processingLevel: L1, L2a, L2b, L3, unknown
  • observationBaseId: Groups related files (e.g., "jw02733-o001_t001_nircam")
  • exposureId: Fine-grained lineage tracking
  • parentId, derivedFrom: Parent-child relationships

Access Control Fields:

  • userId: Owner of the record (null for MAST scan imports)
  • isPublic: true for MAST-imported data, false for user uploads (controls anonymous visibility)

MAST Import Fields:

  • metadata: Dictionary with mast_* prefixed fields from MAST
  • isViewable: true for image files, false for tables/catalogs

Storage Fields:

  • filePath: Relative storage key (e.g., mast/{obs_id}/file.fits), not an absolute filesystem path. Resolved at runtime by the active storage provider (local or S3).

ImageMetadata

Image-specific metadata attached to JwstDataModel.

Astronomical Fields:

  • targetName: Object name (e.g., "NGC-3132")
  • instrument, filter, exposureTime
  • observationDate: Converted from MAST MJD format
  • coordinateSystem, wcs: World coordinate system

MAST-Specific Fields:

  • wavelengthRange: "INFRARED", "OPTICAL", "UV"
  • calibrationLevel: MAST calib_level (0-4)
  • proposalId, proposalPi: JWST program info
  • observationTitle: Program title

SensorMetadata

  • instrument, wavelength, dataPoints
  • samplingRate, integrationTime, detectorType

ProcessingResult

  • algorithm, processedDate, status
  • parameters, results, outputFilePath

MongoDB Configuration

  • Connection string: mongodb://admin:password@mongodb:27017
  • Database name: jwst_data_analysis
  • Collection: jwst_data

Data Types Supported

  • image: Astronomical images (FITS, JPG, PNG, TIFF)
  • sensor: Raw sensor data
  • spectral: Spectral analysis data
  • metadata: Descriptive information

Schema Design

  • Use flexible document structure for metadata
  • Implement proper validation
  • Support for file paths and processing status
  • Include user and timestamp information

Performance Considerations

  • Implement proper indexing for search operations
  • Use efficient queries for large datasets
  • Consider data archiving strategies
  • Implement proper error handling for database operations

Security

  • Current credentials are development-only
  • Implement proper access control in production
  • Use connection string encryption
  • Implement proper backup strategies