Skip to content

backend-development.md

Latest commit

 

History

History
212 lines (175 loc) · 16.8 KB

backend-development.md

File metadata and controls

212 lines (175 loc) · 16.8 KB

.NET Backend Development Rules

Architecture

  • Use .NET 10 Web API with MongoDB integration
  • Follow RESTful API design principles
  • Implement proper error handling and logging
  • Use dependency injection for services

Key Files

Coding Standards

  • Use async/await for all database operations
  • Implement proper validation using DataAnnotations
  • Use structured logging with ILogger
  • Follow C# naming conventions (PascalCase for public members)
  • Use MongoDB.Driver for database operations
  • Implement proper CORS configuration for frontend integration

API Endpoints

JwstDataController (/api/jwstdata)

  • GET /api/jwstdata - Get all data
  • GET /api/jwstdata/{id} - Get by ID
  • GET /api/jwstdata/type/{dataType} - Filter by type
  • GET /api/jwstdata/status/{status} - Filter by status
  • GET /api/jwstdata/search/{searchTerm} - Search data
  • POST /api/jwstdata - Create new data
  • PUT /api/jwstdata/{id} - Update data
  • DELETE /api/jwstdata/{id} - Delete data
  • GET /api/jwstdata/lineage - Get all lineage groups
  • GET /api/jwstdata/lineage/{observationBaseId} - Get lineage for observation
  • POST /api/jwstdata/migrate/processing-levels - Backfill processing levels
  • POST /api/jwstdata/check-availability - Check if observations have existing data (AllowAnonymous)
  • POST /api/jwstdata/generate-thumbnails - Queue thumbnail generation for all viewable records without thumbnails
  • GET /api/jwstdata/{id}/thumbnail - Get thumbnail image for a record

MastController (/api/mast)

Search:

  • POST /api/mast/search/target - Search by target name
  • POST /api/mast/search/coordinates - Search by RA/Dec
  • POST /api/mast/search/observation - Search by observation ID
  • POST /api/mast/search/program - Search by program ID
  • POST /api/mast/products - Get data products for observation

Import:

  • POST /api/mast/download - Download FITS files (no DB records)
  • POST /api/mast/import - Download and import into MongoDB (supports downloadSource: "auto", "s3", "http")
  • GET /api/mast/import-progress/{jobId} - Get import progress
  • POST /api/mast/import/resume/{jobId} - Resume paused import
  • POST /api/mast/import/cancel/{jobId} - Cancel active import
  • GET /api/mast/import/resumable - List resumable jobs
  • POST /api/mast/import/from-existing/{obsId} - Import from downloaded files
  • GET /api/mast/import/check-files/{obsId} - Check if files exist

Metadata Management:

  • POST /api/mast/refresh-metadata/{obsId} - Re-fetch metadata for observation
  • POST /api/mast/refresh-metadata-all - Re-fetch metadata for all imports

DataManagementController (/api/datamanagement)

  • POST /api/datamanagement/search - Faceted search
  • GET /api/datamanagement/statistics - Data distribution stats
  • POST /api/datamanagement/export - Export data
  • POST /api/datamanagement/bulk/tags - Bulk tag updates
  • POST /api/datamanagement/bulk/status - Bulk status updates

CompositeController (/api/composite)

  • POST /api/composite/generate-nchannel - Generate N-channel composite with color mapping (1–N filters mapped to RGB via hue or explicit RGB weights)
  • POST /api/composite/generate-nchannel-async - Same payload as generate-nchannel, but routed through the job queue so authenticated wizard previews emit live SignalR progress (auth required, returns 202 with jobId, result via /api/jobs/{jobId}/result)
  • POST /api/composite/export-nchannel - Async N-channel composite export via job queue (requires auth, returns 202 with jobId, track via SignalR/polling, download via /api/jobs/{jobId}/result)
  • WCS alignment: channels are reprojected to a common celestial grid before RGB stacking
  • Per-channel controls: stretch, blackPoint, whitePoint, gamma, asinhA, curve, weight (0.0–2.0 intensity multiplier)
  • N-channel adds: color (hue 0-360°, explicit RGB weights, or luminance: true for LRGB), label, wavelength_um
  • Luminance channel: at most one per composite; contributes detail (lightness) via HSL blending instead of color
  • Optional global controls: overall.stretch, overall.blackPoint, overall.whitePoint, overall.gamma, overall.asinhA
  • Access model: anonymous users can only use public data; authenticated users can use own/public/shared; admins can use all

MosaicController (/api/mosaic)

  • POST /api/mosaic/generate - Generate WCS-aligned mosaic from 2+ source data IDs (png/jpeg/fits)
  • POST /api/mosaic/generate-and-save - Generate native FITS mosaic server-side and persist as a new data record
  • POST /api/mosaic/export - Async mosaic image export via job queue (requires auth, returns 202 with jobId, track via SignalR/polling, download via /api/jobs/{jobId}/result)
  • POST /api/mosaic/save - Async mosaic FITS save-to-library via job queue (requires auth, returns 202 with jobId, creates new data record)
  • POST /api/mosaic/footprint - Compute WCS footprint polygons for selected source files
  • Large-output pattern: use export or save for mosaics that would be too large for browser download/upload roundtrips
  • FITS mosaic metadata: /mosaic/generate FITS responses include provenance in primary headers plus a SRCMETA extension containing source FITS header cards

JobsController (/api/jobs)

  • GET /api/jobs - List jobs for current user (query: status, type)
  • GET /api/jobs/{jobId} - Get job status (ownership enforced). Returns a defensive snapshot — JobStatus.Messages is a server-side rolling buffer (cap 50, consecutive duplicates collapsed) the frontend LogPanel rehydrates from on SignalR reconnect (#1471).
  • POST /api/jobs/{jobId}/cancel - Cancel a job (ownership enforced)
  • GET /api/jobs/{jobId}/result - Stream blob result or return data ID (extends TTL on access)

DiscoveryController (/api/discovery)

  • GET /api/discovery/featured - Get curated featured targets list (12 targets with metadata, instruments, composite potential)
  • POST /api/discovery/suggest-recipes - Generate ranked composite recipe suggestions for a set of observations (proxies to Python recipe engine)

SearchController (/api/search)

  • GET /api/search/semantic?q=...&topK=20&minScore=0.3 - Natural language search over FITS metadata (anonymous, results access-controlled)
  • POST /api/search/reindex - Trigger full semantic re-index (admin only, returns 202 + jobId)
  • GET /api/search/index-status - Semantic index health (total indexed, model loaded)

AnalysisController (/api/analysis)

  • POST /api/analysis/region-statistics - Compute statistics for rectangle/ellipse regions (mean, median, std, min, max, sum, pixel count)
  • POST /api/analysis/detect-sources - Detect astronomical sources in a FITS image (returns list of sources with coordinates, flux, sharpness, roundness)
    • Parameters: thresholdSigma (1-50, default 5), fwhm (0.5-20, default 3), method (auto/daofind/iraf/segmentation), npixels, deblend
  • GET /api/analysis/table-info?dataId= - Get table HDU metadata for a FITS file
  • GET /api/analysis/table-data?dataId=&hduIndex=&page=&pageSize=&sortColumn=&sortDirection=&search= - Get paginated table data from a FITS binary table HDU
  • GET /api/analysis/spectral-data?dataId=&hduIndex=1 - Get spectral column arrays (wavelength, flux, error) for chart rendering

Viewer Smoothing Parameters

The following query parameters are available on both GET /api/jwstdata/{id}/preview and GET /api/jwstdata/{id}/histogram:

  • smoothMethod: Filter method — gaussian, median, box, astropy_gaussian, astropy_box, or "" (disabled, default)
  • smoothSigma: Gaussian sigma (0.1-10.0, default 1.0) — used for gaussian/astropy_gaussian methods
  • smoothSize: Kernel size (1-25, odd only, default 3) — used for median/box/astropy_box methods

Data Models

  • JwstDataModel: Main data entity with flexible metadata

    • ProcessingLevel: JWST pipeline stage (L1, L2a, L2b, L3)
    • ObservationBaseId: Groups related files by observation
    • ExposureId: Finer-grained lineage tracking
    • Metadata: Dictionary storing all MAST fields with mast_ prefix
    • IsViewable: true for images, false for tables/catalogs

  • ImageMetadata: For image-specific data

    • Core: targetName, instrument, filter, exposureTime, observationDate
    • MAST fields: wavelengthRange, calibrationLevel, proposalId, proposalPi, observationTitle
    • WCS: coordinateSystem, wcs (CRVAL1, CRVAL2)

  • SensorMetadata: For sensor/spectral data

  • SpectralMetadata: For spectral analysis data

  • CalibrationMetadata: For calibration files

  • ProcessingResult: For processing outcomes

  • LineageResponse/LineageFileInfo: DTOs for lineage queries

  • MetadataRefreshResponse: Response for metadata refresh operations

Security Notes

  • Current MongoDB credentials are for development only
  • JWT Bearer authentication is implemented (AuthService + JwtTokenService)
  • Use environment variables for sensitive configuration

Git Workflow

  • ALWAYS create a Pull Request (PR) after pushing
  • NEVER push directly to main
  • Workflow:
    1. Create feature branch (git checkout -b feature/name)
    2. Commit changes (git commit)
    3. Push to origin (git push)
    4. IMMEDIATELY create PR (gh pr create)
  • Use conventional commit messages
  • Atomic, focused commits