X-ray scattering analysis refactor#5
Conversation
- Using HubAppLayout to show both modules now - Updated some packages - Removed column variables
…ionality - Added @blueskyproject/tiled dependency for Tiled integration. - Updated Scattering component to include Tiled selection interface. - Modified RawDataOverviewAccordion to remove fetch button and improve data fetching. - Enhanced ScatterSubplot to handle scan URIs for image fetching. - Updated useAzimuthalIntegration hook to accept scan URIs for processing. - Refactored useRawDataOverview to manage scan URIs and handle Tiled folder selection. - Improved error handling and loading states across components. - Updated Vite configuration to support environment variables for Tiled integration.
- Using uv to handle backend dependencies - Added build arguments for SCATTERING_TILED_URL and SCATTERING_TILED_API_KEY in docker-compose.yml. - Updated frontend Dockerfile to use Node.js 24-alpine and accept build arguments for environment variables. - Set environment variables in Dockerfile to be used during the Vite build process.
…ration - Updated package.json to reflect new package name, version, and structure for exports - Removed old App component and created a new App component with routing for standalone and HubLayout pages - Added HubLayoutPage and StandalonePage components to demonstrate usage of Scattering component - Updated main entry point to use the new App structure - Configured Vite to build the library
- using 4 decimal places for slider min/max limits (now can reach full Q range) and tooltips - using 2 decimal places for axis tick labels
|
Addressed part of the feedback in the following commits:
|
Wiebke
left a comment
Wiebke
left a comment
There was a problem hiding this comment.
Thanks for putting this together. I left some comments based on a initial read-through of the code. Note that this focusses exclusively on the backend.
From a high-level, this is structured quite nicely.
My main concern is that line cuts (aside from azimuthal integration) make use of code that is custom defined here. For grazing incidence it would be more sensible to utilize the equivalent already available functionality in pyFAI. The custom code here looks sensible overall, but relies on snapping to integer positions. Ensuring that there aren't small pixel conversation mistakes or corner cases would ideally involve some tests.
| q_array_initial_1 = ai.qArray(scatter_image_array_1.shape) | ||
| chi_array_1 = ai.center_array(scatter_image_array_1.shape, unit="chi_rad") | ||
|
|
||
| q_array_initial_2 = ai.qArray(scatter_image_array_2.shape) | ||
| chi_array_2 = ai.center_array(scatter_image_array_2.shape, unit="chi_rad") |
There was a problem hiding this comment.
Both images use the same settings for integration, meaning there is an implicit assumption that both images also have the same calibration settings and data shape.
q and chi array's and derived masks for the current region of interest should thus also be the same. Is there another reason to compute them twice?
| for integration in azimuthal_integrations: | ||
| results["azimuthal"][integration["id"]] = create_error_linecut_result( | ||
| "Azimuthal integration not supported for GISAXS" | ||
| ) |
There was a problem hiding this comment.
In principle, azimuthal (and radial) integration can still be supported with GISAXS, but the functions to use are different for FiberIntegrator (integrate1d_polar).
| """Perform 1D azimuthal integration. Returns (q_values, intensities).""" | ||
| method = ("full", "csr", "cython") | ||
|
|
||
| result = ai.integrate1d( |
There was a problem hiding this comment.
correctSolidAngle is afaik just important if the detector is not mounted normally to the incident beam. This would be the case if tilt is != 0.
Since we haven't worked with a lot of non-normally mounted detectors I don't have a good sense how large the effect of this is.
For the polarization_factor, we have the default set to 0.99 based on prior code. pyFAI's documentation lists "synchrotrons are around 0.95".
Both are corrections to the intensity mapping and thus have less relevance for relative comparisons.
| def calculate_q_to_pixel_width( | ||
| q_position: float, | ||
| q_width: float, | ||
| q_matrix: np.ndarray, | ||
| direction: Literal["horizontal", "vertical"], | ||
| ) -> int: | ||
| """ | ||
| Convert a q-space width to pixel width. | ||
|
|
||
| Args: | ||
| q_position: Center position in q-space | ||
| q_width: Width in q-space units | ||
| q_matrix: 2D array of q-values | ||
| direction: "horizontal" or "vertical" | ||
|
|
||
| Returns: | ||
| Width in pixels | ||
| """ | ||
| if q_width <= 0 or q_matrix.size == 0: | ||
| return 0 | ||
|
|
||
| upper_q = q_position + q_width / 2 | ||
| lower_q = q_position - q_width / 2 | ||
|
|
||
| upper_pixel = find_pixel_position_for_q_value(upper_q, q_matrix, direction) | ||
| lower_pixel = find_pixel_position_for_q_value(lower_q, q_matrix, direction) | ||
|
|
||
| return abs(upper_pixel - lower_pixel) |
There was a problem hiding this comment.
This looks like it might introduce some inaccuracies if q_position is not exactly at the pixel center.
| def extract_inclined_linecut( | ||
| image_array: np.ndarray, | ||
| q_x_matrix: np.ndarray, | ||
| q_y_matrix: np.ndarray, | ||
| q_x_position: float, | ||
| q_y_position: float, | ||
| angle: float, | ||
| q_width: float = 0.0, | ||
| ) -> Tuple[np.ndarray, np.ndarray]: |
There was a problem hiding this comment.
This has nothing to do with the code, but I do not quite understand the rationale behind an inclined line cut, as in contrast to all other integration methodologies it doesn't integrate over a variable that stays constant.
I wonder if this functionality is left over from a misunderstanding where the requirement actually intended supporting a narrowly angled azimuthal integration.
| unit_ip="qip_nm^-1", | ||
| unit_oop="qoop_nm^-1", | ||
| incident_angle=incident_angle_deg, | ||
| tilt_angle=0.0, |
There was a problem hiding this comment.
How do incident_angle and tilt_angle interplay if we are working with an actually slightly titled detector (as determined through calibration)?
I think setting the tilt_angle to 0 is incorrect in this case.
| # Negate qoop to match image display convention (same as SAXS qy inversion): | ||
| # pyFAI's qoop follows detector coords where y increases downward, | ||
| # but we want positive qoop at top of image (Q increases going up). | ||
| qoop_values = -result.outofplane | ||
| qoop_pixel_matrix_inverted = -qoop_pixel_matrix |
There was a problem hiding this comment.
This looks like a data transformation for the purpose of visualization. Is there a reason why would we want to go against pyFAI's convention?
Worth noting that pyFAI's visualizations also show qoop increasing going up (https://pyfai.readthedocs.io/en/stable/usage/tutorial/FiberGrazingIncidence.html). Could this be a mismatch between specifying extent and pixel-coordinate to be treated at (0,0) when plotting?
|
|
||
|
|
||
| # ============================================================================= | ||
| # GISAXS Linecut Extraction (from transformed Q-space grid) |
There was a problem hiding this comment.
PyFAI supports direct horizontal and vertical integration. This also bypasses some of the issues of needing to snap to integer pixel coordinates.
###
horizontal_cut_pyFAI = fi.integrate1d_grazing_incidence(
data=detector_image,
mask=mask,
sample_orientation=1,
npt_ip=...,
### To integrate over a single pixel only (not what is usually done, just to show as the example)
oop_range=[target_qz - pixel_size_q_oop / 2, target_qz + pixel_size_q_oop / 2],
vertical_integration=False,
)
vertical_cut_pyFAI = fi.integrate1d_grazing_incidence(
data=detector_image,
mask=mask,
sample_orientation=1,
npt_oop=...,
### To integrate over a single pixel only
### (not what is usually done, just to show as the example)
ip_range=[target_qxy - pixel_size_q_ip / 2, target_qxy + pixel_size_q_ip / 2],
vertical_integration=True,
)
|
Priority summary:
Moving to issues:
Other notes:
|
Allow users to persist linecut extractions and batch processing results to a writable Tiled container. The feature is opt-in: setting SCATTERING_TILED_RESULTS_URL enables "Save to Tiled" buttons across all linecut graph cards and the batch results view.
Add enableTiledCalibration prop to Scattering component that controls whether the Tiled calibration browser is shown. The prop can only force-disable the feature; when omitted, the backend infrastructure check is the sole controller. The Tiled browser, calibration status, and mask resolution dialog are all hidden when disabled and only manual calibration input and mask upload remain available.
Backend save endpoints now capture the write_dataframe() return value and include tiled_id and tiled_uri in the response. Frontend shows a centered overlay with the saved item label, ID, and URI with copy-to-clipboard buttons. Saved items are tracked in sessionStorage for later display in the system status overlay.
- New HealthOverlay component shows service health and saved-to-Tiled items history with copy-to-clipboard support - Sidebar with fixed footer containing a health status button - When backend is unreachable, overlay opens automatically and blocks interaction until the backend recovers
The CalibrationWidget was making a direct fetch to the Tiled calibration server with an API key header and was the only direct frontend-to-Tiled HTTP request outside the <Tiled> component. This moves that logic to a new GET /api/load-calibration endpoint, which also resolves the associated mask in the same request (eliminating the separate /api/resolve-mask call and endpoint entirely).
|
Changes: Save results to Tiled
Dedicated calibration Tiled server
Calibration loading moved to backend
Health check
React unit tests
Other
|
Backend: - GISAXS horizontal/vertical linecut extraction now uses pyFAI - Default azimuthal integration npt to max(image.shape) for detector resolution Frontend: - Switch Q-space tick formatting and overlay width calculations from 2D matrices (qXMatrix/qYMatrix) to 1D vectors (qXVector/qYVector) - Add beam center overlay correction for GISAXS Q-space mode (maps to q_ip=0, q_oop=0 pixel indices) - Add per-figure Y-axis scale selector (linear/log/symlog) for all linecut and azimuthal integration figures - Add "Sync zoom" toggle to image toolbar (on by default) controlling whether heatmap zoom propagates to linecut figure X-axis domains - Add interaction help to image toolbar
|
Changes:
Mask handling, pixel-splitting, and GISAXS oversamplingThe pyFAI integration methods now receive the detector mask. For GISAXS, the grazing-incidence transformation reshapes and oversamples the image, introducing empty (unfilled) pixels that appear as gaps in the transformed detector frame. Pixel-splitting is not used for GISAXS linecuts (pyFAI's 
Following the pyFAI Fiber/Grazing-Incidence tutorial, it seems that the solution is to reduce Full range (1679 points): Half range (840 points): |
4 participants
Summary
This PR delivers an enhanced web platform for X-ray scattering analysis with GISAXS/SAXS support, integrated batch processing and accelerated server-side computation.
Main features
Linecut and azimuthal integration
GISAXS support
Mask management
Batch processing
Visualization
UI/UX
Component libraries
User feedback
Data management
Architecture
Backend
xscattering_backend)Frontend
DevOps & tooling
SCATTERING_BACKEND_prefix, standardized port to 4000Key dependency changes
Backend
Frontend
Added:
Removed: