3D: Auto-adjust date ranges and remove 1-year limit; add data range API #1550

hmhngx · 2025-10-30T01:24:46Z

Description

(Please include a summary of the change and which issue is touched on. Please also include relevant motivation and context.)

This PR enhances the 3D visualization experience and supporting APIs:

Adds backend data range API endpoints to support 3D auto-adjustment.
Implements client-side utilities to auto-adjust 3D graphics based on available data.
Enables 3D date ranges greater than one year by removing the previous one-year limit.
Minor cleanup and comment updates in related UI components.

Fixes #1185

(In general, OED likes to have at least one issue associated with each pull request. Replace [issue] with the OED GitHub issue number. In the preview you will see an issue description if you hover over that number. You can create one yourself before doing this pull request. This is where details are normally given on what is being addressed. Note you should not use the word "Fixes" if it does not completely address the issue since the issue would automatically be closed on merging the pull request. In that case use "Partly Addresses #[issue].)

Type of change

(Check the ones that apply by placing an "x" instead of the space in the [ ] so it becomes [x])

Note merging this changes the database configuration.
This change requires a documentation update

Checklist

(Note what you have done by placing an "x" instead of the space in the [ ] so it becomes [x]. It is hoped you do all of them.)

I have followed the OED pull request ideas
I have removed text in ( ) from the issue request
You acknowledge that every person contributing to this work has signed the OED Contributing License Agreement and each author is listed in the Description section.

Limitations

API and UI changes should be validated across large datasets for performance.
Additional integration tests may be needed for edge cases around very sparse or extremely long time ranges.
Documentation for the new data range endpoints and 3D auto-adjustment behavior should be added in a follow-up if not covered by existing docs.

(Describe any issues that remain or work that should still be done.)

- Add numDays to ThreeDState for user-selectable date range span - Add threeDInterval to GraphState for calculated 3D date range - Add reducers and selectors for both fields - Support numDays in chart link parsing

- Add MAX_3D_DAYS (1095) and DEFAULT_3D_DAYS (365) constants - Implement calculateThreeDDateRange() with 4-case logic: * Bounded/Bounded: warn if > maxDays, else use picker values * Unbounded/Bounded: use [DRE - numDays, DRE] * Bounded/Unbounded: use [max(DRS, end-numDays), latest day] * Unbounded/Unbounded: use [end - numDays, latest day] - Update isValidThreeDInterval() to accept maxDays parameter

- Dropdown with options: 1 month, 6 months, 1 year, 2 years, max, custom - Custom input with validation (1-1095 days) - Updates Redux state on selection - Similar pattern to IntervalControlsComponent

- Calculate 3D date range using 4-case logic in useEffect - Use dataRange API to get available data dates - Store calculated interval in Redux (threeDInterval) - Show warning when date range exceeds maxDays - Use threeDInterval for 3D queries instead of queryTimeInterval - Add ThreeDIntervalControlsComponent to UI options - Fix infinite loop by only dispatching on interval change

- Include numDays in 3D chart link generation - Add translation keys: threeD.numDays, threeD.date.range.exceeds.max - Add translations for 6.months and 2.years - Support English, French, and Spanish

huss · 2025-11-16T18:33:26Z

~~The type check failed in the build and should be addressed.~~

huss

Thanks to @hmhngx for another contribution. I have made some comments to consider. I also have some thoughts that were not easy to put in a specific place so I'm putting them here.

The custom range (and maybe the non-custom ones but not checked) seems to include one too many days. For example, choosing 3 days seems to show 4 on the graphic.
- Update: I'm having problems testing this as the date range is often not honored. This applies to several other ones in the comment.
Setting the date range outside the range of readings caused a server error and it never graphed (ERROR: range lower bound must be less than or equal to range upper bound) and it then went to the client.
- Update: This now says rendering and also pops up a warning about the range.
Some actions are causing a client message of exceeding the max days when it seems valid but this might naturally go away with other items.
If set the upper bound on the date range that is above the actual readings available then get client message about exceeding days but still graphs the result.
~~Maybe change "threeD.y.axis.label": "Days of Calendar Year", on the 3D graphic label for the axis to "Date" since no longer just one year.~~
I'm thinking about whether the behavior should be changed from the issue if have only DRS is set. In this case, it would then do DRS to DRS + numDays. If there is not enough data in readings then the graphic would end early. This allows a user to easily get data from a given date forward. This also seems parallel to only have the DRE. What do you think?

As always, I'm happy to clarify and discuss as desired.

src/client/app/redux/api/readingsApi.ts

src/server/routes/unitReadings.js

src/client/app/components/ThreeDIntervalControlsComponent.tsx

src/client/app/redux/selectors/chartQuerySelectors.ts

src/client/app/redux/slices/graphSlice.ts

src/client/app/translations/data.ts

huss · 2025-11-17T18:42:27Z

src/client/app/types/redux/graph.ts

 	meterOrGroupID: MeterOrGroupID | undefined;
 	meterOrGroup: MeterOrGroup | undefined;
 	readingInterval: ReadingInterval;
+	numDays: number | undefined; // Number of days for 3D graphic span (undefined means use default: 1 year)


I'm curious about using undefined vs setting the default value to 1 year in src/client/app/redux/slices/graphSlice.ts.

- Add Reading.getMeterDataRange() and Reading.getGroupDataRange() methods - Add /dataRange/meters/:meter_id and /dataRange/groups/:group_id routes - Fix validation functions to accept single meter/group IDs - Fix day calculation mismatch: use diff('days') + 1 to match client calculation - Fix validateGroupThreeDReadingsParams to check group_id instead of meter_ids - Add proper error logging for data range requests

- Add dataRange query endpoint to readingsApi - Fix meterOrGroup type to use MeterOrGroup enum instead of string literal - Endpoint fetches min/max timestamps for meter or group data

- Fix 'one too many days' bug: subtract (numDays - 1) for correct calendar days - Add validation to prevent server errors when range exceeds available data - Ensure threeDStartDate >= minDataDate and threeDEndDate <= maxDataDate - Fix calculateThreeDDateRange to handle edge cases with data boundaries - Improve autoAdjustThreeDInterval validation for maxDataDate

- Fix undefined args error in selectThreeDQueryArgs selector - Return undefined args when shouldSkipQuery is true - Add non-null assertion for args in useQuery (safe when skip=true) - Remove unused updateTimeInterval import - Fix comment placement for numDays in graphSlice (OED style preference)

- Provide default 30-day range when queryTimeInterval is unbounded - Prevents DateRangePicker from showing error state (red highlighting) - Fix TimeInterval import path to use correct relative path - Add non-null assertion for args in ReadingsPerDaySelectComponent

- Change 'max' to '3 years' in 3D date range span dropdown - Add '3.years' translation key in English, French, and Spanish - Change y-axis label from 'Days of Calendar Year' to 'Date' - Fix alphabetical order of translation keys (2.years, 3.years, 6.months) - Remove unused getLabelForValue function

huss

Thanks to @hmhngx for the updated code. There are comments from the past that I think are unresolved & I made some new comments based on the changes. As indicated, it might be good to discuss the big picture ones to decide how they will be resolved. Also, there are three failing tests due to changed in 3D that need to be fixed so they pass.

huss · 2025-12-06T17:18:07Z

src/client/app/components/DateRangeComponent.tsx

+	const displayInterval = queryTimeInterval.getIsBounded()
+		? queryTimeInterval
+		: new TimeInterval(
+			moment().subtract(30, 'days').startOf('day'),


This seems to be causing several undesirable behaviors.

The date range is shown and set to the thirty days back from the current day. However, the actual 3D data goes back 1 year from the last day with data. Thus, they do not correspond to each other.

In src/client/app/components/ThreeDIntervalControlsComponent.tsx there is DEFAULT_3D_DAYS. How does this relate to the 30 days here? This relates to item 1.

It is directly setting the data range shown and not setting queryTimeInterval. As a result, the date range is set but not shown on several other graphic pages. It also is not honored.

I'm seeing warning, it not honoring the time interval & resetting the time interval in 3D.

My current thinking is the date range should not be modified and the 3D data is gotten from the server by going back by DEFAULT_3D_DAYS (currently 1 year). Thus, date range remains unbounded but 3D gets the correct data. I think it may be possible that this was causing an error given your comment. If so, maybe we should discuss options.

huss · 2025-12-06T18:08:15Z

src/client/app/redux/selectors/chartQuerySelectors.ts

 	selectSelectedUnit,
 	selectThreeDState,
-	(queryTimeInterval, selectedUnit, threeD) => {
+	(queryTimeInterval, threeDInterval, selectedUnit, threeD) => {


queryTimeInterval seems unused.

huss · 2025-12-06T18:32:47Z

src/client/app/translations/data.ts

 		"1.week": "1 week",
 		"1.year": "1 year",
 		"2.months": "2 months",
+		"2.years": "2 years",


Moving the items does not appear in other languages.

huss · 2025-12-06T18:33:49Z

src/client/app/translations/data.ts

 		"threeD.time": "Time",
 		"threeD.x.axis.label": "Hours of Day",
-		"threeD.y.axis.label": "Days of Calendar Year",
+		"threeD.y.axis.label": "Date",


The text change does not appear in other languages.

huss · 2025-12-06T18:44:10Z

src/client/app/utils/dateRangeCompatibility.ts

 	return TimeInterval.unbounded();
 }

 /**


I'm still unsure that the calculation should happen on the client. I think we should discuss to resolve. Then I can carefully review the code.

huss · 2025-12-06T18:53:28Z

src/server/models/Reading.js

+	 * @returns {Promise<{minDate: Moment | null, maxDate: Moment | null}>}
+	 */
+	static async getMeterDataRange(meterID, conn) {
+		const result = await conn.oneOrNone(`


OED normally prefers to have an SQL file for queries. Unless there is a reason not to do this then it should be changed. Also, I think using $1 is less desirable and not done in our SQL files.

This also applies to group below.

huss · 2025-12-06T18:58:25Z

src/server/models/Reading.js

+			FROM readings 
+			WHERE meter_id = $1
+		`, [meterID]);
+		if (result && result.min_date && result.max_date) {


My testing found (as I would expect) that if there are not readings so no min/max then you get:

result: { min_date: null, max_date: null }

Thus, I don't think you need the extra logic for this.

huss · 2025-12-06T19:04:40Z

src/server/routes/unitReadings.js

+				const durationInDays = timeInterval.endTimestamp.diff(timeInterval.startTimestamp, 'days') + 1;
+				// Security: Limit 3D to MAX_3D_DAYS (1095 days = 3 years) to prevent abuse from unauthenticated users.
+				// Frontend auto-adjustment provides UX, but backend must enforce security limits.
+				if (durationInDays > 1095) {


As was done on the client, it is nice to make limits a const though it will be a little different due to JS not TS. I think that should be done here, esp. since the value is used more than once.

huss · 2025-12-06T19:19:10Z

src/client/app/components/ThreeDIntervalControlsComponent.tsx

+			<div style={divTopBottomPadding}>
+				<p style={labelStyle}>
+					{translate('threeD.numDays')}:
+					<TooltipMarkerComponent page='home' helpTextId='help.home.threeD.numDays.tip' />


There should be a placeholder in src/client/app/components/TooltipHelpComponent.tsx for this popup help.

- Create get_meter_data_range.sql and get_group_data_range.sql - Use named parameters (${meterID}, ${groupID}) instead of $1 - Simplify null check logic using optional chaining

- Define MAX_3D_DAYS = 1095 at module level - Replace hardcoded 1095 values with constant - Improves maintainability and consistency

- Allow date range to remain unbounded when queryTimeInterval is unbounded - 3D component handles its own date range calculation using DEFAULT_3D_DAYS - Fixes mismatch between displayed date range and actual 3D data range

- queryTimeInterval was in dependencies but not used in selector - Selector uses threeDInterval instead for 3D date range

- Fix alphabetical order of 2.years, 3.years, 6.months in French and Spanish - Update y-axis label from 'Days of Calendar Year' to 'Date' in French and Spanish - Add help text placeholder for help.home.threeD.numDays.tip

- Compare useMeterZone and warnOnCumulativeReset against defaults - Previously compared values against themselves, always returning false - Now correctly detects when these fields differ from defaults

- Change 'reject_disable' to 'reject_disabled' in migration files - Matches TypeScript/JavaScript code and main enum definition - Prevents constraint violations when migrating existing data

hmhngx added 9 commits October 29, 2025 00:37

feat: add auto-adjustment for 3D date ranges > 1 year

8735d98

feat: add dataRange endpoint for 3D auto-adjustment

8688168

feat: add auto-adjustment utilities for 3D graphics

935204f

feat: add data range API endpoints and remove 3D 1-year limit

fa5716a

feat: add numDays and threeDInterval to Redux state

ba1f39b

- Add numDays to ThreeDState for user-selectable date range span - Add threeDInterval to GraphState for calculated 3D date range - Add reducers and selectors for both fields - Support numDays in chart link parsing

feat: add ThreeDIntervalControlsComponent for numDays selection

3554750

- Dropdown with options: 1 month, 6 months, 1 year, 2 years, max, custom - Custom input with validation (1-1095 days) - Updates Redux state on selection - Similar pattern to IntervalControlsComponent

feat: add chart link support and translations for numDays

d2656e6

- Include numDays in 3D chart link generation - Add translation keys: threeD.numDays, threeD.date.range.exceeds.max - Add translations for 6.months and 2.years - Support English, French, and Spanish

huss requested changes Nov 17, 2025

View reviewed changes

hmhngx added 6 commits December 2, 2025 01:18

Add dataRange API endpoint for 3D graphics

09d54e6

- Add dataRange query endpoint to readingsApi - Fix meterOrGroup type to use MeterOrGroup enum instead of string literal - Endpoint fetches min/max timestamps for meter or group data

huss requested changes Dec 6, 2025

View reviewed changes

hmhngx added 7 commits December 10, 2025 22:10

Move data range SQL queries to separate files

07a81a3

- Create get_meter_data_range.sql and get_group_data_range.sql - Use named parameters (${meterID}, ${groupID}) instead of $1 - Simplify null check logic using optional chaining

Add MAX_3D_DAYS constant to unitReadings routes

7965df2

- Define MAX_3D_DAYS = 1095 at module level - Replace hardcoded 1095 values with constant - Improves maintainability and consistency

Remove 30-day default from DateRangeComponent

5a89d21

- Allow date range to remain unbounded when queryTimeInterval is unbounded - 3D component handles its own date range calculation using DEFAULT_3D_DAYS - Fixes mismatch between displayed date range and actual 3D data range

Remove unused queryTimeInterval from selectThreeDQueryArgs

6b3b543

- queryTimeInterval was in dependencies but not used in selector - Selector uses threeDInterval instead for 3D date range

Fix translations and add 3D numDays help text

7f2062d

- Fix alphabetical order of 2.years, 3.years, 6.months in French and Spanish - Update y-axis label from 'Days of Calendar Year' to 'Date' in French and Spanish - Add help text placeholder for help.home.threeD.numDays.tip

Fix unsaved changes detection for CSV upload fields

c7a0d63

- Compare useMeterZone and warnOnCumulativeReset against defaults - Previously compared values against themselves, always returning false - Now correctly detects when these fields differ from defaults

Fix enum value mismatch in disable_checks migration

930ed03

- Change 'reject_disable' to 'reject_disabled' in migration files - Matches TypeScript/JavaScript code and main enum definition - Prevents constraint violations when migrating existing data

3D: Auto-adjust date ranges and remove 1-year limit; add data range API #1550

Are you sure you want to change the base?

3D: Auto-adjust date ranges and remove 1-year limit; add data range API #1550

Uh oh!

Conversation

hmhngx commented Oct 30, 2025

Description

Type of change

Checklist

Limitations

Uh oh!

huss commented Nov 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

huss left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

huss left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

huss commented Nov 16, 2025 •

edited

Loading

huss left a comment •

edited

Loading