Skip to content

Fixes #634 - Implemented data entry date option for TS data retrieval #927

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 14 commits into
base: develop
Choose a base branch
from
Open

Fixes #634 - Implemented data entry date option for TS data retrieval #927

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
867d05f
Implemented data entry data option for TS data retrieval, serializati…
zack-rma Oct 22, 2024
b920d9e
Created subclass of TimeSeries Record with custom deserializer to han…
zack-rma Oct 25, 2024
91ecb7b
Fixes build error
zack-rma Nov 4, 2024
4854ebf
Fixes test case failure
zack-rma Nov 4, 2024
f9c14ff
634 TimeSeries Subclass update
zack-rma Nov 5, 2024
7b89a8a
634 TimeSeries Subclass update - added Mixin test
zack-rma Nov 5, 2024
d9e00c1
Updated max version handling, added test case
zack-rma Dec 2, 2024
a3002c3
Updated based on Adam's feedback. Added serializer test on file, clea…
zack-rma Dec 19, 2024
6a0cc16
Updated based on Adam's feedback. Added error when attempting to stor…
zack-rma Dec 19, 2024
f609901
Renamed json file, fixed documentation
zack-rma Dec 20, 2024
ca235d0
Fixed data entry date request body checking
zack-rma Jan 6, 2025
ddc89a2
Cleaned up TS retrieval with data entry dates.
zack-rma Feb 12, 2025
67bcd17
Updated comment
zack-rma Feb 14, 2025
0ac074b
Added null check
zack-rma Apr 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions cwms-data-api/src/main/java/cwms/cda/api/Controllers.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ public final class Controllers {
public static final String UNIT_SYSTEM = "unit-system";

public static final String TIMESERIES_CATEGORY_LIKE = "timeseries-category-like";
public static final String INCLUDE_ENTRY_DATE = "include-entry-date";

public static final String LOCATION_CATEGORY_LIKE = "location-category-like";
public static final String LOCATION_GROUP_LIKE = "location-group-like";
Expand Down
71 changes: 24 additions & 47 deletions cwms-data-api/src/main/java/cwms/cda/api/TimeSeriesController.java
View file
Open in desktop
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Post/Patch should throw an error if the client is sending a time series with a data-entry-date as that field isn't editable

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a check for data entry date values

Original file line number Diff line number Diff line change
@@ -1,45 +1,7 @@
package cwms.cda.api;

import static com.codahale.metrics.MetricRegistry.name;
import static cwms.cda.api.Controllers.BEGIN;
import static cwms.cda.api.Controllers.CREATE;
import static cwms.cda.api.Controllers.CREATE_AS_LRTS;
import static cwms.cda.api.Controllers.CURSOR;
import static cwms.cda.api.Controllers.DATE_FORMAT;
import static cwms.cda.api.Controllers.DATUM;
import static cwms.cda.api.Controllers.DELETE;
import static cwms.cda.api.Controllers.END;
import static cwms.cda.api.Controllers.END_TIME_INCLUSIVE;
import static cwms.cda.api.Controllers.EXAMPLE_DATE;
import static cwms.cda.api.Controllers.FORMAT;
import static cwms.cda.api.Controllers.GET_ALL;
import static cwms.cda.api.Controllers.GET_ONE;
import static cwms.cda.api.Controllers.MAX_VERSION;
import static cwms.cda.api.Controllers.NAME;
import static cwms.cda.api.Controllers.NOT_SUPPORTED_YET;
import static cwms.cda.api.Controllers.OFFICE;
import static cwms.cda.api.Controllers.OVERRIDE_PROTECTION;
import static cwms.cda.api.Controllers.PAGE;
import static cwms.cda.api.Controllers.PAGE_SIZE;
import static cwms.cda.api.Controllers.RESULTS;
import static cwms.cda.api.Controllers.SIZE;
import static cwms.cda.api.Controllers.START_TIME_INCLUSIVE;
import static cwms.cda.api.Controllers.STATUS_200;
import static cwms.cda.api.Controllers.STATUS_400;
import static cwms.cda.api.Controllers.STATUS_404;
import static cwms.cda.api.Controllers.STATUS_501;
import static cwms.cda.api.Controllers.STORE_RULE;
import static cwms.cda.api.Controllers.TIMESERIES;
import static cwms.cda.api.Controllers.TIMEZONE;
import static cwms.cda.api.Controllers.UNIT;
import static cwms.cda.api.Controllers.UPDATE;
import static cwms.cda.api.Controllers.VERSION;
import static cwms.cda.api.Controllers.VERSION_DATE;
import static cwms.cda.api.Controllers.addDeprecatedContentTypeWarning;
import static cwms.cda.api.Controllers.queryParamAsClass;
import static cwms.cda.api.Controllers.queryParamAsZdt;
import static cwms.cda.api.Controllers.requiredParam;
import static cwms.cda.api.Controllers.requiredZdt;
import static cwms.cda.api.Controllers.*;

import com.codahale.metrics.Histogram;
import com.codahale.metrics.MetricRegistry;
Expand Down Expand Up @@ -68,6 +30,7 @@
import io.javalin.plugin.openapi.annotations.OpenApiRequestBody;
import io.javalin.plugin.openapi.annotations.OpenApiResponse;
import java.io.IOException;
import java.io.StringWriter;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
Expand All @@ -78,13 +41,13 @@
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.servlet.http.HttpServletResponse;
import org.apache.commons.io.IOUtils;
import org.jetbrains.annotations.NotNull;
import org.jooq.DSLContext;
import org.jooq.exception.DataAccessException;

public class TimeSeriesController implements CrudHandler {
private static final Logger logger = Logger.getLogger(TimeSeriesController.class.getName());

public static final String TAG = "TimeSeries";
public static final String STORE_RULE_DESC = "The business rule to use "
+ "when merging the incoming with existing data\n"
Expand Down Expand Up @@ -163,7 +126,8 @@ private Timer.Context markAndTime(String subject) {

@OpenApi(
description = "Used to create and save time-series data. Data to be stored must have "
+ "time stamps in UTC represented as epoch milliseconds ",
+ "time stamps in UTC represented as epoch milliseconds. If data entry date is included in the "
+ "request, it will be dropped. ",
requestBody = @OpenApiRequestBody(
content = {
@OpenApiContent(from = TimeSeries.class, type = Formats.JSONV2),
Expand Down Expand Up @@ -204,7 +168,7 @@ public void create(@NotNull Context ctx) {
TimeSeries timeSeries = deserializeTimeSeries(ctx);
dao.create(timeSeries, createAsLrts, storeRule, overrideProtection);
ctx.status(HttpServletResponse.SC_OK);
} catch (IOException | DataAccessException ex) {
} catch (DataAccessException | IOException ex) {
CdaError re = new CdaError("Internal Error");
logger.log(Level.SEVERE, re.toString(), ex);
ctx.status(HttpServletResponse.SC_INTERNAL_SERVER_ERROR).json(re);
Expand Down Expand Up @@ -382,6 +346,10 @@ public void delete(@NotNull Context ctx, @NotNull String timeseries) {
+ "\n* `xml`"
+ "\n* `wml2` (only if name field is specified)"
+ "\n* `json` (default)"),
@OpenApiParam(name = INCLUDE_ENTRY_DATE, type = Boolean.class, description = "Specifies "
+ "whether to include the data entry date of each value in the response. Including the data entry "
+ "date will increase the size of the array containing each data value from three to four, "
+ "changing the format of the response. Default is false."),
@OpenApiParam(name = PAGE, description = "This end point can return large amounts "
+ "of data as a series of pages. This parameter is used to describes the "
+ "current location in the response stream. This is an opaque "
Expand Down Expand Up @@ -431,6 +399,9 @@ public void getAll(@NotNull Context ctx) {

ZonedDateTime versionDate = queryParamAsZdt(ctx, VERSION_DATE);

boolean includeEntryDate = ctx.queryParamAsClass(INCLUDE_ENTRY_DATE, Boolean.class)
.getOrDefault(false);

// The following parameters are only used for jsonv2 and xmlv2
String cursor = queryParamAsClass(ctx, new String[]{PAGE, CURSOR},
String.class, "", metrics, name(TimeSeriesController.class.getName(),
Expand Down Expand Up @@ -463,7 +434,7 @@ public void getAll(@NotNull Context ctx) {

String office = requiredParam(ctx, OFFICE);
TimeSeries ts = dao.getTimeseries(cursor, pageSize, names, office, unit,
beginZdt, endZdt, versionDate, trim.getOrDefault(true));
beginZdt, endZdt, versionDate, trim.getOrDefault(true), includeEntryDate);

results = Formats.format(contentType, ts);

Expand Down Expand Up @@ -562,7 +533,6 @@ public void update(@NotNull Context ctx, @NotNull String id) {

TimeSeriesDao dao = getTimeSeriesDao(dsl);
TimeSeries timeSeries = deserializeTimeSeries(ctx);

boolean createAsLrts = ctx.queryParamAsClass(CREATE_AS_LRTS, Boolean.class)
.getOrDefault(false);
StoreRule storeRule = ctx.queryParamAsClass(STORE_RULE, StoreRule.class)
Expand All @@ -573,17 +543,24 @@ public void update(@NotNull Context ctx, @NotNull String id) {
dao.store(timeSeries, createAsLrts, storeRule, overrideProtection);

ctx.status(HttpServletResponse.SC_OK);
} catch (IOException | DataAccessException ex) {
} catch (DataAccessException | IOException ex) {
CdaError re = new CdaError("Internal Error");
logger.log(Level.SEVERE, re.toString(), ex);
ctx.status(HttpServletResponse.SC_INTERNAL_SERVER_ERROR).json(re);
}
}

private TimeSeries deserializeTimeSeries(Context ctx) throws IOException {
private TimeSeries deserializeTimeSeries(Context ctx) throws IOException
{
String contentTypeHeader = ctx.req.getContentType();
StringWriter writer = new StringWriter();
IOUtils.copy(ctx.bodyAsInputStream(), writer, StandardCharsets.UTF_8);
if (writer.toString().contains("data-entry-date"))
{
throw new IllegalArgumentException("Data entry date is not allowed in the request");
}
ContentType contentType = Formats.parseHeader(contentTypeHeader, TimeSeries.class);
return Formats.parseContent(contentType, ctx.bodyAsInputStream(), TimeSeries.class);
return Formats.parseContent(contentType, writer.toString(), TimeSeries.class);
}

/**
Expand Down
2 changes: 1 addition & 1 deletion cwms-data-api/src/main/java/cwms/cda/data/dao/LocationLevelsDaoImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,7 +249,7 @@ private static SEASONAL_VALUE_TAB_T getSeasonalValues(LocationLevel locationLeve
SEASONAL_VALUE_TAB_T pSeasonalValues = null;
if (seasonalValues != null && !seasonalValues.isEmpty()) {
pSeasonalValues = new SEASONAL_VALUE_TAB_T();
for(SeasonalValueBean seasonalValue : seasonalValues) {
for (SeasonalValueBean seasonalValue : seasonalValues) {
SEASONAL_VALUE_T seasonalValueT = new SEASONAL_VALUE_T();
seasonalValueT.setOFFSET_MINUTES(toBigDecimal(seasonalValue.getOffsetMinutes()));
if (seasonalValue.getOffsetMonths() != null) {
Expand Down
2 changes: 1 addition & 1 deletion cwms-data-api/src/main/java/cwms/cda/data/dao/TimeSeriesDao.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ void store(TimeSeries timeSeries, boolean createAsLrts,

TimeSeries getTimeseries(String cursor, int pageSize, String names, String office,
String unit, ZonedDateTime begin, ZonedDateTime end,
ZonedDateTime versionDate, boolean trim);
ZonedDateTime versionDate, boolean trim, boolean includeEntryDate);

String getTimeseries(String format, String names, String office, String unit, String datum,
ZonedDateTime begin, ZonedDateTime end, ZoneId timezone);
Expand Down
Loading
Loading