Incorrect long values returned due to json to java conversion

[gmegan]

I am having a problem with bad long values being returned due to the conversion to double that happens when all JSON numeric values are stored as Doubles.

This came up first when trying to get back time values as nanoseconds instead of rfc3339 string values. But we need to store very large long values in other fields as well. The application here is scientific instruments, so numerical errors are pretty high priority problems when we are looking at data storage.

From looking around, it does not look like there is a good way to intercept the JSON and convert to Long instead of Double... but maybe I am missing something? I would like to find some way to get these values returned as Long instead of Double. Maybe a different converter? I would appreciate any suggestions or workarounds.

Here is a snippet that recreates the problem:

        long lval = 1485370052974000001L;

        BatchPoints batchPoints = BatchPoints
                .database(dbName)
                .tag("async", "true")
                .retentionPolicy("autogen")
                .consistency(ConsistencyLevel.ALL)
                .build();
        Point point = Point.measurement("testProblems")
                .time(innano, TimeUnit.NANOSECONDS)
                .addField("double", 3.14)
                .addField("long", lval)
                .build();

        batchPoints.point(point);
        influxDB.write(batchPoints);

        Series series0 = result.getResults().get(0).getSeries().get(0);
        List<String> cols = series0.getColumns();
        List<Object> val0 = series0.getValues().get(0);

        Object outlval_obj = val0.get(cols.indexOf("long"));
        if (outlval_obj instanceof Double)
        {
            long outlval = ((Double)outlval_obj).longValue();
            if (outlval != lval)
            {
                System.err.println("Got bad lval back as double [" + (outlval_obj) + "] -> " + outlval + " != " + lval);
            }
        }

Here is a gist with more code and outputs:
https://gist.github.com/ghmegan/4ed652c8c8dbaf18976c6f5f4c0c6b55

[gmegan]

As a note, when I use the influx client and query the database directly, the long field shows the correct value. So it is stored correctly. The JSON contains the correct value in response to a curl query. This is why I am thinking it is the JSON to java conversion and not some other step in the process.

[Aleishus]

I met the same problems

[majst01]

Ok, please write a unit-test which shows the problem and raise a PR therefore, so anybody can see what exact problem you face.

Unit Test go here: https://github.com/influxdata/influxdb-java/blob/master/src/test/java/org/influxdb/TicketTests.java

[Aleishus]

I thought the json library moshi cause this problem ，just look at the method

com.squareup.moshi.JsonUtf8Reader#nextDouble

 int p = peeked;
    if (p == PEEKED_NONE) {
      p = doPeek();
    }

    if (p == PEEKED_LONG) {
      peeked = PEEKED_NONE;
      pathIndices[stackSize - 1]++;
      return (double) peekedLong;  //  force convert (｡•ˇ‸ˇ•｡)
    }
...

[majst01]

But just looking at the code gives an idea, showing the error in a unit-test is believing !

[gmegan]

I added the unit tests and made a pull request from my fork of this library.

#283

[majst01]

Any idea howto solve this ?

[gmegan]

I am either going to have to try to switch the converter, since this is a moshi problem... or, if that doesn't work, write my own query functions and translate the json by hand. I will have to find some fix, since we are working with large numerical data, so this problem is a deal breaker.

I have just been working along on other things, waiting to see if there are other people with similar problem who might have other suggestions... But at this point, I will probably need to be doing a fix in my fork in the next couple of weeks since I will need to start pushing our product out for testing.

[majst01]

Maybe we should link this issue to a new moshi issue, the square people are very helpful from my experience.

[majst01]

But reading the moshi code gives me:

https://github.com/square/moshi/blob/master/moshi/src/main/java/com/squareup/moshi/JsonReader.java

  /**
   * Returns the {@linkplain Token#NUMBER long} value of the next token, consuming it. If the next
   * token is a string, this method will attempt to parse it as a long. If the next token's numeric
   * value cannot be exactly represented by a Java {@code long}, this method throws.
   *
   * @throws JsonDataException if the next token is not a literal value, if the next literal value
   *     cannot be parsed as a number, or exactly represented as a long.
   */
  public abstract long nextLong() throws IOException;

So in principal, moshi is able to deserialize Long values. Probably the problem is somewhere up the stack.

[majst01]

Which is also tested:

https://github.com/square/moshi/blob/master/moshi/src/test/java/com/squareup/moshi/JsonReaderTest.java#L358

[echernyshev]

Hi!
Problem in org.influxdb.dto.QueryResult.Series class.

...
 public static class Series {
...
    private List<List<Object>> values;
...

A simple test reproducing an error:

public class MoshiTest{
    public static class Bean{
        private List<List<Object>> list;
        private List<Object> objectList;
        public List<List<Object>> getList(){
            return list;
        }

        public List<Object> getObjectList(){
            return objectList;
        }

        public void setList(List<List<Object>> list){
            this.list = list;
        }

        public void setObjectList(List<Object> objectList){
            this.objectList = objectList;
        }
    }

    @Test
    public void test() throws IOException{
        String json = "{\"list\":[[22, 11000]],\"objectList\":[22, 11]}";

        Moshi moshi = new Moshi.Builder().build();
        JsonAdapter<Bean> jsonAdapter = moshi.adapter(Bean.class);

        Bean bean = jsonAdapter.fromJson(json);
        Assert.assertEquals(bean.getList().get(0).get(0).getClass(), Double.class);
        Assert.assertEquals(bean.getList().get(0).get(1).getClass(), Double.class);
        Assert.assertEquals(bean.getObjectList().get(0).getClass(), Double.class);
        Assert.assertEquals(bean.getObjectList().get(1).getClass(), Double.class);
    }
}

The method com.squareup.moshi.StandardJsonAdapters.ObjectJsonAdapter.fromJson(JsonReader) always calls the function com.squareup.moshi.JsonReader.nextDouble():

...
    case NUMBER:
          return reader.nextDouble();
...

[kkarski]

Having the same issue. Was considering storing the datatypes along with the values but essentially came to the same conclusion that this is a client limitation, not DB.