JSONDecode Error with dataset containing CAN-Bus signals

[dominikdh]

Hello,
as mentioned in the "README.md incorrect" issue, I have a problem with importing a dataset of an imc cronos measurement.
I get an JSONDecode Error (s. below). In the meantime I discovered, that the error only is raised, if the dataset contains CAN-Bus signals. If I remove them, I do not get this error.
But nevertheless, reading the data seems then only work for one channel - in all other channels the xdata and ydata are empty.

Please find below the error and attached an exemplary dataset.
Measurement.zip

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
Input In [18], in <cell line: 1>()
----> 1 data = raw.get_channels(True)

File IMCtermite.pyx:25, in IMCtermite.imctermite.get_channels()

File ~\.conda\envs\myAnaconda\lib\json\__init__.py:346, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    341     s = s.decode(detect_encoding(s), 'surrogatepass')
    343 if (cls is None and object_hook is None and
    344         parse_int is None and parse_float is None and
    345         parse_constant is None and object_pairs_hook is None and not kw):
--> 346     return _default_decoder.decode(s)
    347 if cls is None:
    348     cls = JSONDecoder

File ~\.conda\envs\myAnaconda\lib\json\decoder.py:337, in JSONDecoder.decode(self, s, _w)
    332 def decode(self, s, _w=WHITESPACE.match):
    333     """Return the Python representation of ``s`` (a ``str`` instance
    334     containing a JSON document).
    335 
    336     """
--> 337     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338     end = _w(s, end).end()
    339     if end != len(s):

File ~\.conda\envs\myAnaconda\lib\json\decoder.py:355, in JSONDecoder.raw_decode(self, s, idx)
    353     obj, end = self.scan_once(s, idx)
    354 except StopIteration as err:
--> 355     raise JSONDecodeError("Expecting value", s, err.value) from None
    356 return obj, end

JSONDecodeError: Expecting value: line 1 column 400 (char 399)
[Measurement.zip](https://github.com/RecordEvolution/IMCtermite/files/9615982/Measurement.zip)

[Priyanka260895]

is there any solution for multiple channel raw file?

[Priyanka260895]

`import IMCtermite
import json
import os
import datetime
import pdb

declare and initialize instance of "imctermite" by passing a raw-file

try :
imcraw = IMCtermite.imctermite(b"dn1.raw") # multiple channel raw file
except RuntimeError as e :
raise Exception("failed to load/parse raw-file: " + str(e))
pdb.set_trace()

obtain list of channels as list of dictionaries (without data)

channels = imcraw.get_channels(False)
print(json.dumps(channels,indent=4, sort_keys=False))
print("number of channels: " + str(len(channels)))

#pdb.set_trace()

obtain all channels (including full data)

channelsdata = imcraw.get_channels(True)

everything that follows is an example that specifically makes use only of

the first (index = 0) channel ...

idx = 0

if len(channelsdata) > 0 :

# get first channel's data
chnydata = channelsdata[idx]['ydata']
chnxdata = channelsdata[idx]['xdata']
print("xdata: " + str(len(chnxdata)))
print("ydata: " + str(len(chnydata)))

# extract trigger-time
trigtim = datetime.datetime.fromisoformat(channels[idx]["trigger-time"])
print(trigtim)

# file output of data with absolute timestamp in 1st column
filname = os.path.join("./",channelsdata[idx]['name']+".csv")
print("writing output into " + filname)
with open(filname,'w') as fout :
    # include column header
    fout.write( str(channelsdata[idx]['xname']) + '[' + str(channelsdata[idx]['xunit']) + "]"
              + ","
              + str(channelsdata[idx]['yname']) + '[' + str(channelsdata[idx]['yunit']) + "]"
              + "\n" )
    # add data (introduce time shift according to trigger-time)
    for row in range(0,len(chnxdata)) :
        fout.write( str( (trigtim + datetime.timedelta(seconds=chnxdata[row])).isoformat() )
                  + ","
                  + str( chnydata[row])
                  + "\n" )
`

[Priyanka260895]

but getting this error...

number of channels: 6
Traceback (most recent call last):
File "C:\Users\guptap15\Desktop\RAW\Raw4.py", line 20, in
channelsdata = imcraw.get_channels(True)
File "IMCtermite.pyx", line 25, in IMCtermite.imctermite.get_channels
File "C:\Users\guptap15\AppData\Local\Programs\Python\Python310\lib\json_init_.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Users\guptap15\AppData\Local\Programs\Python\Python310\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\guptap15\AppData\Local\Programs\Python\Python310\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 641 (char 640)

[mario-fink]

@Priyanka260895 on ef0bb75#commitcomment-100284938 : Have you tried installing the latest version 2.0.6 of the package (https://pypi.org/project/IMCtermite/) ? Any of the given Python examples should be appropriate to decode a multichannel dataset. Are you able to share a sample file of the data you're trying to read?

[Priyanka260895]

@mario-fink Multi-channel file like in the above comment measurement.raw is having multiple channels and that means when it will be converted to csv every channel will have a different column. But after updating the package, that conversion to csv is happening but data of only one channel is being generated. I'm stuck because I'm having raw files with multiple channels which if we convert to csv should have multiple columns. Due to some reasons, I'm not able to share the raw file with you. But if you find some solution to the problem I have explained, then it would be of great help.  

[mario-fink]

In order to extract the different channels of the above Measurement.raw file you might, for instance, use the following snippet:

import IMCtermite
import pandas

if __name__ == "__main__" :

    imctm = IMCtermite.imctermite(b"Measurement.raw")

    chns = imctm.get_channels(True)

    df = pandas.DataFrame()
    
    for idx,chn in enumerate(chns) :
        xcol = (chn['xname'] if chn['xname'] != '' else "x_"+str(idx))+" ["+chn['xunit']+"]"
        df[xcol] = pandas.Series(chn['xdata'])
        ycol = chn['yname']+" ["+chn['yunit']+"]"
        df[ycol] = pandas.Series(chn['ydata'])

    print(df)
    df.to_csv("Measurement.csv",header=True,sep='\t')

that writes all 12 channels and their specific dependencies in the time domain in individual columns.

[Priyanka260895]

@mario-fink Thankyou for your Continuous support.
It is work fine but, I have converted "Measurement.raw" file into csv format using FAMOS software and I am getting the data with timestamp in first column.

Data example:
meas.csv

could we get the output csv file in the same format (with time stamp) as given above?

[mario-fink]

@Priyanka260895 Please check out the new Python example in order to get the requested output format. And don't forget to upgrade to version v2.0.7 :)

[Priyanka260895]

@mario-fink Thankyou I have updated the version but for given code output is as below:-

    time [s]  AI_a01_x [m/s^2]  ...  AI_p05 [bar]  AI_p06 [bar]

0 0.00000 -0.119239 ... 1.480193 1.022401
1 0.00005 -0.124917 ... 1.480193 1.037661
2 0.00010 -0.068136 ... 1.464933 1.022401
3 0.00015 -0.096527 ... 1.419154 0.976622
4 0.00020 -0.238478 ... 1.464933 0.976622
... ... ... ... ... ...
112791 5.63955 0.198731 ... 1.525972 1.052921
112792 5.63960 0.102205 ... 1.525972 1.068180
112793 5.63965 0.193053 ... 1.449673 1.083440
112794 5.63970 0.227122 ... 1.434414 1.052921
112795 5.63975 0.153307 ... 1.480193 1.068180

[112796 rows x 13 columns]

but is it possible to change the time column value as date & time as below:-
17-02-2023 22:04:22

[mario-fink]

To get the full timestamp and a comma delimiter you might consider using:

import IMCtermite
import pandas
import datetime

def add_trigger_time(trigger_time, add_time) :
    trgts = datetime.datetime.strptime(trigger_time,'%Y-%m-%dT%H:%M:%S')
    dt = datetime.timedelta(seconds=add_time)
    return (trgts + dt).strftime('%Y-%m-%dT%H:%M:%S:%f')

if __name__ == "__main__" :

    # read file and extract data
    imctm = IMCtermite.imctermite(b"Measurement.raw")
    chns = imctm.get_channels(True)
    
    # prepare abscissa
    xcol = "timestamp ['%Y-%m-%dT%H:%M:%S:%f']"
    xsts = [add_trigger_time(chns[0]['trigger-time'],tm) for tm in chns[0]['xdata']]

    # sort channels
    chnnms = sorted([chn['name'] for chn in chns], reverse=False)
    chnsdict = {}
    for chn in chns :
        chnsdict[chn['name']] = chn

    # construct dataframe
    df = pandas.DataFrame()
    df[xcol] = pandas.Series(xsts)
    for chnnm in chnnms :
        chn = chnsdict[chnnm]
        ycol = chn['yname']+" ["+chn['yunit']+"]"
        df[ycol] = pandas.Series(chn['ydata'])

    # show entire dataframe and write file
    print(df)
    df.to_csv("Measurement.csv",header=True,sep=',',index=False)

resulting in

timestamp ['%Y-%m-%dT%H:%M:%S:%f'],AI_a01_x [m/s^2],AI_a01_y [m/s^2],AI_a01_z [m/s^2],AI_a02_x [m/s^2],AI_a02_y [m/s^2],AI_a02_z [m/s^2],AI_p01 [bar],AI_p02 [bar],AI_p03 [bar],AI_p04 [bar],AI_p05 [bar],AI_p06 [bar]
2022-08-30T11:40:21:000000,-0.119238804,-0.034912617,-0.070025491,-0.006025557,0.014546924,0.0,0.0,-0.152597204,0.137337484,0.0,1.480192883,1.02240127
2022-08-30T11:40:21:000050,-0.124916842,-0.058187695,-0.075860948,-0.003012778,0.014546924,0.002928929,-0.030519441,-0.076298602,0.167856925,0.0,1.480192883,1.03766099
2022-08-30T11:40:21:000100,-0.06813646,-0.069825235,0.0,-0.003012778,0.014546924,0.0,0.0,-0.01525972,0.167856925,0.01525972,1.464933162,1.02240127
2022-08-30T11:40:21:000150,-0.096526651,-0.069825235,-0.040848203,-0.006025557,0.014546924,0.0,-0.01525972,-0.030519441,0.106818043,0.030519441,1.419154001,0.976622108
2022-08-30T11:40:21:000200,-0.238477608,0.011637539,-0.029177288,-0.003012778,0.014546924,0.0,-0.01525972,0.01525972,0.091558323,-0.030519441,1.464933162,0.976622108

Simply adjust the time format string '%Y-%m-%dT%H:%M:%S:%f' according to your needs.
@Priyanka260895

[Priyanka260895]

@mario-fink Thankyou so much

[Priyanka260895]

@mario-fink Hi, could you please help me to this issues it would be great help.
blackary/st_pages#27 (comment)