Loading historical data

[S54Mpower]

Hi,

When using following:
docker run --name vuegraf -d -v /home/myuser/vuegraf:/opt/vuegraf/conf jertel/vuegraf --resetdatabase --historydays=180

I am getting response,
docker: Error response from daemon: Conflict. The container name "/vuegraf" is already in use by container "abf94c9ef348d0e715ed6434ad19bfbbfbd9a9b72b249325dee1304b8950820d". You have to remove (or rename) that container to be able to reuse that name.

What would be docker command to just reset database and ingest historical data?

Thanks

[jertel]

Replace the --name vuegraf args with --rm. That will start an unnamed container to avoid the collision error, and then when it's done ingesting historical data it will remove the completed container, instead of leave it on the system.

[S54Mpower]

Hi,

I am still trying and unable to retrieve the historical data. Response I get is:
docker run --rm -d -v /home/sanjinhasanovic/vuegraf:/opt/vuegraf/conf jertel/vuegraf --resetdatabase --historydays=240 d509b69aa2b965fa38305470b43ae802ce05e257cec7aae0c90e14655326da35

However, Influx does not see data in the database.

[jertel]

Do you get any logging output by running docker logs d509b69aa2b965fa38305470b43ae802ce05e257cec7aae0c90e14655326da35?

[S54Mpower]

I get response no such container.

If I try following:
docker run -d -v /home/sanjinhasanovic/vuegraf:/opt/vuegraf/conf jertel/vuegraf --resetdatabase --historydays=240

I get following reply:
usage: vuegraf.py [-h] [--version] [-v] [-q] [--historydays HISTORYDAYS] [--resetdatabase] configFilename vuegraf.py: error: the following arguments are required: configFilename

Current data is working and logs are showing that new data points are being submitted, names are not working even if they are discovered as new channels, Influx is showing them as SHVue-1,2,3... (SHVue is name of device).

But logs are showing correct channel names:

2023-08-07 13:28:47.517929 | INFO | Using InfluxDB version 2
2023-08-07 13:28:47.595405 | INFO | Settings -> updateIntervalSecs: 60, detailedEnabled: true, detailedIntervalSecs: 3600
2023-08-07 13:28:52.854346 | INFO | Login completed
2023-08-07 13:28:53.990130 | INFO | Discovered new channel: SHVue (1,2,3)
2023-08-07 13:28:54.555988 | INFO | Discovered new channel: STEDNJAK (1)
2023-08-07 13:28:54.556135 | INFO | Discovered new channel: PC_IT (2)
2023-08-07 13:28:54.556184 | INFO | Discovered new channel: FRIZ_PECNICA_TV (3)
2023-08-07 13:28:54.556241 | INFO | Discovered new channel: PERILICA_SUDJA (4)
2023-08-07 13:28:54.556294 | INFO | Discovered new channel: MASINE_ROBA (5)
2023-08-07 13:28:54.556342 | INFO | Discovered new channel: RASVJETA (6)
2023-08-07 13:28:54.556383 | INFO | Discovered new channel: UTICNICE (7)
2023-08-07 13:28:54.556983 | INFO | Discovered new channel: RASVJETA_NAPA (8)
2023-08-07 13:28:54.557080 | INFO | Discovered new channel: None (9)
2023-08-07 13:28:54.557127 | INFO | Discovered new channel: None (10)
2023-08-07 13:28:54.557168 | INFO | Discovered new channel: None (11)
2023-08-07 13:28:54.557210 | INFO | Discovered new channel: None (12)
2023-08-07 13:28:54.557251 | INFO | Discovered new channel: MIKROVALNA (13)
2023-08-07 13:28:54.557292 | INFO | Discovered new channel: KLIME (14)
2023-08-07 13:28:54.557331 | INFO | Discovered new channel: GARAZA (15)
2023-08-07 13:28:54.557371 | INFO | Discovered new channel: RASVJETA (16)
2023-08-07 13:28:54.928257 | INFO | Submitting datapoints to database; account="SHVue"; points=14
2023-08-07 13:29:56.072486 | INFO | Submitting datapoints to database; account="SHVue"; points=14
2023-08-07 13:30:56.824453 | INFO | Submitting datapoints to database; account="SHVue"; points=14
2023-08-07 13:31:57.583974 | INFO | Submitting datapoints to database; account="SHVue"; points=14
2023-08-07 13:32:58.293080 | INFO | Submitting datapoints to database; account="SHVue"; points=14

So in total for now I cannot get the names right and struggling to ingest history data.

:)

[jertel]

As far as the channel name problem it appears you have a misconfiguration in your vuegraf.json. If you paste your config, after removing the token and other sensitive information, I may be able to help identify the problem.

Then use this docker command to ingest history:

docker run --rm -it -v /home/sanjinhasanovic/vuegraf:/opt/vuegraf/conf jertel/vuegraf /opt/vuegraf/conf/vuegraf.json --resetdatabase --historydays=240

This command will cause Vuegraf to remain in the foreground with log output sending to console, instead of running in the background. When it finishes ingesting, hit CTRL+C to exit. Then restart Vuegraf as you had previously been doing, before you attempted to ingest.

[S54Mpower]

Hi,

thanks, I will try to ingest the data again, in the meantime, please have a look below at my vuegraf.json.

{
    "influxDb": {
        "version": 2,   
        "url": "10.0.4.132:8086",
        "user": "vuegraf",
        "pass": "xxx",
        "database": "vuegraf",
        "bucket": "vuegraf",
        "org": "vuegraf",
        "reset": false,
        "ssl_enable": false,
        "ssl_verify": true,
        "token": "xxx"
    },
    "updateIntervalSecs": 60,
    "detailedDataEnabled": "true",
    "accounts": [
        {
            "name": "SHVue",
            "email": "xxx",
            "password": "xxx",
            "devices": [
                {
                    "name": "Right Panel",
                    "channels": [
                        "Stednjak",
                        "PC_IT",
                        "Friz_Pecnica_TV",
                        "Perilica_sudja",
                        "Masine_roba",
                        "Rasvjeta",
                        "Uticnice",
                        "Rasvjeta_Napa"
                    ]
                },
                {
                    "name": "Left Panel",
                    "channels": [
                        "none",
                        "none",
                        "none",
                        "none",
                        "Mikrovalna",
                        "Klime",
                        "Garaza",
                        "Rasvjeta"
                    ]
                }
            ]
        }
    ]
}

[jertel]

Your appear to only have one device yet your config file shows two devices. The device name in the config file must match the device name used by Vue (SHVue in your case). So you need to remove one of the panels from the config and then update the device name. Ex:

            "devices": [
                {
                    "name": "SHVue",
                    "channels": [
                        "Stednjak",
                        "PC IT",
                        "Friz Pecnica TV",
                        "Perilica sudja",
                        "Masine roba",
                        "Rasvjeta",
                        "Uticnice",
                        "Rasvjeta Napa",
                        "none",
                        "none",
                        "none",
                        "none",
                        "Mikrovalna",
                        "Klime",
                        "Garaza",
                        "Rasvjeta"
                    ]
                }
            ]

[jertel]

Also, you might want to consider changing the name of your device from SHVue to "SH Panel", so that the dashboards and alerts work properly. This is mentioned in the readme docs.

[S54Mpower]

After ingesting the data log is following, seems like data is being picked up but problem with database:

2023-08-07 14:39:15.703510 | INFO  | Resetting database
2023-08-07 14:39:16.444222 | INFO  | Settings -> updateIntervalSecs: 60, detailedEnabled: true, detailedIntervalSecs: 3600
2023-08-07 14:39:21.699115 | INFO  | Login completed
2023-08-07 14:39:22.722702 | INFO  | Discovered new channel: SHVue (1,2,3)
2023-08-07 14:39:23.323506 | INFO  | Discovered new channel: STEDNJAK (1)
2023-08-07 14:39:23.323650 | INFO  | Discovered new channel: PC_IT (2)
2023-08-07 14:39:23.323704 | INFO  | Discovered new channel: FRIZ_PECNICA_TV (3)
2023-08-07 14:39:23.323754 | INFO  | Discovered new channel: PERILICA_SUDJA (4)
2023-08-07 14:39:23.323827 | INFO  | Discovered new channel: MASINE_ROBA (5)
2023-08-07 14:39:23.323875 | INFO  | Discovered new channel: RASVJETA (6)
2023-08-07 14:39:23.323919 | INFO  | Discovered new channel: UTICNICE (7)
2023-08-07 14:39:23.323960 | INFO  | Discovered new channel: RASVJETA_NAPA (8)
2023-08-07 14:39:23.324011 | INFO  | Discovered new channel: None (9)
2023-08-07 14:39:23.324056 | INFO  | Discovered new channel: None (10)
2023-08-07 14:39:23.324098 | INFO  | Discovered new channel: None (11)
2023-08-07 14:39:23.324142 | INFO  | Discovered new channel: None (12)
2023-08-07 14:39:23.324187 | INFO  | Discovered new channel: MIKROVALNA (13)
2023-08-07 14:39:23.324229 | INFO  | Discovered new channel: KLIME (14)
2023-08-07 14:39:23.324272 | INFO  | Discovered new channel: GARAZA (15)
2023-08-07 14:39:23.324313 | INFO  | Discovered new channel: RASVJETA (16)
2023-08-07 14:39:23.669298 | INFO  | Loading historical data: 240 day(s) ago
2023-08-07 14:43:28.007499 | INFO  | Submitting datapoints to database; account="SHVue"; points=78365
2023-08-07 14:43:42.557538 | ERROR | Failed to record new usage data: (<class 'urllib3.exceptions.ReadTimeoutError'>, ReadTimeoutError("HTTPConnectionPool(host='10.0.4.132', port=8086): Read timed out. (read timeout=9.984789651000028)"), <traceback object at 0xffffb2a4bc80>)
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 466, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 461, in _make_request
    httplib_response = conn.getresponse()
                       ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/http/client.py", line 1378, in getresponse
    response.begin()
  File "/usr/local/lib/python3.11/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/socket.py", line 706, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/vuegraf/vuegraf.py", line 376, in <module>
    write_api.write(bucket=bucket, record=usageDataPoints)
  File "/usr/local/lib/python3.11/site-packages/influxdb_client/client/write_api.py", line 378, in write
    results = list(map(write_payload, payloads.items()))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/influxdb_client/client/write_api.py", line 376, in write_payload
    return self._post_write(_async_req, bucket, org, final_string, payload[0])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/influxdb_client/client/write_api.py", line 509, in _post_write
    return self._write_service.post_write(org=org, bucket=bucket, body=body, precision=precision,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/influxdb_client/service/write_service.py", line 60, in post_write
    (data) = self.post_write_with_http_info(org, bucket, body, **kwargs)  # noqa: E501
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/influxdb_client/service/write_service.py", line 90, in post_write_with_http_info
    return self.api_client.call_api(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/influxdb_client/_sync/api_client.py", line 343, in call_api
    return self.__call_api(resource_path, method,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/influxdb_client/_sync/api_client.py", line 173, in __call_api
    response_data = self.request(
                    ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/influxdb_client/_sync/api_client.py", line 388, in request
    return self.rest_client.POST(url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/influxdb_client/_sync/rest.py", line 311, in POST
    return self.request("POST", url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/influxdb_client/_sync/rest.py", line 220, in request
    r = self.pool_manager.request(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/request.py", line 78, in request
    return self.request_encode_body(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/request.py", line 170, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/poolmanager.py", line 376, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 798, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 525, in increment
    raise six.reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 714, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 468, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 357, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='10.0.4.132', port=8086): Read timed out. (read timeout=9.984789651000028)

[jertel]

It looks like your InfluxDB is too slow so Vuegraf is assuming it's failing. What kind of hardware are you running on?

[S54Mpower]

I am running it on RPi, but looking at the trends of Disk R/W times, CPU and Memory loads it doesn't look like there was any high loads.

[jertel]

You could try a very small historydays values, like 5 and see if it succeeds. If it does then it confirms it's a performance issue with the DB.