solr_dist have unhandeled exception when parent_dataset is not found.

[magnarem]

In the dmci solr_dist, the following line of code might throw an exception if no parent is found:

https://github.com/metno/discovery-metadata-catalog-ingestor/blob/main/dmci/distributors/solr_dist.py#L103

The corresponding code called in the solrindexer is:

def get_dataset(self, id):
        """
        Use real-time get to fetch latest dataset
        based on id.
        """
        res = None
        try:
            res = requests.get(self.solr_url + '/get?wt=json&id=' + id,
                               auth=self.authentication)
            res.raise_for_status()
        except requests.exceptions.HTTPError as errh:
            logger.error("Http Error: %s", errh)
        except requests.exceptions.ConnectionError as errc:
            logger.error("Error Connecting: %s", errc)
        except requests.exceptions.Timeout as errt:
            logger.error("Timeout Error: %s", errt)
        except requests.exceptions.RequestException as err:
            logger.error("OOps: Something Else went wrong: %s", err)

        if res is None:
            return None
        else:
            dataset = res.json()
            return dataset

The error stacktrace from dmci looks like this:

[2024-10-04 09:40:45,234] [140060971999232] [MainThread]        solrindexer.indexdata:1481 ERROR    Http Error: 404 Client Error: Not Found for url: http://solr:8983/solr/senda/get?wt=json&id=no-met-e644c282-16eb-46b7-944d-36baaa14b179                                                                                                                                      │
│ [2024-10-04 09:40:45,235]                 dmci.api.app:838  ERROR    Exception on /v1/insert [POST]                                                                                                                                                                                                                                                                              │
│ Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                               │
│   File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 974, in json                                                                                                                                                                                                                                                                                           │
│     return complexjson.loads(self.text, **kwargs)                                                                                                                                                                                                                                                                                                                                │
│   File "/usr/lib/python3.10/json/__init__.py", line 346, in loads                                                                                                                                                                                                                                                                                                                │
│     return _default_decoder.decode(s)                                                                                                                                                                                                                                                                                                                                            │
│   File "/usr/lib/python3.10/json/decoder.py", line 337, in decode                                                                                                                                                                                                                                                                                                                │
│     obj, end = self.raw_decode(s, idx=_w(s, 0).end())                                                                                                                                                                                                                                                                                                                            │
│   File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode                                                                                                                                                                                                                                                                                                            │
│     raise JSONDecodeError("Expecting value", s, err.value) from None                                                                                                                                                                                                                                                                                                             │
│ json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)                                                                                                                                                                                                                                                                                                          │
│                                                                                                                                                                                                                                                                                                                                                                                  │
│ During handling of the above exception, another exception occurred:                                                                                                                                                                                                                                                                                                              │
│                                                                                                                                                                                                                                                                                                                                                                                  │
│ Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                               │
│   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1473, in wsgi_app                                                                                                                                                                                                                                                                                            │
│     response = self.full_dispatch_request()                                                                                                                                                                                                                                                                                                                                      │
│   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 882, in full_dispatch_request                                                                                                                                                                                                                                                                                │
│     rv = self.handle_user_exception(e)                                                                                                                                                                                                                                                                                                                                           │
│   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 880, in full_dispatch_request                                                                                                                                                                                                                                                                                │
│     rv = self.dispatch_request()                                                                                                                                                                                                                                                                                                                                                 │
│   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 865, in dispatch_request                                                                                                                                                                                                                                                                                     │
│     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]                                                                                                                                                                                                                                                                      │
│   File "/usr/local/lib/python3.10/dist-packages/dmci/api/app.py", line 69, in post_insert                                                                                                                                                                                                                                                                                        │
│     msg, code = self._insert_update_method_post("insert", request)                                                                                                                                                                                                                                                                                                               │
│   File "/usr/local/lib/python3.10/dist-packages/dmci/api/app.py", line 157, in _insert_update_method_post                                                                                                                                                                                                                                                                        │
│     err = self._distributor_wrapper(worker)                                                                                                                                                                                                                                                                                                                                      │
│   File "/usr/local/lib/python3.10/dist-packages/dmci/api/app.py", line 198, in _distributor_wrapper                                                                                                                                                                                                                                                                              │
│     status, valid, _, failed, skipped, failed_msg = worker.distribute()                                                                                                                                                                                                                                                                                                          │
│   File "/usr/local/lib/python3.10/dist-packages/dmci/api/worker.py", line 199, in distribute                                                                                                                                                                                                                                                                                     │
│     obj_status, obj_msg = obj.run()                                                                                                                                                                                                                                                                                                                                              │
│   File "/usr/local/lib/python3.10/dist-packages/dmci/distributors/solr_dist.py", line 78, in run                                                                                                                                                                                                                                                                                 │
│     status, msg = self._add()                                                                                                                                                                                                                                                                                                                                                    │
│   File "/usr/local/lib/python3.10/dist-packages/dmci/distributors/solr_dist.py", line 103, in _add                                                                                                                                                                                                                                                                               │
│     isIndexed = self.mysolr.get_dataset(newdoc['id'])                                                                                                                                                                                                                                                                                                                            │
│   File "/usr/local/lib/python3.10/dist-packages/solrindexer/indexdata.py", line 1492, in get_dataset                                                                                                                                                                                                                                                                             │
│     dataset = res.json()                                                                                                                                                                                                                                                                                                                                                         │
│   File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 978, in json                                

So it is the line in solrindexer.

dataset = res.json(), witch fails.

So what will be the best solution to handle this?

add a try: and except block at https://github.com/metno/discovery-metadata-catalog-ingestor/blob/main/dmci/distributors/solr_dist.py#L103

or I have to add a line with res = None in the get_dataset()-function in the solrindexer module when an exception is thrown?

This error does not break the dmci, but It would be good to not have unhandeled exceptions in the code.

[magnarem]

Maybe this is not so critical or important after all.

I cannot se more of those errors from dmci, so It seems like this error happend when I had deleted the solr core, to create a new one with the new solr schema config. So there were no cores for this query to happend on.
So this is a very special case, that will not happen unless special circumstances like this.

So this issue does not need hight proprity