[LOGS] Extracting attributes from json in the UI

[nityanandagohain]

In the UI we render the log body as json if it's one. We can provide a way to convert them to fields as well.

There are two ways to do it.

• Do the extraction and materialization both at the clickhouse level.
• Perform extraction at collector level and materialize at clickhouse leve.

[nityanandagohain]

At clickhouse level

• There are limitation to the alter command https://clickhouse.com/docs/en/sql-reference/statements/alter/column#limitations
  • if a long SELECT is running at the time of the ALTER query, the ALTER query will wait for it to complete. At the same time, all new queries to the same table will wait while this ALTER is running.
  • when running ALTER for a Distributed table, you will also need to run ALTER for the tables on all remote servers.
• Since we already have parsing pipelines which works on expression language we will also need to do proper mapping of those.

[ankitnayan]

Should we show this as parsing pipeline then? Eventually we would be creating a parsing pipeline internally or should we create a separate API? We also need to handle same key in nested json issue in here, right?

[nityanandagohain]

So we can go we a default json parser pipeline.

This pipeline will also be manipulated by a separete UI which can be called from the logs explorer page. That way user can have a control of all the facets in one place as well