[Bug report] Table comment doesn't support UTF-8

[shaofengshi]

Version

main branch

Describe what's wrong

I have hive table, and I enter the comment with some Chinese character. After save the table, on UI, the comments just shows with "?", like this:

I checked the HTTP response, which just carries "?" in the json content.

Error message and/or stacktrace

No error.

How to reproduce

Enter some chinese character in a table's comment field, you will see it:

Additional context

No response

[shaofengshi]

A mysql table doesn't have this problem. So seems it is only related with Hive tables.

[yuqi1129]

Yeah, the data in MySQL attached to the Hive cluster is as follows:

mysql> SELECT PARAM_VALUE FROM TABLE_PARAMS WHERE PARAM_KEY = 'comment' AND TBL_ID IN (SELECT TBL_ID FROM TBLS WHERE TBL_NAME = 't1');
+-------------+
| PARAM_VALUE |
+-------------+
| ??          |
+-------------+
1 row in set (0.01 sec)

We need to change the charset of the MySQL cluster, but it does not work when I take try, Let me dive into it.

[shaofengshi]

Yeah, the data in MySQL attached to the Hive cluster is as follows:

mysql> SELECT PARAM_VALUE FROM TABLE_PARAMS WHERE PARAM_KEY = 'comment' AND TBL_ID IN (SELECT TBL_ID FROM TBLS WHERE TBL_NAME = 't1');
+-------------+
| PARAM_VALUE |
+-------------+
| ??          |
+-------------+
1 row in set (0.01 sec)

We need to change the charset of the MySQL cluster, but it does not work when I take try, Let me dive into it.

I also modified the connection string in hive-site.xml to "jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true&useSSL=false&characterEncoding=UTF-8", but that doesn't work.

[yuqi1129]

This is the limitation of Hive Metastoreat present. For MySQL, Hive metastore only supports latin1 and does not support UTF8, see https://issues.apache.org/jira/browse/HIVE-18083

[yuqi1129]

This problem lies in the storage itself and Gravitino server and web do not have character issues.