grpc-labview serializes strings using the ANSI code page, not UTF-8

[bkeryan]

LabVIEW strings are encoded using the system's ANSI code page. Protobuf strings are UTF-8. grpc-labview should translate strings between ANSI and UTF-8, but it doesn't.

Steps to reproduce

1. Install LabVIEW, InstrumentStudio, NI-DMM, and https://github.com/ni/measurement-plugin-labview
2. Use HWCU to add a simulated PXIe-1062Q and a PXIe-4080 named DMM1.
3. Open C:\Program Files\National Instruments\LabVIEW 2023\examples\Plug-Ins\Measurements\NI-DMM Measurement\NIDmmMeasurement\Get Measurement Details.vi and change the display name to use https://graphemica.com/%C3%A9:
   
4. Run C:\Program Files\National Instruments\LabVIEW 2023\examples\Plug-Ins\Measurements\NI-DMM Measurement\NIDmmMeasurement\Run Service.vi

Expected behavior

The service registers with the discovery service using a display name of NI-DMM Méasuremént (LabVIEW)

Actual behavior

Run Service.vi:

Error -1002 occurred at ni.measurementlink.discovery.v1.api.lvlib:DiscoveryService RegisterService.vi:2390001

Possible reason(s):

Exception was thrown by handler.


Complete call chain:
     ni.measurementlink.discovery.v1.api.lvlib:DiscoveryService RegisterService.vi:2390001
     NI Measurement Plug-In SDK.lvlib:Register to Discovery Service.vi
     NI Measurement Plug-In SDK.lvlib:Run Service.vi:3120001
     NIDmmMeasurement.lvlib:Run Service.vi

C:\ProgramData\National Instruments\Plug-Ins\Logs\DiscoveryService.txt:

===== 07/23/2025 14:27:38 ===== Information: Error reading message.
===== 07/23/2025 14:27:38 ===== Error: Error when executing service method '"RegisterService"'.
===== 07/23/2025 14:27:38 ===== Information: HTTP "POST" "/ni.measurementlink.discovery.v1.DiscoveryService/RegisterService" responded 200 in 0.5765 ms
===== 07/23/2025 14:27:38 ===== Information: Service unregistered successfully. RegistrationId: ""
===== 07/23/2025 14:27:38 ===== Information: HTTP "POST" "/ni.measurementlink.discovery.v1.DiscoveryService/UnregisterService" responded 200 in 0.1059 ms

Notes

'é' is 0xE9 in the Windows 1252 code page and U+00E9 in Unicode, but 0xE9 is not valid UTF-8. The UTF-8 encoding of U+00E9 is 0xC3 0xA9.

Reference: https://graphemica.com/%C3%A9

Wireshark confirms that grpc-labview is sending 0xE9 over the network without converting it to UTF-8:

AB#3207176

[bkeryan]

Further testing shows that ConvertSystemStringToUTF8 returns an empty string by nulling out the handle, and this is crashing.

>	labview_grpc_server.dll!grpc_labview::GetLVString(grpc_labview::LStr * * lvString) Line 192	C++
 	labview_grpc_server.dll!grpc_labview::MessageElementMetadata::MessageElementMetadata(grpc_labview::IMessageElementMetadataOwner * owner, grpc_labview::LVMessageElementMetadata * lvElement, int metadataVersion) Line 58	C++
 	labview_grpc_server.dll!std::_Construct_in_place<grpc_labview::MessageElementMetadata,grpc_labview::IMessageElementMetadataOwner * &,grpc_labview::LVMessageElementMetadata * &,int &>(grpc_labview::MessageElementMetadata & _Obj, grpc_labview::IMessageElementMetadataOwner * & <_Args_0>, grpc_labview::LVMessageElementMetadata * & <_Args_1>, int & <_Args_2>) Line 476	C++
 	labview_grpc_server.dll!std::_Ref_count_obj2<grpc_labview::MessageElementMetadata>::_Ref_count_obj2<grpc_labview::MessageElementMetadata><grpc_labview::IMessageElementMetadataOwner * &,grpc_labview::LVMessageElementMetadata * &,int &>(grpc_labview::IMessageElementMetadataOwner * & <_Args_0>, grpc_labview::LVMessageElementMetadata * & <_Args_1>, int & <_Args_2>) Line 2100	C++
 	labview_grpc_server.dll!std::make_shared<grpc_labview::MessageElementMetadata,grpc_labview::IMessageElementMetadataOwner * &,grpc_labview::LVMessageElementMetadata * &,int &>(grpc_labview::IMessageElementMetadataOwner * & <_Args_0>, grpc_labview::LVMessageElementMetadata * & <_Args_1>, int & <_Args_2>) Line 2913	C++
 	labview_grpc_server.dll!grpc_labview::MessageMetadata::InitializeElements(grpc_labview::IMessageElementMetadataOwner * metadataOwner, grpc_labview::LVMessageElementMetadata * lvElement, int elementCount, int metadataVersion) Line 130	C++
 	labview_grpc_server.dll!grpc_labview::MessageMetadata::MessageMetadata(grpc_labview::IMessageElementMetadataOwner * metadataOwner, grpc_labview::LVMessageMetadata2 * lvMetadata) Line 120	C++
 	labview_grpc_server.dll!std::_Construct_in_place<grpc_labview::MessageMetadata,grpc_labview::MessageElementMetadataOwner *,grpc_labview::LVMessageMetadata2 * &>(grpc_labview::MessageMetadata & _Obj, grpc_labview::MessageElementMetadataOwner * && <_Args_0>, grpc_labview::LVMessageMetadata2 * & <_Args_1>) Line 476	C++
 	labview_grpc_server.dll!std::_Ref_count_obj2<grpc_labview::MessageMetadata>::_Ref_count_obj2<grpc_labview::MessageMetadata><grpc_labview::MessageElementMetadataOwner *,grpc_labview::LVMessageMetadata2 * &>(grpc_labview::MessageElementMetadataOwner * && <_Args_0>, grpc_labview::LVMessageMetadata2 * & <_Args_1>) Line 2100	C++
 	labview_grpc_server.dll!std::make_shared<grpc_labview::MessageMetadata,grpc_labview::MessageElementMetadataOwner *,grpc_labview::LVMessageMetadata2 * &>(grpc_labview::MessageElementMetadataOwner * && <_Args_0>, grpc_labview::LVMessageMetadata2 * & <_Args_1>) Line 2913	C++
 	labview_grpc_server.dll!RegisterMessageMetadata2(grpc_labview::gRPCid * * id, grpc_labview::LVMessageMetadata2 * lvMetadata) Line 244	C++
 	00000243ea5ee3cc()	Unknown
 	LabVIEW.exe!DoCaughtExtFuncCall(long(*)(DSHeader *) funcPtr, DSHeader * dsp) Line 106	C++
 	LabVIEW.exe!ExtFuncWrapper(long(*)(DSHeader *) funcPtr, void * pSetTLSIndex, long bExcludeTryCatch, DSHeader * dsp, long qeOffset) Line 232	C++
 	00000243ea5ed619()	Unknown

[jasonmreding]

Further testing shows that ConvertSystemStringToUTF8 returns an empty string by nulling out the handle, and this is crashing.

Hmm... when I was reviewing this, I thought it was only getting used when copying around message data on the wire to/from LV. In this call stack, it is actually reading metadata produced from compiling the proto file. I wonder if we should be more deliberate about when we try to perform the translation to/from UTF8 so that it only applies to string message data sent across the wire. Then again, maybe it doesn't matter much in the end either way.

For the metadata, I suspect it doesn't really matter other than some performance overhead since I think that should be limited to ASCII characters anyway. However, I'm not as certain about the string used to represent the protobuf descriptor for reflection purposes. I think that is just using the string as a binary string and not something that should be interpreted as UTF8.

[bkeryan]

@jasonmreding Protobuf distinguishes between string and bytes. Does grpc-labview?

I'm testing the original use case from AB#3163834 and I'm getting unexpected field ids in the measurement plug-in response, so it's plausible that this may be affecting more than just strings.

If grpc-labview is using strings for binary data, then I will revert this PR for now and try again, this time with a feature toggle.

[jasonmreding]

@jasonmreding Protobuf distinguishes between string and bytes. Does grpc-labview?

I guess it depends on what you mean. Both string and bytes protobuf types are supported so from that perspective it does distinguish between the two. However, in addition to integrating with the grpc stack, the library also provides C interfaces between LV code and the C++ code which interfaces with the grpc stack. There are several scenarios where strings are used here:

1. User facing LV APIs the user writes their program against. In this case the strings are created in LV and so use whatever string encoding LV uses.
2. Internal metadata related APIs. These APIs are used by the code generator where there are shared LV/C++ structs defined for passing data between the environments. When the user generates code from a proto file, the LV generator passes the proto file to the C++ code for parsing, and it returns a bunch of metadata about the services/messages via clusters which map to the C++ structs. In this case, the strings match message names, field names, etc. from the proto file. I'm not sure what encoding is technically used here, but I imagine it is always ANSI characters so it probably doesn't really matter.
3. Reflection related APIs. This is mostly internal APIs used to extract the protobuf descriptor string produced by the protobuf compiler. Internally, grpc presents this as std::string, but I would imagine this is a straight byte buffer without any particular character encoding provided.

I think some/all of the above will end up using the GetLVString and SetLVString, and it probably isn't appropriate for it to apply utf8 encoding translation as part of the conversion in all cases. Maybe the best thing would be to add overloads or optional parameters for encoding to be applied as part of the conversion and then explicitly add the encoding in the places where we are reading/writing data from the wire. This is probably going to be in ClusterDataCopier and few other spots dealing with packing/unpacking Any data.

[bkeryan]

Reverting: #466

[bkeryan]

Besides descriptor serialization, handling of "bytes" (e.g. ClusterDataCopier::CopyBytesToCluster) needs to be distinct from handling of strings. This seems to be what causes measurement services to fail with the UTF-8 string support.