Important
The materials in this repository are relevant to the ucoming NavSuite 3.15 release (UCOM Version 1).
This repository contains the code to build and install the UCOM Decoder Library, as well as an example tool which can convert UCOM data from UDP stream or file to CSV file.
This project makes use of the following third-party open-source libraries:
- nlohmann/json for JSON parsing
- pybind11 for C++ / Python bindings
See the relevant sub-folder for their licences.
git clone --recursive https://github.com/OxfordTechnicalSolutions/ucom-decoder.gitThe repository must be cloned recursively as the UCOM decoder uses a sub-module for the JSON parser.
- UCOM Decoder
- Tests
- Decoding UCOM
The project is set up to allow cross-platform build (Windows / Linux) using Visual Studio 2022. The Linux build would most commonly use WSL2, but it is possible to use other remote hosts or build in Linux directly.
To start, open the project folder in Visual Studio. VS should automatically detect the CMake files and make the configurations available in the menu.
Select the host, e.g.: Local Machine for Windows build WSL: Ubuntu for Linux build
Select the build configuration, e.g.: x64 Debug
Select the 'Startup Item', e.g.: ucom_decoder.exe
Run the selected configuration
mkdir build && cd build
cmake ..
cmake --build .To build and install the library, replace cmake --build . with
cmake --build . --target installIf there are permission errors, re-run the above command as root with sudo.
The executable is found in the build/ucom_to_csv directory named ucom_to_csv.
Running UCOM to CSV with no arguments produces the following output:
Usage: ucom_to_csv [options] -u <.dbu filename>
Options:
-h Help - displays this message
-f <input file> Extract data from a file instead of UDP stream
-m <id1 id2 id3 ...> Message IDs to process, e.g. 0 1 3
-c <packets> Number of packets to process (UDP stream only)
-t <duration> Maximum capture duration in seconds (UDP stream only)
-i <address> Source IP address (UDP stream only)
-o <output file> Output file prefix (default is output_)
-a Disable user-abort
The example_data directory contains:
- UCOM file, logged from an OXTS INS
- .cfg file, showing how the unit is configured
- mobile.dbu file (in dbu sub-directory) - contains message definitions which are parsed by the decoder. This allows the decoder flexibility to decode new messages when provided with updated definitions.
To use this data with the UCOM to CSV tool:
./ucom_to_csv -u ../../example_data/dbu/mobile.dbu -f ../../example_data/input.ucomThere is a Python version of the UCOM decoder SDK.
The code has been tested with Python 3.12. It may work with earlier versions, but if any issues with package dependencies are encountered then it is recommended to upgrade to at least Python 3.12.
- Create a working folder and clone the UCOM_decoder repository:
mkdir ucompy_test
cd ucompy_test
git clone https://github.com/OxfordTechnicalSolutions/ucom-decoder.git --recursive ucom_decoder- Create Python virtual environment:
NB Use the appropriate command (instead of python) to invoke the Python interpreter on your system, e.g. py, python, python3.
python -m venv venv[Windows Command Prompt]
venv\Scripts\activate.bat[Windows Powershell]
venv\Scripts\Activate.ps1[Linux]
source venv/bin/activate- Install the oxts.ucompy module:
python -m pip install ucom_decoder/ucom_decoder_py- Test the installation:
a. Start Python interactive shell
pythonEnter the text following the prompts (>>>) line by line. You should see the output shown:
>>> from oxts.ucompy import UcomDbu
>>> u = UcomDbu("ucom_decoder/example_data/dbu/oxts.dbu")
>>> u.get_valid()
True
>>> u.get_filename()
'ucom_decoder/example_data/dbu/oxts.dbu'
>>> quit()b. Run the ucom_to_csv example:
ucom_decoder\ucom_decoder_py\examples\ucom_to_csv\test.bat- Uninstall:
python -m pip uninstall ucompyTo run all of the automated tests, first build ucom_to_csv and then from the ucom_decoder/test/ folder, run:
Windows
.\run_tests.batLinux
./run_tests.shThe automated unit tests run on the Python bindings to simultaneously test the bindings and the underlying (bound) C++ code.
There is also a test of the overall (C++) decoder functionality using automatically-generated UCOM data. A range of known values are encoded into UCOM packets. These packets are then decoded (using ucom_to_csv) and the extracted values are compared with the original values.
The Python unit tests can be run on their own by changing to the ucom_decoder_py/tests folder and running:
Windows
.\run_tests.batLinux
./run_tests.shUCOM provides the facility to customise the data that is output by an OXTS INS by using user-defined messages. These user-defined messages are specified in a mobile.dbu file. For ease of explanation, hereafter the term DBU will be used to refer to .dbu files, such as oxts.dbu and mobile.dbu, or their content.
UCOM data is sent in the form of packets that are made up of:
- 16-byte header
- Variable length payload
- CRC integrity check (32-bit)
- Bytes 14-15 : Payload length in bytes
The data associated with each signal is contained in the payload as a sequence of bytes whose layout is determined by the messages defined in the DBU.
Each message contains an array of signals (named 'SignalsInMessage' in the DBU), and each signal contains a number of properties (key : value pairs). One of these properties is DataType. When decoding UCOM DataType is used to determine the size in bytes of the signal data in the payload.
Example:
[DBU excerpt]
"SignalsInMessage": [
{
"SourceID": "TIME",
"SignalID": "GNSST",
"Unit": "s",
"ScaleFactor": 1,
"Offset": 0,
"DataType": "S64"
},
{
"SourceID": "BNS_SDN",
"SignalID": "Lat",
"Unit": "deg",
"ScaleFactor": 1,
"Offset": 0,
"DataType": "F64"
},
{
"SourceID": "BNS_SDN",
"SignalID": "Lon",
"Unit": "deg",
"ScaleFactor": 1,
"Offset": 0,
"DataType": "F64"
},
These three signals are each 64 bits - or 8 bytes - long (as are the vast majority of the signals), so their representation in the payload will be as follows:
A0 A1 A2 A3 A4 A5 A6 A7 B0 B1 B2 B3 B4 B5 B6 B7 C0 C1 C2 C3 C4 C5 C6 C7
where A0 ... A7, B0 ... B7, C0 ... C7 are the bytes representing the three messages A = GNSST, B = Lat, C = Lon above.
If message A is 8 bytes, message B is 1 byte and message C is 4 bytes:
A0 A1 A2 A3 A4 A5 A6 A7 B0 C0 C1 C2 C3
For full details of the UCOM protocol see the UCOM manual.
The steps required to decode UCOM are:
- Extract message and signal information from the DBU
- Read UCOM data (from file, UDP etc.)
a. Decode the header b. Decode the payload c. Calculate the CRC - Repeat from 2. as required
This example code demonstrates usage of the UCOM decoder SDK to decode UCOM data and write it to .csv files.
Clone repo (see above)
Open a 'Developer Command Prompt for VS'
Navigate to the UCOM_decoder folder and then build
mkdir build && cd build
cmake ..
cmake --build . --config ReleaseStart capturing UCOM packets from UDP:
cd ..
mkdir test && cd test
..\build\ucom_to_csv\Release\ucom_to_csv.exe -u ..\example_data\dbu\oxts.dbu -i anyCalling ucom_to_csv.exe with no arguments (or with -h) will display the options:
Usage: ucom_to_csv [options] -u <.dbu filename>
Options:
-h Help - displays this message
-f <input file> Extract data from a file instead of UDP stream
-m <id1 id2 id3 ...> Message IDs to process, e.g. 0 1 3
-c <packets> Number of packets to process (UDP stream only)
-t <duration> Maximum capture duration in seconds (UDP stream only)
-i <address> Source IP address (UDP stream only)
-o <output file> Output file prefix (default is output_)
-a Disable user-abort
The -u argument is required as it specifies the DBU that is needed for decoding the UCOM data.
Usually, when capturing packets from UDP, you will specify the source IP address using the '-i' flag, e.g. '-i 192.168.1.100' will only capture packets originating from the INS with that IP address. If there is only one INS connected to your network, then you can specify 'any' and there will be no IP filtering, i.e. use '-i any'.
This class accepts a DBU filename as an argument to its constructor and parses the JSON contained in the DBU to generate collections of UcomMessages and UcomSignals.
- UcomDbu(std::string filename)
- Constructs a UcomDbu instance from a DBU file . The file must be valid JSON conforming to the DBU schema.
- std::string get_filename()
- Gets the filename of the DBU.
- bool get_valid()
- Gets the 'valid' status of the instance. Returns true if the DBU file was parsed without error; false otherwise.
- bool message_id_exists(uint16_t message_id)
- Returns true if exists in the collection of messages in the DBU; false otherwise.
- std::map<uint16_t, UcomMessage>& get_messages()
- Gets a collection containing all of the UCOM message definitions in the DBU, stored as key : value pairs whose key is the message ID.
- std::list<uint16_t>& get_message_ids()
- Gets a collection containing the message IDs of all of the UCOM message definitions in the DBU.
- const UcomMessage& get_message(int id)
- Gets the UcomMessage whose ID is <id>
- const std::vector<ucom_signal_ptr_t> &get_signals(uint16_t message_id)
- Gets a collection containing all of the signals in the UcomMessage whose ID is <message_id>.
- static OxTS::Enum::BASIC_TYPE get_data_type(const std::string& data_type)
- Gets the OxTS::Enum::BASIC_TYPE from the string representation of the type.
This class decodes a UCOM packet and allows read-access to the signal values contained within. To decode a UCOM packet, a corresponding valid UcomDbu instance is required to provide the necessary signal layout information.
- UcomData(const uint8_t* data, int size, UcomDbu& dbu)
- Constructs a UcomData instance from the byte array passed as the <data> argument, whose length in bytes is contained in <size>. <dbu> is a reference to a valid UcomDbu instance.
- static const int peek(const uint8_t* data, int max_size, bool& need_more_data)
- Inspect <data> to determine if it contains a candidate UCOM packet.
Returns:- -1 : if no candidate found or if potential candidate found but more data is needed (need_more_data is true)
- packet length : if candidate found
- const std::string get_csv() const
- Gets the signal values as a comma-separated string
- std::string to_string()
- Gets a string representation of the instance, consisting of message ID, message version and payload length
- uint16_t get_message_id()
- Gets the message ID
- uint16_t get_message_version()
- Gets the message version
- uint8_t get_time_frame()
- Gets the time frame of the 'Arbitrary Time' value
- int64_t get_arbitrary_time()
- Gets the 'Arbitrary Time' value
- uint16_t get_payload_length()
- Gets the payload length (in bytes)
- size_t get_signal_count()
- Gets the signal count
- uint32_t get_calc_crc()
- Gets the calculated CRC
- bool get_valid()
- Gets the 'valid' status of the signal. Returns true if the UCOM packet has been successfully decoded; false otherwise
- bool get(std::string signal_id, UcomDbu& dbu, double &value)
- Gets the value of the signal whose ID is <signal_id> as a double. The value is assigned to the argument <value>. Returns true if the signal value is successfully assigned; false otherwise (e.g. if the signal ID doesn't exist)
Represents a UCOM message. Allows read-access to the signals contained within. Derives from json class.
- UcomMessage(json message)
- Constructs a UcomMessage instance from a JSON representation (usually part of a DBU)
- bool is_valid()
- Returns true if the UcomMessage is valid; false otherwise
- int get_id()
- Gets the ID of the message
- std::string get_header()
- Gets a comma-separated string of the names of the signals contained in the message (mainly intended for use when generating CSV output). The string is prepended with 'Time (<MessageTiming>)', representing the 'Arbitrary Time' time frame, e.g. 'Time (SDN)'
- size_t get_signal_count()
- Gets the number of signals contained in the message
- const std::vector<ucom_signal_ptr_t> &get_signals()
- Gets a collection of (smart, shared) pointers to the signals contained in the message
- const ucom_signal_ptr_t get_signal(std::string id)
- Gets a (smart, shared) pointer to the signal whose ID is <id>
- int get_signal_index(std::string id)
- Gets the zero-based index into the signals collection of the signal whose ID is <id>
Represents a UcomSignal. Contains the meta-data required to decode a signal from a data stream
- UcomSignal(json signal)
- Constructs a UcomSignal instance from a JSON representation (usually part of a DBU)
- UcomSignal(std::string signal_id, UcomSignal::SignalType signal_type)
- Constructs a UcomSignal instance from the signal ID and signal type
- std::string get_signal_id()
- Gets the signal ID
- const OxTS::Enum::BASIC_TYPE get_data_type()
- Gets the signal data type
This class accepts a DBU filename as an argument to its constructor and parses the JSON contained in the DBU to generate collections of UcomMessages and UcomSignals.
- UcomDbu(filename: str) -> UcomDbu
- Constructs a UcomDbu instance from a DBU file . The file must be valid JSON conforming to the DBU schema.
- get_filename() -> str
- Gets the filename of the DBU.
- get_valid() -> bool
- Gets the 'valid' status of the instance. Returns True if the DBU file was parsed without error; False otherwise.
- message_id_exists(message_id: int) -> bool
- Returns True if exists in the collection of messages in the DBU; False otherwise.
- get_messages() -> dict[int, UcomMessage]
- Gets a dictionary containing all of the UCOM message definitions in the DBU, stored as key : value pairs whose key is the message ID.
- get_message_ids() -> list[int]
- Gets a list containing the message IDs of all of the UCOM message definitions in the DBU.
- get_message(id: int) -> UcomMessage
- Gets the UcomMessage whose ID is <id>
- get_signals(message_id: int) -> list[UcomSignal]
- Gets a list containing all of the signals in the UcomMessage whose ID is <message_id>.
- get_data_type(data_type: str) -> OxTS::Enum::BASIC_TYPE
- Gets the OxTS::Enum::BASIC_TYPE from the string representation of the type.
This class decodes a UCOM packet and allows read-access to the signal values contained within. To decode a UCOM packet, a corresponding valid UcomDbu instance is required to provide the necessary signal layout information.
- UcomData(data: str, size: int, dbu: UcomDbu) -> UcomData
- Constructs a UcomData instance from the byte array passed as the <data> argument, whose length in bytes is contained in <size>. <dbu> is a valid UcomDbu instance.
- peek(data: str, max_size: int) -> tuple[int,
- Inspect <data> to determine if it contains a candidate UCOM packet.
Returns:- [-1, False] : if no candidate found
- [-1, True] : if potential candidate found but more data is needed
- [packet length, False] : if candidate found
- get_csv() -> str
- Gets the signal values as a comma-separated string
- to_string() -> str
- Gets a string representation of the instance, consisting of message ID, message version and payload length
- get_message_id() -> int
- Gets the message ID
- get_message_version() -> int
- Gets the message version
- get_time_frame() -> int
- Gets the time frame of the 'Arbitrary Time' value
- get_arbitrary_time() -> int
- Gets the 'Arbitrary Time' value
- get_payload_length() -> int
- Gets the payload length (in bytes)
- get_signal_count() -> int
- Gets the signal count
- get_calc_crc() -> int
- Gets the calculated CRC
- get_valid() -> bool
- Gets the 'valid' status of the signal. Returns True if the UCOM packet has been successfully decoded; False otherwise
- get(signal_id: str, dbu: UcomDbu) -> tuple[bool, float]
- Gets the value of the signal whose ID is <signal_id>. Returns [True, value] if the signal ID exists and value contains valid data; [False, 0.0] otherwise
Represents a UCOM message. Allows read-access to the signals contained within. Derives from json class.
- UcomMessage(message: json) -> UcomMessage
- Constructs a UcomMessage instance from a JSON representation (usually part of a DBU)
- is_valid() -> bool
- Returns True if the UcomMessage is valid; False otherwise
- get_id() -> int
- Gets the ID of the message
- get_header() -> str
- Gets a comma-separated string of the names of the signals contained in the message (mainly intended for use when generating CSV output). The string is prepended with 'Time (<MessageTiming>)', representing the 'Arbitrary Time' time frame, e.g. 'Time (SDN)'
- get_signal_count() -> int
- Gets the number of signals contained in the message
- get_signals() -> list[UcomSignal]
- Gets a list of signals contained in the message
- get_signal(id: str) -> UcomSignal
- Gets the signal whose ID is <id>
- get_signal_index(id: str) -> int
- Gets the zero-based index into the signals collection of the signal whose ID is <id>
Represents a UcomSignal. Contains the meta-data required to decode a signal from a data stream
- UcomSignal(signal: json) -> UcomSignal
- Constructs a UcomSignal instance from a JSON representation (usually part of a DBU)
- get_signal_id() -> str
- Gets the signal ID
- get_data_type() -> OxTS::Enum::BASIC_TYPE
- Gets the signal data type
Warning
The example pictures may not reflect the current state of the messages within oxts.dbu file however the code examples should still work and you should see data reflective of the provided DBU file. If you do not, please raise an issue or see OXTS support.
#include <iostream>
// Include the header file for UcomDbu
#include "ucom\ucom_dbu.hpp"
int main()
{
// Parse the DBU
UcomDbu dbu("..\\..\\oxts.dbu");
std::cout << (dbu.get_valid() ? "DBU valid" : "DBU invalid") << std::endl;
// Display the message IDs defined in the DBU
for (auto id : dbu.get_message_ids())
std::cout << id << ' ';
std::cout << std::endl;
// Display the header of message ID 0
std::cout << "Header: " << dbu.get_message(0).get_header() << std::endl;
}
Generates the following output:
#include <iostream>
#include "ucom\ucom_dbu.hpp"
#include "ucom\ucom_data.hpp"
#include "ucom\example_ucom_data.hpp"
int main(int argc, char *argv[])
{
// Parse the DBU
UcomDbu dbu("..\\..\\oxts.dbu");
// Decode the data
UcomData data(_data, sizeof(_data), dbu);
std::cout << "UcomData: " << (data.get_valid() ? "valid" : "invalid") << std::endl;
double value;
std::cout << "Values: " << std::endl;
// Iterate over the signal collection and display the values
for (auto signal : dbu.get_message(data.get_message_id()).get_signals())
{
std::string signal_id = signal->get_signal_id();
if (data.get(signal_id, dbu, value))
std::cout << signal_id << ": " << value << '\n';
}
}
Generates the following output:
from oxts.ucompy import UcomDbu
# Parse the DBU
dbu = UcomDbu("ucom_decoder/oxts.dbu")
print("DBU valid" if dbu.get_valid() else "DBU invalid")
# Display the message IDs defined in the DBU
for id in dbu.get_message_ids():
print(f"{id} ")
# Display the header of message ID 0
print(f"Header: {dbu.get_message(0).get_header()}")Generates the following output:
from oxts.ucompy import UcomDbu, UcomData
_data = (
b'\x55\x4d\x00\x00\x01\xcc\xb2\x9a\xd0\x3c\x00\x00\x00\x00\xd0\x00' +
b'\xc4\x85\x05\x79\x00\x00\x00\x00\x34\x33\x33\x33\x33\x33\x26\x40' +
b'\x9a\x99\x99\x99\x99\x19\x35\x40\x9a\x99\x99\x99\x99\x19\x3f\x40' +
b'\xcd\xcc\xcc\xcc\xcc\x8c\x44\x40\xcd\xcc\xcc\xcc\xcc\x8c\x49\x40' +
b'\xcd\xcc\xcc\xcc\xcc\x8c\x4e\x40\x67\x66\x66\x66\x66\xc6\x51\x40' +
b'\x66\x66\x66\x66\x66\x46\x54\x40\x66\x66\x66\x66\x66\xc6\x56\x40' +
b'\x66\x66\x66\x66\x66\x46\x59\x40\x66\x66\x66\x66\x66\xc6\x5b\x40' +
b'\x66\x66\x66\x66\x66\x46\x5e\x40\x33\x33\x33\x33\x33\x63\x60\x40' +
b'\x33\x33\x33\x33\x33\xa3\x61\x40\x33\x33\x33\x33\x33\xe3\x62\x40' +
b'\x33\x33\x33\x33\x33\x23\x64\x40\x33\x33\x33\x33\x33\x63\x65\x40' +
b'\x33\x33\x33\x33\x33\xa3\x66\x40\x33\x33\x33\x33\x33\xe3\x67\x40' +
b'\x33\x33\x33\x33\x33\x23\x69\x40\x33\x33\x33\x33\x33\x63\x6a\x40' +
b'\x33\x33\x33\x33\x33\xa3\x6b\x40\x33\x33\x33\x33\x33\xe3\x6c\x40' +
b'\x33\x33\x33\x33\x33\x23\x6e\x40\x33\x33\x33\x33\x33\x63\x6f\x40' +
b'\xb7\xef\x80\xa4')
# Parse the DBU
dbu = UcomDbu("ucom_decoder/oxts.dbu")
# Decode the data
data = UcomData(_data, len(_data), dbu)
print(f"UcomData: {'valid' if data.get_valid() else 'invalid'}")
# double value;
print(f"Signal values in message {data.get_message_id()}: ")
# Iterate over the signal collection and display the values
for signal in dbu.get_message(data.get_message_id()).get_signals():
signal_id = signal.get_signal_id()
success, value = data.get(signal_id, dbu)
print(f"{signal_id} : {value:.2f}")Generates the following output:
These instructions are for Visual Studio 2022, and assume that you have already cloned and built UCOM decoder in Windows.
- Open Visual Studio
- Create new project
- Select:
- C++
- Windows
- Console - Select the required project type, e.g.
Console App, clickNext - Select the location and enter a project name
- Click
Create
- Right-click project folder in
Solution Explorerand selectProperties. - Select
Configuration Properties | C/C++ | Generaland inAdditional Include Directoriesadd the paths toucom_decoder\includeanducom_decoder\json\include. - Select
Configuration Properties | Linker | Generaland inAdditional Library Directoriesadd the path to the directory that contains ucom_decoder.lib (something likebuild\ucom_decoder\Debugfor a Debug build). - Select
Configuration Properties | Linker | Inputand inAdditional Dependenciesadducom_decoder.lib.
Add the required include files, and code. For example:
#include <iostream>
#include "ucom\ucom_dbu.hpp"
#include "ucom\ucom_data.hpp"
#include "ucom\example_ucom_data.hpp"
int main()
{
std::cout << "Ucom Decoder Example\n";
UcomDbu dbu("..\\..\\ucom_decoder\\example_data\\dbu\\oxts.dbu");
if (!dbu.get_valid())
{
std::cout << "Error getting DBU, exiting..." << std::endl;
return 1;
}
UcomData data(_data, sizeof(_data), dbu);
std::cout << "Message ID: " << data.get_message_id() << "\n";
auto msg = dbu.get_message(data.get_message_id());
double value = 0;
for (auto signal : msg.get_signals())
{
std::cout << signal->get_signal_id() << ": ";
if (data.get(signal->get_signal_id(), dbu, value))
std::cout << std::setprecision(2) << std::fixed << value << '\n';
else
std::cout << "not found\n";
}
return 0;
}Build and run:
These instructions assume that you have already cloned, built and installed UCOM decoder in Linux.
cmake_minimum_required(VERSION 3.10)
project(ucom_decoder_example)
add_executable(${PROJECT_NAME} src/main.cpp)
target_include_directories(${PROJECT_NAME}
PRIVATE ${CMAKE_SOURCE_DIR}/../ucom_decoder/ucom_decoder/include
PRIVATE ${CMAKE_SOURCE_DIR}/../ucom_decoder/ucom_decoder/json/include
)
target_link_libraries(${PROJECT_NAME} ucom_decoder)#include <iostream>
#include "ucom/ucom_dbu.hpp"
#include "ucom/ucom_data.hpp"
#include "ucom/example_ucom_data.hpp"
int main()
{
std::cout << "Ucom Decoder Example\n";
UcomDbu dbu("../../ucom_decoder/example_data/dbu/oxts.dbu");
if (!dbu.get_valid())
{
std::cout << "Error getting DBU, exiting..." << std::endl;
return 1;
}
UcomData data(_data, sizeof(_data), dbu);
std::cout << "Message ID: " << data.get_message_id() << "\n";
auto msg = dbu.get_message(data.get_message_id());
double value = 0;
for (auto signal : msg.get_signals())
{
std::cout << signal->get_signal_id() << ": ";
if (data.get(signal->get_signal_id(), dbu, value))
std::cout << std::setprecision(2) << std::fixed << value << '\n';
else
std::cout << "not found\n";
}
}mkdir build && cd build
cmake ..
cmake --build .
./ucom_decoder_exampleOutput will be the same as for Visual Studio / Windows above.