Dunfell yocto support for NNStreamer.

[mritunjai-capgemini]

Neural Network Support kirkstone yocto (a new yocto release). Do we have dunfell yocto version support for NNStreamer. Kindly let us know how to download dunfell yocto version support for nnstreamer?

[taos-ci]

cibot: Thank you for posting issue #86. The person in charge will reply soon.

[myungjoo]

You may use dunfell branch: https://github.com/nnstreamer/meta-neural-network/commits/dunfell

[mritunjai-capgemini]

Thanks I have downloaded the dunfell version of the nnstreamer and have added this meta layer into my yocto build.

My build/bblayer.conf:
.....
${TOPDIR}/../meta-neural-network-dunfell
.....

My build/local.conf:
IMAGE_INSTALL_append = "
nnstreamer
nnstreamer-plugin-tensorflow
nnstreamer-plugin-tensorflow-lite
nnstreamer-plugin-tensorflow-lite-gst
nnstreamer-plugin-meta
"
I am getting below error while build this nnstreamer:
ERROR: Nothing RPROVIDES 'nnstreamer-plugins' (but /home/labuser/rzv_vlp_v3.0.2/build/../poky/meta/recipes-graphics/images/core-image-weston.bb RDEPENDS on or otherwise requires it)

Is above IMAGE_INSTALL_append variable correct or do i need to add some other package name to this variable. Kindly guide?

[mritunjai-capgemini]

@myungjoo Any help on above issue?

[myungjoo]

I don't see any string such as "nnstreamer-plugin-tensorflow" or "nnstreamer-plugin-meta" in my github repos. What are those? Whose codes are you referring to? I didn't write "core-image-weston.bb".

[mritunjai-capgemini]

I don't see any string such as "nnstreamer-plugin-tensorflow" or "nnstreamer-plugin-meta" in my github repos. What are those? Whose codes are you referring to? I didn't write "core-image-weston.bb".

Actually I am using dunfell version of renesas rz/v2l yocto build in-order to integrate this dunfel version of your's nnstreamer. So i am using "core-image-weston" target to build which will also buid the nnstreamer component in addition to other components.

[mritunjai-capgemini]

It was my ignorance to use "nnstreamer-plugin-tensorflow" or "nnstreamer-plugin-meta" strings in configuration files without looking at the recipes available in your github repo. Now i have modified the configuration files content to align with the recipes available for nnstreamer. Now it builds fine except for nntrainer. It throws me the following error:

**_Log data follows:
| DEBUG: Executing shell function do_configure
| NOTE: Executing meson -Dplatform=yocto -Denable-blas=true -Denable-cublas=false -Denable-app=false -Dinstall-app=false -Duse_gym=false -Denable-capi=true -Denable-test=false -Denable-logging=true -Denable-tizen-feature-check=true -Denable-nnstreamer-tensor-filter=false -Denable-nnstreamer-backbone=false -Denable-tflite-backbone=false -Denable-tflite-interpreter=false --bindir=/usr/lib64/nntrainer/bin ...
| The Meson build system
| Version: 0.53.2
| Source dir: /home/labuser/rzv_vlp_v3.0.2/build/tmp/work/aarch64-poky-linux/nntrainer/0.2.0+gitAUTOINC+ed18457393-r0/git
| Build dir: /home/labuser/rzv_vlp_v3.0.2/build/tmp/work/aarch64-poky-linux/nntrainer/0.2.0+gitAUTOINC+ed18457393-r0/build
| Build type: cross build
|
| meson.build:1:0: ERROR: Value "true" for combo option is not one of the choices. Possible choices are: "enabled", "disabled", "auto".
|
| A full log can be found at /home/labuser/rzv_vlp_v3.0.2/build/tmp/work/aarch64-poky-linux/nntrainer/0.2.0+gitAUTOINC+ed18457393-r0/build/meson-logs/meson-log.txt
| ERROR: meson failed
| WARNING: exit code 1 from a shell command.
| ERROR: Execution of '/home/labuser/rzv_vlp_v3.0.2/build/tmp/work/aarch64-poky-linux/nntrainer/0.2.0+gitAUTOINC+ed18457393-r0/temp/run.do_configure.1006245' failed with exit code 1
ERROR: Task (/home/labuser/rzv_vlp_v3.0.2/build/../meta-neural-network-dunfell/recipes-nntrainer/nntrainer/nntrainer_0.1.1.bb:do_configure) failed with exit code '1'
NOTE: Tasks Summary: Attempted 6175 tasks of which 6163 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
/home/labuser/rzv_vlp_v3.0.2/build/../meta-neural-network-dunfell/recipes-nntrainer/nntrainer/nntrainer_0.1.1.bb:do_configure
Summary: There were 7 WARNING messages shown.
Summary: There were 2 ERROR messages shown, returning a non-zero exit code._**

Let me know if i am missing anything here. All other recipes like nnstreamer, tensorflow-lite builds fine. Kindly help in resolving this issue.

[mritunjai-capgemini]

I am successfully able to integrate dunfell yocto version of your NNStreamer into my yocto which is dunfell based renesas yocto(rz/v2l vlp 3.0.2). Now while testing using below pipeline:

root@smarc-rzv2l:~# GST_DEBUG=3 gst-launch-1.0 v4l2src ! videoconvert ! videoscale ! video/x-raw,width=640,height=480,format=RGB,framerate=30/1 ! tee name=t t. ! queue leaky=2 max-size-buffers=2 ! videoscale ! video/x-raw,width=256,height=256,format=RGB ! tensor_converter ! tensor_transform mode=arithmetic option=typecast:float32,add:0.0,div:255.0 ! tensor_filter framework=tensorflow-lite model=v5_nano_unquantized.tflite ! tensor_decoder mode=bounding_boxes option1=yolov5 option2=v5_nano_label.txt option4=640:480 option5=256:256 ! compositor name=mix sink_0::zorder=2 sink_1::zorder=1 ! queue ! videoconvert ! timeoverlay ! omxh264enc ! mp4mux ! filesink location=test/xyz.mp4 -e t. ! queue leaky=2 max-size-buffers=10 ! mix.

I am getting below error:

** (gst-launch-1.0:319): WARNING **: 12:29:38.517: Cannot identify the given neural network framework, tensorflow-lite

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.757: gst_mini_object_unref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.757: gst_mini_object_ref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.757: gst_caps_can_intersect: assertion 'GST_IS_CAPS (caps1)' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.758: gst_mini_object_unref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.759: gst_mini_object_unref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.759: gst_mini_object_ref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.759: gst_mini_object_unref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.759: gst_mini_object_ref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.760: gst_caps_can_intersect: assertion 'GST_IS_CAPS (caps2)' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.760: gst_mini_object_unref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.760: gst_mini_object_unref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.760: gst_mini_object_ref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.760: gst_pad_template_new: assertion 'caps != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.760: gst_mini_object_unref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.760: gst_element_request_compatible_pad: assertion 'GST_IS_PAD_TEMPLATE (templ)' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.760: gst_object_unref: assertion 'object != NULL' failed
0:00:00.368240542 319 0xaaab0c882300 ERROR GST_PIPELINE grammar.y:740:gst_parse_perform_link: could not link tensortransform0 to tensorfilter0

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.761: gst_mini_object_unref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.761: gst_mini_object_ref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.761: gst_caps_can_intersect: assertion 'GST_IS_CAPS (caps2)' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.761: gst_mini_object_unref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.761: gst_mini_object_unref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.761: gst_mini_object_ref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.762: gst_pad_template_new: assertion 'caps != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.762: gst_mini_object_unref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.762: gst_element_request_compatible_pad: assertion 'GST_IS_PAD_TEMPLATE (templ)' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.762: gst_object_unref: assertion 'object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.762: gst_mini_object_unref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.762: gst_mini_object_ref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.763: gst_caps_can_intersect: assertion 'GST_IS_CAPS (caps1)' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.763: gst_mini_object_unref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.763: gst_mini_object_unref: assertion 'mini_object != NULL' failed

(gst-launch-1.0:319): GStreamer-CRITICAL **: 12:29:38.763: gst_mini_object_ref: assertion 'mini_object != NULL' failed
0:00:00.371510709 319 0xaaab0c882300 ERROR GST_PIPELINE grammar.y:740:gst_parse_perform_link: could not link tensorfilter0 to tensordecoder0

** (gst-launch-1.0:319): CRITICAL **: 12:29:38.764: bb_getOutCaps: assertion 'dim[0] == (data->labeldata.total_labels + YOLOV5_DETECTION_NUM_INFO)' failed

** (gst-launch-1.0:319): CRITICAL **: 12:29:38.765: bb_getOutCaps: assertion 'dim[0] == (data->labeldata.total_labels + YOLOV5_DETECTION_NUM_INFO)' failed
0:00:00.392302167 319 0xaaab0c882300 ERROR GST_PIPELINE grammar.y:740:gst_parse_perform_link: could not link omxh264enc-omxh264enc0 to mp4mux0
WARNING: erroneous pipeline: could not link tensortransform0 to tensorfilter0
root@smarc-rzv2l:~#

Can you guide why i am getting the above error.

[myungjoo]

What is the input tensor requirement and output tensor requirement of your model file, "v5_nano_unquantized.tflite"?

My first guess is dimension/type mismatch and tf-lite installation inconsistency.

I'd recommend to write a minimal pipeline that reproduces the same issue to pinpoint the issue.
E.g.,

videotestsrc ! videoconvert ! videoscale ! video/x-raw,.... ! tensor_converter ! ... ! tensor_filter ... ! file_sink location=...

Then, add elements one by one.

To ensure the installcation consistency, run

$ nnstreamer-check

[mritunjai-capgemini]

Thanks for your reply, sure i will check as per your suggestion.

My main concern is below error:
WARNING **: 12:29:38.517: Cannot identify the given neural network framework, tensorflow-lite

Why it cannot identify tensorflow-lite, i can confirm that tensorflow-lite plugin was built successfully.

Any method to check on the board to confirm if tensorflow-lite is installed on the board correctly?

[myungjoo]

My main concern is below error:
WARNING **: 12:29:38.517: Cannot identify the given neural network framework, tensorflow-lite

For that, nnstreamer-check will give some answers.
It could be the install location or the configuration (env-var or /etc/nnstreamer.ini)

Example:

mzx@kohaku:/source/AutoDrv/NNStreamer/$ nnstreamer-check
Cannot load nnstreamer plugin. Please check if nnstreamer is installed and GST_PLUGIN_PATH is properly configured.
mzx@kohaku:/source/AutoDrv/NNStreamer/$ export GST_PLUGIN_PATH=/source/AutoDrv/NNStreamer/build/gst
mzx@kohaku:/source/AutoDrv/NNStreamer/$ nnstreamer-check
NNStreamer version: 2.3.0
           loaded : TRUE
           path   : /source/AutoDrv/NNStreamer/build/gst/nnstreamer/libnnstreamer.so

NNStreamer configuration:
============================================================
Configuration Loaded: TRUE
Configuration file path: /etc/nnstreamer.ini
    Candidates: envvar(NNSTREAMER_CONF): (null)
                build-config: /usr/local/etc/nnstreamer.ini
                hard-coded: /etc/nnstreamer.ini
[Common]
  Enable envvar: TRUE
  Enable sym-linked subplugins: TRUE
[Filter]
  Filter paths from .ini: /source/AutoDrv/NNStreamer/build/ext/nnstreamer/tensor_filter
             from envvar: (null)
         from hard-coded: /usr/lib/nnstreamer/filters/

============================================================

** (nnstreamer-check:8094): CRITICAL **: 15:47:11.774: Cannot open tensorflow2-lite-custom(/source/AutoDrv/NNStreamer/build/ext/nnstreamer/tensor_filter/libnnstreamer_filter_tensorflow2-lite-custom.so) with error libtensorflow2-lite-custom.so: cannot open shared object file: No such file or directory.

NNStreamer registered sub-plugins:
============================================================

[Filter]
  tensorflow2-lite
    - NumThreads: Number of threads. Set 0 for default behaviors.
    - Delegate: TF-Lite delegation options: {'NNAPI', 'GPU', 'XNNPACK', 'External'}. Do not specify to disable delegation.
    - ExtDelegateLib: Path to external delegate shared library
    - ExtDelegateKeyVal: key/values pairs optional parameters for delegate. Format ExtDelegateKeyVal=key1#value1;key2#value2...

[Decoder]
  tensor_region
    - No custom property found
  octet_stream
    - No custom property found
  flexbuf
    - No custom property found
  protobuf
    - No custom property found
  direct_video
    - No custom property found
  image_labeling
    - No custom property found
  python3
    - No custom property found
  flatbuf
    - No custom property found
  image_segment
    - No custom property found
  bounding_boxes
    - No custom property found
  pose_estimation
    - No custom property found

[Conterver]
  flexbuf
    - No custom property found
  flatbuf
    - No custom property found
  protobuf
    - No custom property found
  python3
    - No custom property found

[Trainer]

============================================================
mzx@kohaku:/source/AutoDrv/NNStreamer/$