Basler Camera with Pylon SDK Errors

[lumyus]

Hey,

I built Fictrac with Cmake and the following command: cmake -D CMAKE_TOOLCHAIN_FILE=/home/vcpkg/scripts/buildsystems/vcpkg.cmake -D BASLER_USB3=ON -D BASLER_DIR="\opt\pylon5" ..
I am using PylonSDK 5 (I also tried PylonSDK 6 but the build would not work) and we are running everything on Ubuntu.

This is the error I get when running Fictrac:

Looking for config file: config.txt ..
Config file parsed (21 key/value pairs).
[ INFO:0] global /home/nely/opencv/modules/videoio/src/videoio_registry.cpp (187) VideoBackendRegistry VIDEOIO: Enabled backends(7, sorted by priority): FFMPEG(1000); GSTREAMER(990); INTEL_MFX(980); V4L2(970); CV_IMAGES(960); CV_MJPEG(950); FIREWIRE(940)
[ INFO:0] global /home/nely/opencv/modules/videoio/src/backend_plugin.cpp (248) getPluginCandidates VideoIO pluigin (INTEL_MFX): glob is 'libopencv_videoio_intel_mfx*.so', 1 location(s)
[ INFO:0] global /home/nely/opencv/modules/videoio/src/backend_plugin.cpp (256) getPluginCandidates - /usr/local/lib: 0
[ INFO:0] global /home/nely/opencv/modules/videoio/src/backend_plugin.cpp (259) getPluginCandidates Found 0 plugin(s) for INTEL_MFX
[ INFO:0] global /home/nely/opencv/modules/videoio/src/cap_images.cpp (282) icvExtractPattern Pattern: %08d @ 40022761
Could not interpret source type (40022761)!
Error! Could not open input frame source (40022761)!
Error loading parameters from specified config file (config.txt)!

I also tried to change the src_fn from 40022761 to 0 but the error stayed the same. (40022761 is the camera id in PylonViewerApp)
I also tried to build without the Pylon SDK (cmake -D CMAKE_TOOLCHAIN_FILE=/home/vcpkg/scripts/buildsystems/vcpkg.cmake) but the error stayed the same.
Somehow we are not getting any output with Fictrac, neither with the ConfigGui.
The config.txt used is the same as the one provided in the /sample folder. (Except src_fn has been changed)
Any idea on what is going wrong?

Thanks for your help!

[rjdmoore]

Hey, ok so camera id is probably a poor name. It's actually the USB id (nth device on bus). If you only have one usb device connected, src_fn should be 0. If you have more than one device, src_fn might be 1, 2, ... depending on device enumeration.

If you've tried setting src_fn to 0, 1, 2.. and still not getting it to work, then this suggests something else is wrong. You should make sure the camera is properly configured and streaming before starting fictrac. You should probably be also close basler viewer app prior to starting fictrac.

[lumyus]

Thanks for getting back to me. We tried 0,1,2 etc but had no success.
You mentioned "properly configured and streaming": We configure the camera as always with the Basler Viewer App and then open the stream (Red to Green button) and close the Basler Viewer App subsequently. The issue remains.
Is it necessary to build Fictrac with BASLER_USB3=ON in order to use the Basler Cameras? Or could we also run without the Pylon SDK?
If the Pylon SDK is necessary, which version did you test it with (5 or 6)?

[rjdmoore]

Hmm, ok. Basler cameras are a little poorly tested unfortunately, so there could be an API change in the newer versions that's causing an issue. Basler code was originally added June 2019 I think, so that's probably the last version that was tested properly.

I will try and get hold of a basler camera to test with, but if there's API issues you should be seeing other error messages, at least in the console.

In the meantime you could try posting in the forum (Reddit) to see if anyone has more recent experience with basler and fictrac. You could also try contacting directly those that have posted basler questions in the past.

You should also try just using the basler camera without the SDK. Just build without those options. You probably won't be able to use advanced camera modes, so try setting to some default standard res/FPS/bitdepth first and see if it works! You'll still need to find which USB id your camera is using (should be zero if you have no other cameras).

[lumyus]

Okey so I did the following and it still doesn't work:
I built Fictrac with no SDK specified. (Regular build command). Then I called lsusb which shows:

_**Bus 014 Device 005: ID 2676:ba02 Basler AG ace**
Bus 014 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 013 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 012 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 011 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 009 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 010 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 008 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 007 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC
Bus 001 Device 003: ID 046d:c062 Logitech, Inc. M-UAS144 [LS1 Laser Mouse]
Bus 001 Device 004: ID 046a:0023 Cherry GmbH CyMotion Master Linux Keyboard G230
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub_

I therefor put src_fn = 5 .
The error I get when running Fictrac is now:

_Error! Unable to set process priority!
Looking for config file: config.txt ..
Config file parsed (21 key/value pairs).
VIDEOIO ERROR: V4L: can't open camera by index 5
[ INFO:0] global /home/nely/opencv/modules/videoio/src/backend_plugin.cpp (248) getPluginCandidates VideoIO pluigin (INTEL_MFX): glob is 'libopencv_videoio_intel_mfx*.so', 1 location(s)
[ INFO:0] global /home/nely/opencv/modules/videoio/src/backend_plugin.cpp (256) getPluginCandidates     - /usr/local/lib: 0
[ INFO:0] global /home/nely/opencv/modules/videoio/src/backend_plugin.cpp (259) getPluginCandidates Found 0 plugin(s) for INTEL_MFX
[ INFO:0] global /home/nely/opencv/modules/videoio/src/cap_images.cpp (282) icvExtractPattern Pattern: %01d @ 5
Could not interpret source type (5)!
Error! Could not open input frame source (5)!
Closing sphere tracker
Closing recorder..
Closing recorder.._

Does this help you to debug? I am still lost to be honest. I will reach out on Reddit to the others and thanks for the help so far!

[lumyus]

Actually upon building with SDK 5 specified we get the following:

Error! Unable to set process priority!
Looking for config file: config.txt ..
Config file parsed (21 key/value pairs).
Error opening capture device! Error was: Failed to open device '2676:ba02:14:4:5' for XML file download. Error: 'Device is exclusively opened by another client.'
Error! Could not open input frame source (5)!

So of course we closed the Pylon viewer to not have another client using the camera but this will result in the following error:

Error! Unable to set process priority!
Looking for config file: config.txt ..
Config file parsed (21 key/value pairs).
Opening Basler camera device: �*�p�
Error opening capture device! Error was: NULL pointer dereferenced
Error! Could not open input frame source (5)!
Closing sphere tracker

[rjdmoore]

Hmm, ok good debugging. Device 5 with SDK looks the way to go. It also looks unfortunately like it's an SDK problem now. It's not something I'm going to be able to fix myself in the next week or so as I don't have access to a Basler camera right now. I suggest you try installing pylon 5.2 - I think this was the active version last time I tested. Let me know how you go and I can update the docs if that works.

…

On Thu, 22 Apr 2021, 17:19 lumyus, ***@***.***> wrote: Actually upon building with SDK 5 specified we get the following: Error! Unable to set process priority! Looking for config file: config.txt .. Config file parsed (21 key/value pairs). Error opening capture device! Error was: Failed to open device '2676:ba02:14:4:5' for XML file download. Error: 'Device is exclusively opened by another client.' Error! Could not open input frame source (5)! So of course we closed the Pylon viewer to not have another client using the camera but this will result in the following error: Error! Unable to set process priority! Looking for config file: config.txt .. Config file parsed (21 key/value pairs). Opening Basler camera device: �*�p� Error opening capture device! Error was: NULL pointer dereferenced Error! Could not open input frame source (5)! Closing sphere tracker — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5KTA46EICVRZNH2KG6SZTTKA46PANCNFSM43IALQRA> .

[lumyus]

So I verified that Pylon SDK 5.2 is not working either. The errors remain.
What OpenCV version did you use? Found OpenCV: /usr/local (found version "4.1.0") is what we use.

Any other ideas on what might cause the issue? How exactly are we supposed to 'open' the camera stream? Do I have to edit anything at all in the Pylon Viewer?

[lumyus]

Actually while comparing with: https://github.com/basler/pylon-ros-camera/blob/ed094fad02eed38af830a052c7420befc6483ef3/pylon_camera/src/pylon_camera/write_device_user_id_to_camera.cpp

I noticed that we actually never open the camera?

[rjdmoore]

Hmm sounds odd. The Basler code was contributed by another user but has been working before. I will have to look into it. I don't have a Basler cam right now, but if you can help test it will help!

…

On Fri, 23 Apr 2021, 13:57 lumyus, ***@***.***> wrote: Actually while comparing with: https://github.com/basler/pylon-ros-camera/blob/ed094fad02eed38af830a052c7420befc6483ef3/pylon_camera/src/pylon_camera/write_device_user_id_to_camera.cpp I noticed that we actually never open the camera? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5KTA5DCL2YIKEOIPQ3LQLTKFOCHANCNFSM43IALQRA> .

[lumyus]

Yes we can do that together. We have a bunch of Basler Ace at hand in our lab. How do you want to proceed?

[lumyus]

I did some more testing today and it seems that first of all:
LOG("Opening Basler camera device: %s", _cam.GetDeviceInfo().GetModelName());
does not show a proper model name in the terminal but this did:

 // Print the model name of the camera.
cout << "Opening Basler camera device:" << camera.GetDeviceInfo().GetModelName() << endl;

and I get the correct camera name as output (which is already a good sign).
Now the null pointer exception I was talking about happens in this method:
_fps = getFPS();
but I saw that you didn't test this yet. (It's a TODO).
Right now I still have a frame grabbing error and I don't know how to resolve that.

I also saw that you defined "BaslerSource::BaslerSource(int index)" but never actually use the index anywhere, right?

Let me know if this helps you. Thanks again for your awesome work!

[lumyus]

Config file parsed (21 key/value pairs).
Using device acA1920-150um
Getting frame rate..
Basler camera initialised (1920x1200 @ 112.499 fps)!
Key (output_fn) not found.
Key (fisheye) not found.
Key (roi_c) not found.
Warning! Re-computed sphere ROI centred at [-0.490091 -0.243093 0.837088], with radius 0.023668 rad from 4 roi_circ points.
Input sphere mask automatically generated using 2 ignore ROIs!
roi_to_cam_r: -0.2572 0.5186 0.0000
Found C2A rotational transform: [0.722445 -0.131314 -0.460878].
Key (sphere_map_fn) not found.
Warning! No optimisation error threshold specified in config file (opt_max_err) - poor matches will not be dropped!
Key (sock_port) not found.
Key (com_port) not found.
Error! Recorder processing thread unable to set thread priority!
Key (thr_rgb_tfrm) not found.
Thresholding window size: 15 (ROI: 60 x 60)
Wrote 823 bytes to disk!
Error! Unable to set thread priority!
Starting sphere tracking loop!
Error! Unable to set thread priority!
Starting frame grabbing loop!
Error! Unable to set thread priority!
Frame captured 1920x1200x17301505 @ -1.000000 (1619520173168.530029)
Segmentation fault (core dumped)

I got to make some progress. This is the line I changed to get to this point: const GenApi::CFloatPtr camFrameRate = _cam.GetNodeMap().GetNode("ResultingFrameRate");

[lumyus]

Fictrac now works with the Ace cameras! The last issue was the following:

 Pylon::CImageFormatConverter formatConverter;
        formatConverter.OutputPixelFormat = PixelType_BGR8packed;
        formatConverter.Convert(_pylonImg, _ptrGrabResult);

I will make a proper pull request once I cleaned everything up.
There is still a bug in the configGUI. I will keep you posted.

[DanTurner-Evans]

Now it's my turn to try to get fictrac up and running with an ace camera (on a Windows machine). I modified the cmake file to point to the most recent library (PylonBase_v6_3.lib) and everything compiled nicely. When I try to run the executable, however, I get the issues that @lumyus was having (a garbled camera name and NULL pointer dereferenced). @lumyus were you able to resolve everything and, if so, would you be willing to outline exactly what I should change? Thanks!

[DanTurner-Evans]

As suggested above, I changed:
const GenApi::CFloatPtr camFrameRate = _cam.GetNodeMap().GetNode("ResultingFrameRate");
and added
formatConverter.OutputPixelFormat = PixelType_BGR8packed;
in BaslerSource.cpp, recompiled, and the camera seems to work now.