Skip to content

Commit 2322935

Browse files
JetsonHacksNanoJetsonHacksNano
authored
Merge pull request #49 from JetsonHacksNano/develop
Faster Gstreamer pipelines and exception handling in Python
2 parents cbec935 + 91a8d7a commit 2322935

13 files changed

+211
-844
lines changed

‎LICENSE

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
MIT License
22

3-
Copyright (c) 2019 JetsonHacks
3+
Copyright (c) 2019-2022 JetsonHacks
44

55
Permission is hereby granted, free of charge, to any person obtaining a copy
66
of this software and associated documentation files (the "Software"), to deal

‎README.md

100644100755
+50-31
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
11
# CSI-Camera
2-
Simple example of using a MIPI-CSI(2) Camera (like the Raspberry Pi Version 2 camera) with the NVIDIA Jetson Nano Developer Kit. This is support code for the article on JetsonHacks: https://wp.me/p7ZgI9-19v
2+
Simple example of using a MIPI-CSI(2) Camera (like the Raspberry Pi Version 2 camera) with the NVIDIA Jetson Developer Kits with CSI camera ports. This includes the recent Jetson Nano and Jetson Xavier NX. This is support code for the article on JetsonHacks: https://wp.me/p7ZgI9-19v
33

4-
The camera should be installed in the MIPI-CSI Camera Connector on the carrier board. The pins on the camera ribbon should face the Jetson Nano module, the stripe faces outward.
4+
For the Nanos and Xavier NX, the camera should be installed in the MIPI-CSI Camera Connector on the carrier board. The pins on the camera ribbon should face the Jetson module, the tape stripe faces outward.
55

6-
The new Jetson Nano B01 developer kit has two CSI camera slots. You can use the sensor_mode attribute with nvarguscamerasrc to specify the camera. Valid values are 0 or 1 (the default is 0 if not specified), i.e.
6+
Some Jetson developer kits have two CSI camera slots. You can use the sensor_mode attribute with the GStreamer nvarguscamerasrc element to specify which camera. Valid values are 0 or 1 (the default is 0 if not specified), i.e.
77

88
```
99
nvarguscamerasrc sensor_id=0
@@ -21,63 +21,67 @@ $ gst-launch-1.0 nvarguscamerasrc sensor_id=0 ! nvoverlaysink
2121
# Example also shows sensor_mode parameter to nvarguscamerasrc
2222
# See table below for example video modes of example sensor
2323
$ gst-launch-1.0 nvarguscamerasrc sensor_id=0 ! \
24-
'video/x-raw(memory:NVMM),width=3280, height=2464, framerate=21/1, format=NV12' ! \
25-
nvvidconv flip-method=0 ! 'video/x-raw,width=960, height=720' ! \
26-
nvvidconv ! nvegltransform ! nveglglessink -e
27-
28-
Note: The cameras appear to report differently than show below on some Jetsons. You can use the simple gst-launch example above to determine the camera modes that are reported by the sensor you are using. As an example the same camera from below may report differently on a Jetson Nano B01:
29-
30-
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000
31-
32-
You should adjust accordingly. As an example, for 3264x2464 @ 21 fps on sensor_id 1 of a Jetson Nano B01:
33-
$ gst-launch-1.0 nvarguscamerasrc sensor_id=1 ! \
34-
'video/x-raw(memory:NVMM),width=3264, height=2464, framerate=21/1, format=NV12' ! \
35-
nvvidconv flip-method=0 ! 'video/x-raw, width=816, height=616' ! \
24+
'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1' ! \
25+
nvvidconv flip-method=0 ! 'video/x-raw,width=960, height=540' ! \
3626
nvvidconv ! nvegltransform ! nveglglessink -e
27+
```
3728

38-
Also, it's been noticed that the display transform is sensitive to width and height (in the above example, width=816, height=616). If you experience issues, check to see if your display width and height is the same ratio as the camera frame size selected (In the above example, 816x616 is 1/4 the size of 3264x2464).
29+
Note: The cameras may report differently than show below. You can use the simple gst-launch example above to determine the camera modes that are reported by the sensor you are using.
30+
```
31+
GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;
3932
```
4033

41-
There are several examples:
34+
Also, the display transform may be sensitive to width and height (in the above example, width=960, height=540). If you experience issues, check to see if your display width and height is the same ratio as the camera frame size selected (In the above example, 960x540 is 1/4 the size of 1920x1080).
35+
4236

43-
Note: You may need to install numpy for the Python examples to work, ie $ pip3 install numpy
37+
## Samples
4438

45-
simple_camera.py is a Python script which reads from the camera and displays to a window on the screen using OpenCV:
4639

40+
### simple_camera.py
41+
simple_camera.py is a Python script which reads from the camera and displays the frame to a window on the screen using OpenCV:
42+
```
4743
$ python simple_camera.py
44+
```
45+
### face_detect.py
4846

4947
face_detect.py is a python script which reads from the camera and uses Haar Cascades to detect faces and eyes:
50-
48+
```
5149
$ python face_detect.py
52-
50+
```
5351
Haar Cascades is a machine learning based approach where a cascade function is trained from a lot of positive and negative images. The function is then used to detect objects in other images.
5452

5553
See: https://docs.opencv.org/3.3.1/d7/d8b/tutorial_py_face_detection.html
5654

57-
The third example is a simple C++ program which reads from the camera and displays to a window on the screen using OpenCV:
5855

56+
### dual_camera.py
57+
Note: You will need install numpy for the Dual Camera Python example to work:
5958
```
60-
$ g++ -std=c++11 -Wall -I/usr/lib/opencv simple_camera.cpp -L/usr/lib -lopencv_core -lopencv_highgui -lopencv_videoio -o simple_camera
61-
62-
$ ./simple_camera
59+
$ pip3 install numpy
6360
```
64-
65-
The final example is dual_camera.py. This example is for the newer rev B01 of the Jetson Nano board, identifiable by two CSI-MIPI camera ports. This is a simple Python program which reads both CSI cameras and displays them in a window. The window is 960x1080. For performance, the script uses a separate thread for reading each camera image stream. To run the script:
61+
This example is for the newer Jetson boards (Jetson Nano, Jetson Xavier NX) with two CSI-MIPI camera ports. This is a simple Python program which reads both CSI cameras and displays them in one window. The window is 1920x540. For performance, the script uses a separate thread for reading each camera image stream. To run the script:
6662

6763
```
6864
$ python3 dual_camera.py
6965
```
7066

71-
The directory 'instrumented' contains instrumented code which can help adjust performance and frame rates.
67+
### simple_camera.cpp
68+
The last example is a simple C++ program which reads from the camera and displays to a window on the screen using OpenCV:
69+
70+
```
71+
$ g++ -std=c++11 -Wall -I/usr/lib/opencv -I/usr/include/opencv4 simple_camera.cpp -L/usr/lib -lopencv_core -lopencv_highgui -lopencv_videoio -o simple_camera
72+
73+
$ ./simple_camera
74+
```
75+
This program is a simple outline, and does not handle needed error checking well. For better C++ code, use https://github.com/dusty-nv/jetson-utils
7276

7377
<h2>Notes</h2>
7478

7579
<h3>Camera Image Formats</h3>
7680
You can use v4l2-ctl to determine the camera capabilities. v4l2-ctl is in the v4l-utils:
77-
81+
```
7882
$ sudo apt-get install v4l-utils
79-
80-
For the Raspberry Pi V2 camera, typically the output is (assuming the camera is /dev/video0):
83+
```
84+
For the Raspberry Pi V2 camera, a typical output is (assuming the camera is /dev/video0):
8185

8286
```
8387
$ v4l2-ctl --list-formats-ext
@@ -130,6 +134,21 @@ If you can open the camera in GStreamer from the command line, and have issues o
130134

131135
<h2>Release Notes</h2>
132136

137+
v3.2 Release January, 2022
138+
* Add Exception handling to Python code
139+
* Faster GStreamer pipelines, better performance
140+
* Better naming of variables, simplification
141+
* Remove Instrumented examples
142+
* L4T 32.6.1 (JetPack 4.5)
143+
* OpenCV 4.4.1
144+
* Python3
145+
* Tested on Jetson Nano B01, Jetson Xavier NX
146+
* Tested with Raspberry Pi V2 cameras
147+
148+
149+
v3.11 Release April, 2020
150+
* Release both cameras in dual camera example (bug-fix)
151+
133152
v3.1 Release March, 2020
134153
* L4T 32.3.1 (JetPack 4.3)
135154
* OpenCV 4.1.1

‎dual_camera.py

100644100755
+73-70
Original file line numberDiff line numberDiff line change
@@ -1,33 +1,24 @@
11
# MIT License
2-
# Copyright (c) 2019,2020 JetsonHacks
3-
# See license
4-
# A very simple code snippet
2+
# Copyright (c) 2019-2022 JetsonHacks
3+
4+
# A simple code snippet
55
# Using two CSI cameras (such as the Raspberry Pi Version 2) connected to a
6-
# NVIDIA Jetson Nano Developer Kit (Rev B01) using OpenCV
6+
# NVIDIA Jetson Nano Developer Kit with two CSI ports (Jetson Nano, Jetson Xavier NX) via OpenCV
77
# Drivers for the camera and OpenCV are included in the base image in JetPack 4.3+
88

99
# This script will open a window and place the camera stream from each camera in a window
1010
# arranged horizontally.
1111
# The camera streams are each read in their own thread, as when done sequentially there
1212
# is a noticeable lag
13-
# For better performance, the next step would be to experiment with having the window display
14-
# in a separate thread
1513

1614
import cv2
1715
import threading
1816
import numpy as np
1917

20-
# gstreamer_pipeline returns a GStreamer pipeline for capturing from the CSI camera
21-
# Flip the image by setting the flip_method (most common values: 0 and 2)
22-
# display_width and display_height determine the size of each camera pane in the window on the screen
23-
24-
left_camera = None
25-
right_camera = None
26-
2718

2819
class CSI_Camera:
2920

30-
def __init__ (self) :
21+
def __init__(self):
3122
# Initialize instance variables
3223
# OpenCV video capture element
3324
self.video_capture = None
@@ -39,54 +30,54 @@ def __init__ (self) :
3930
self.read_lock = threading.Lock()
4031
self.running = False
4132

42-
4333
def open(self, gstreamer_pipeline_string):
4434
try:
4535
self.video_capture = cv2.VideoCapture(
4636
gstreamer_pipeline_string, cv2.CAP_GSTREAMER
4737
)
48-
38+
# Grab the first frame to start the video capturing
39+
self.grabbed, self.frame = self.video_capture.read()
40+
4941
except RuntimeError:
5042
self.video_capture = None
5143
print("Unable to open camera")
5244
print("Pipeline: " + gstreamer_pipeline_string)
53-
return
54-
# Grab the first frame to start the video capturing
55-
self.grabbed, self.frame = self.video_capture.read()
45+
5646

5747
def start(self):
5848
if self.running:
5949
print('Video capturing is already running')
6050
return None
6151
# create a thread to read the camera image
6252
if self.video_capture != None:
63-
self.running=True
53+
self.running = True
6454
self.read_thread = threading.Thread(target=self.updateCamera)
6555
self.read_thread.start()
6656
return self
6757

6858
def stop(self):
69-
self.running=False
59+
self.running = False
60+
# Kill the thread
7061
self.read_thread.join()
62+
self.read_thread = None
7163

7264
def updateCamera(self):
7365
# This is the thread to read images from the camera
7466
while self.running:
7567
try:
7668
grabbed, frame = self.video_capture.read()
7769
with self.read_lock:
78-
self.grabbed=grabbed
79-
self.frame=frame
70+
self.grabbed = grabbed
71+
self.frame = frame
8072
except RuntimeError:
8173
print("Could not read image from camera")
8274
# FIX ME - stop and cleanup thread
8375
# Something bad happened
84-
8576

8677
def read(self):
8778
with self.read_lock:
8879
frame = self.frame.copy()
89-
grabbed=self.grabbed
80+
grabbed = self.grabbed
9081
return grabbed, frame
9182

9283
def release(self):
@@ -98,30 +89,32 @@ def release(self):
9889
self.read_thread.join()
9990

10091

101-
# Currently there are setting frame rate on CSI Camera on Nano through gstreamer
102-
# Here we directly select sensor_mode 3 (1280x720, 59.9999 fps)
92+
"""
93+
gstreamer_pipeline returns a GStreamer pipeline for capturing from the CSI camera
94+
Flip the image by setting the flip_method (most common values: 0 and 2)
95+
display_width and display_height determine the size of each camera pane in the window on the screen
96+
Default 1920x1080
97+
"""
98+
99+
103100
def gstreamer_pipeline(
104101
sensor_id=0,
105-
sensor_mode=3,
106-
capture_width=1280,
107-
capture_height=720,
108-
display_width=1280,
109-
display_height=720,
102+
capture_width=1920,
103+
capture_height=1080,
104+
display_width=1920,
105+
display_height=1080,
110106
framerate=30,
111107
flip_method=0,
112108
):
113109
return (
114-
"nvarguscamerasrc sensor-id=%d sensor-mode=%d ! "
115-
"video/x-raw(memory:NVMM), "
116-
"width=(int)%d, height=(int)%d, "
117-
"format=(string)NV12, framerate=(fraction)%d/1 ! "
110+
"nvarguscamerasrc sensor-id=%d ! "
111+
"video/x-raw(memory:NVMM), width=(int)%d, height=(int)%d, framerate=(fraction)%d/1 ! "
118112
"nvvidconv flip-method=%d ! "
119113
"video/x-raw, width=(int)%d, height=(int)%d, format=(string)BGRx ! "
120114
"videoconvert ! "
121115
"video/x-raw, format=(string)BGR ! appsink"
122116
% (
123117
sensor_id,
124-
sensor_mode,
125118
capture_width,
126119
capture_height,
127120
framerate,
@@ -132,15 +125,17 @@ def gstreamer_pipeline(
132125
)
133126

134127

135-
def start_cameras():
128+
def run_cameras():
129+
window_title = "Dual CSI Cameras"
136130
left_camera = CSI_Camera()
137131
left_camera.open(
138132
gstreamer_pipeline(
139133
sensor_id=0,
140-
sensor_mode=3,
134+
capture_width=1920,
135+
capture_height=1080,
141136
flip_method=0,
142-
display_height=540,
143137
display_width=960,
138+
display_height=540,
144139
)
145140
)
146141
left_camera.start()
@@ -149,45 +144,53 @@ def start_cameras():
149144
right_camera.open(
150145
gstreamer_pipeline(
151146
sensor_id=1,
152-
sensor_mode=3,
147+
capture_width=1920,
148+
capture_height=1080,
153149
flip_method=0,
154-
display_height=540,
155150
display_width=960,
151+
display_height=540,
156152
)
157153
)
158154
right_camera.start()
159155

160-
cv2.namedWindow("CSI Cameras", cv2.WINDOW_AUTOSIZE)
161-
162-
if (
163-
not left_camera.video_capture.isOpened()
164-
or not right_camera.video_capture.isOpened()
165-
):
166-
# Cameras did not open, or no camera attached
156+
if left_camera.video_capture.isOpened() and right_camera.video_capture.isOpened():
167157

168-
print("Unable to open any cameras")
169-
# TODO: Proper Cleanup
170-
SystemExit(0)
158+
cv2.namedWindow(window_title, cv2.WINDOW_AUTOSIZE)
171159

172-
while cv2.getWindowProperty("CSI Cameras", 0) >= 0 :
173-
174-
_ , left_image=left_camera.read()
175-
_ , right_image=right_camera.read()
176-
camera_images = np.hstack((left_image, right_image))
177-
cv2.imshow("CSI Cameras", camera_images)
178-
179-
# This also acts as
180-
keyCode = cv2.waitKey(30) & 0xFF
181-
# Stop the program on the ESC key
182-
if keyCode == 27:
183-
break
160+
try:
161+
while True:
162+
_, left_image = left_camera.read()
163+
_, right_image = right_camera.read()
164+
# Use numpy to place images next to each other
165+
camera_images = np.hstack((left_image, right_image))
166+
# Check to see if the user closed the window
167+
# Under GTK+ (Jetson Default), WND_PROP_VISIBLE does not work correctly. Under Qt it does
168+
# GTK - Substitute WND_PROP_AUTOSIZE to detect if window has been closed by user
169+
if cv2.getWindowProperty(window_title, cv2.WND_PROP_AUTOSIZE) >= 0:
170+
cv2.imshow(window_title, camera_images)
171+
else:
172+
break
173+
174+
# This also acts as
175+
keyCode = cv2.waitKey(30) & 0xFF
176+
# Stop the program on the ESC key
177+
if keyCode == 27:
178+
break
179+
finally:
180+
181+
left_camera.stop()
182+
left_camera.release()
183+
right_camera.stop()
184+
right_camera.release()
185+
cv2.destroyAllWindows()
186+
else:
187+
print("Error: Unable to open both cameras")
188+
left_camera.stop()
189+
left_camera.release()
190+
right_camera.stop()
191+
right_camera.release()
184192

185-
left_camera.stop()
186-
left_camera.release()
187-
right_camera.stop()
188-
right_camera.release()
189-
cv2.destroyAllWindows()
190193

191194

192195
if __name__ == "__main__":
193-
start_cameras()
196+
run_cameras()

0 commit comments

Comments
 (0)