< All Topics
Print

NEPI Engine – ZED Stereo Camera Systems

Introduction

This guide covers interfacing a StereoLab ZED 2 or 2i Stereo Camera to a NEPI software enabled edge-compute hardware system using NEPI’s IDX driver system.

What you will need:

1) 1x NEPI-enabled device. This tutorial uses an edge-compute processor box that includes an NVIDIA Jetson Xavier NX embedded GPU module with NEPI Engine software installed. (Optional) If you are connecting to your NEPI device through a NEPI configured access point, your NEPI device will also need Wi-Fi support.

NOTE: See available off-the-shelf NEPI enabled edge-compute options at:

https://numurus.com/products-nepi-enabled-hardware/

2. 1x NEPI-enabled device connected to the internet and your PC. This can be done either with a network switch connected to the internet, the PC, and the NEPI device, or a PC connected directly to the NEPI device through and Ethernet cable, and the NEPI device connected to the internet through WiFi.

3. 1x StereoLabs ZED 2 or 2i Stereo Camera (the IDX auto discovery system had issues with the ZED Mini cameras)

Software Setup

While NEPI’s IDX driver system can autodetect StereoLab ZED stereo cameras, the driver requires that a calibration file for your camera is installed on your NEPI device in order to connect.  There are two options for installing the calibration file:

1) NEPI will automatically download the calibration file from the StereoLab website when the camera is connected and your NEPI device is connected to the internet.  Before plugging in your ZED camera, open your NEPI device’s RUI SYSTEM/ADMIN tab and either connect your NEPI device to an Ethernet enabled switch and enable DHCP, or connect to an internet supported WiFi network.  Wait for the “Internet Connected” icon at the top of the page to turn Green, then connect your ZED camera. 

2) Manually download your camera’s calibration file from the StereoLab website, or create your own custom calibration file, and copy it to your NEPI device’s file system folder at 

/usr/local/zed/settings/

ZED Camera Interfacing

1) Plug your ZED camera into your NEPI device. After the NEPI IDX driver connects to your camera and configures the interface, you should see a blue LED light on the front of your camera.

2) The ZED IDX camera driver creates all of the standard IDX driver ROS status, control, and data messages listed in the IDX section of the NEPI Engine – Hardware Driver Interface System guide at:

https://nepi.com/documentation/nepi-engine-hardware-driver-interface-system/

you can see a full list of IDX ROS interface controls and data topics by opening an SSH window to your NEPI device and typing:

rostopic list | grep idx

which should produce the following list:

  1. /nepi/s2x/zed2i_stereo_camera/idx/bw_2d_image
  2. /nepi/s2x/zed2i_stereo_camera/idx/color_2d_image
  3. /nepi/s2x/zed2i_stereo_camera/idx/depth_image
  4. /nepi/s2x/zed2i_stereo_camera/idx/depth_map
  5. /nepi/s2x/zed2i_stereo_camera/idx/navpose_support
  6. /nepi/s2x/zed2i_stereo_camera/idx/odom
  7. /nepi/s2x/zed2i_stereo_camera/idx/pointcloud
  8. /nepi/s2x/zed2i_stereo_camera/idx/pointcloud_image
  9. /nepi/s2x/zed2i_stereo_camera/idx/set_brightness
  10. /nepi/s2x/zed2i_stereo_camera/idx/set_contrast
  11. /nepi/s2x/zed2i_stereo_camera/idx/set_range_window
  12. /nepi/s2x/zed2i_stereo_camera/idx/set_thresholding
  13. /nepi/s2x/zed2i_stereo_camera/idx/status

Configuring the Camera in the RUI

1) Open the NEPI device’s RUI SENSORS/IMAGING tab, select one of the ZED camera’s IDS 2D image streams from the dropdown box, and configure any settings you want to adjust.

2) You can save any camera settings changes by going to the RUI SYSTEM/ADMIN tab and selecting the “Save” button.

A screenshot of a video Description automatically generated

4) View camera output streams simultaneously using the RUI’s APPLICATIONS/IMAGE VIEWER tab by selecting the onboard image topics you want to have shown in each box.

Viewing 2D and Depth in the RUI

Open the NEPI device’s RUI APPLICATION/IMAGERY tab, select one or multiple camera streams to view simultaneously, including the device’s depth_image stream.

A screenshot of a computer Description automatically generated

Pointcloud Viewing with ROS RVIZ

You can view the ZED cameras 3D Point Cloud using the ROS rviz application on a NEPI ROS configured PC connected to your NEPI device following these instructions.

1) Follow the instructions in the tutorial link below to setup your Linux PC for NEPI ROS interfacing:

https://nepi.com/nepi-tutorials/nepi-engine-configuring-a-pc-for-nepi-ros-env/

2) Open a terminal on your Linux PC and type:

rviz

This will open the rviz ROS application.

3) Download the “rviz_zed_config.rviz” configuration file provided as an attachement to this tutorial, then from within the rviz application, select File/Open Config option. You should see both the ZED camera’s left color image and a point cloud viewer window connected to the ZED camera’s pointcloud ROS stream.

A screenshot of a computer Description automatically generated

Automation Script Examples

You can access the ZED camera’s ROS image, depth, and pointcloud data streams, as well as send camera controls in custom NEPI automation scripts using the python rospy library. A few example ZED camera automation scripts are described below and available for download from the NEPI Engine Automation Script repo at:

https://github.com/numurus-nepi/nepi_sample_auto_scripts

NOTE: For more details on using NEPI automation scripts and the NEPI RUI Automation application, see the NEPI Engine – Automation Application tutorial at:

https://nepi.com/nepi-tutorials/nepi-engine-automation-application/

1) This automation example uses the following scripts downloaded from the link above:

  1. _object_detect_and_3d_targeting_startup_script.py
  2. ai_detector_config_script.py
  3. ai_3d_targeting_process_scropt.py” scripts

Download these scripts and install on your NEPI device’s user storage drive in the “automation_scripts” folder.

2) Open the NEPI device’s RUI APPLICATION/AUTOMATION tab and start the ““_object_detect_and_3d_targeting_startup_script.py” script which automates starting the “ai_detector_config_script.py” and “ai_3d_targeting_process_scropt.py” scripts.

A screenshot of a computer program Description automatically generated

Together, these NEPI automation script starts an AI detector, then uses its detection outputs along with the depth data from the ZED camara’s depth_map image stream to estimate range and bearing of detected targets.  The script then publishes a TargetLocation ROS data stream and a new image with detection and range/bearing data to a new image topic named targeting_image which you can view in the RUI APPLICATIONS/IMAGERY tab

A person sitting at a desk using a computer Description automatically generated

The range, horizontal angle, and vertical targeting stream can be used by other scripts and can be viewed from an SSH terminal connected to your NEPI device using the folloing command:

rostopic echo /nepi/s2x/targeting/targeting_data

A screen shot of a computer Description automatically generated

Table of Contents