NEPI Documentation
< All Topics
Print
Updated

NEPI Engine – ZED Stereo Camera Systems

Introduction

This guide covers interfacing a StereoLab ZED 2 or 2i Stereo Camera to a NEPI software enabled edge-compute hardware system using NEPI’s IDX driver system.

What you will need:

1) 1x NEPI-enabled device. This tutorial uses an edge-compute processor box that includes an NVIDIA Jetson Xavier NX embedded GPU module with NEPI Engine software installed. (Optional) If you are connecting to your NEPI device through a NEPI configured access point, your NEPI device will also need Wi-Fi support.

NOTE: See available off-the-shelf NEPI enabled edge-compute options at:

https://numurus.com/products-nepi-enabled-hardware/

2. 1x NEPI-enabled device connected to the internet and your PC. This can be done either with a network switch connected to the internet, the PC, and the NEPI device, or a PC connected directly to the NEPI device through and Ethernet cable, and the NEPI device connected to the internet through WiFi.

3. 1x StereoLabs ZED 2 or 2i Stereo Camera (the IDX auto discovery system had issues with the ZED Mini cameras)

Software Setup

While NEPI’s IDX driver system can autodetect StereoLab ZED stereo cameras, the driver requires that a calibration file for your camera is installed on your NEPI device in order to connect.  

1) NEPI will automatically download the calibration file from the StereoLab website when the camera is connected and your NEPI device is connected to the internet.  Before plugging in your ZED camera, connect your NEPI device to the internet open your NEPI device’s RUI SYSTEM/ADMIN tab and either connect your NEPI device to an Ethernet enabled switch and enable DHCP, or connect to an internet supported WiFi network.  Wait for the “Internet Connected” icon at the top of the page to turn Green, then connect your ZED camera. 

NOTE: Detailed instructions for connecting NEPI devices to the internet are available at:

https://nepi.com/nepi-tutorials/nepi-engine-connecting-and-setup/

NOTE: You can optionally install the zed calibration files using the zed tools installed on your system:

cd /usr/local/zed/tools

./ZED_Explorer –dc

or manually download your camera’s calibration file from the StereoLab website, or create your own custom calibration file, and copy it to your NEPI device’s file system folder at 

/usr/local/zed/settings/

 

2) Plug your ZED camera into your NEPI device. After the NEPI IDX driver connects to your camera and configures the interface, you should see a blue LED light on the front of your camera.

3) (Optional) The ZED IDX camera driver creates all of the standard IDX driver ROS status, control, and data messages listed in the IDX section of the NEPI Engine – Hardware Driver Interface System guide at:

https://nepi.com/documentation/nepi-engine-hardware-driver-interface-system/

NOTE: Instructions for SSH into your NEPI device are available at:

https://nepi.com/nepi-tutorials/nepi-engine-accessing-the-nepi-file-system/

You can see a full list of IDX ROS interface controls and data topics by opening an SSH window to your NEPI device and typing:

rostopic list | grep idx

which should produce the following list:

  1. /nepi/s2x/zed2i_stereo_camera/idx/bw_2d_image
  2. /nepi/s2x/zed2i_stereo_camera/idx/color_2d_image
  3. /nepi/s2x/zed2i_stereo_camera/idx/depth_image
  4. /nepi/s2x/zed2i_stereo_camera/idx/depth_map
  5. /nepi/s2x/zed2i_stereo_camera/idx/navpose_support
  6. /nepi/s2x/zed2i_stereo_camera/idx/odom
  7. /nepi/s2x/zed2i_stereo_camera/idx/pointcloud
  8. /nepi/s2x/zed2i_stereo_camera/idx/pointcloud_image
  9. /nepi/s2x/zed2i_stereo_camera/idx/set_brightness
  10. /nepi/s2x/zed2i_stereo_camera/idx/set_contrast
  11. /nepi/s2x/zed2i_stereo_camera/idx/set_range_window
  12. /nepi/s2x/zed2i_stereo_camera/idx/set_thresholding
  13. /nepi/s2x/zed2i_stereo_camera/idx/status

Configuring the Camera in the RUI

1) Open the NEPI device’s RUI SENSORS/IMAGING tab, select one of the ZED camera’s IDS 2D image streams from the dropdown box, and configure any settings you want to adjust.

2) You can save any camera settings changes by going to the RUI SYSTEM/ADMIN tab and selecting the “Save” button.

A screenshot of a video Description automatically generated

4) View camera output streams simultaneously using the RUI’s APPLICATIONS/IMAGE VIEWER tab by selecting the onboard image topics you want to have shown in each box.

Viewing 2D and Depth in the RUI

Open the NEPI device’s RUI APPLICATION/IMAGERY tab, select one or multiple camera streams to view simultaneously, including the device’s depth_image stream.

A screenshot of a computer Description automatically generated

Pointcloud Viewing with ROS RVIZ

You can view the ZED cameras 3D Point Cloud using the ROS rviz application on a NEPI ROS configured PC connected to your NEPI device following these instructions.

1) There are two options for running rviz,

a) Connect a display, keyboard, and mouse to your NEPI device and run rviz from a terminal on the NEPI device’s desktop.

b) Run rviz on a network connected PC with ROS installed and configured to communicate with the NEPI ROS environment. Follow the instructions in the tutorial link below to setup your Linux PC for NEPI ROS interfacing:

https://nepi.com/nepi-tutorials/nepi-engine-configuring-a-pc-for-nepi-ros-env/

2) Open a terminal on your Linux PC and type:

rviz

This will open the rviz ROS application.

3) Select the “Add ” button on the bottom of the left menu, select the “PointCloud2” item from the popup list.  This will put a new PointCloud2 Display item in the left menu bar.  Maximize the PointCloud2 Display item and click on the “Topic” selection line, then select the pointcloud you want to visualize from the dropdown list.

A screenshot of a computer Description automatically generated

Automation Script Examples

You can access the ZED camera’s ROS image, depth, and pointcloud data streams, as well as send camera controls in custom NEPI automation scripts using the python rospy library. A few example ZED camera automation scripts are described below and available for download from the NEPI Engine Automation Script repo at:

https://github.com/numurus-nepi/nepi_sample_auto_scripts

NOTE: For more details on using NEPI automation scripts and the NEPI RUI Automation application, see the NEPI Engine – Automation Application tutorial at:

https://nepi.com/nepi-tutorials/nepi-engine-automation-application/

1) This automation example uses the following scripts downloaded from the link above:

  1. _object_detect_and_3d_targeting_startup_script.py
  2. ai_detector_config_script.py
  3. ai_3d_targeting_process_scropt.py” scripts

Download these scripts and install on your NEPI device’s user storage drive in the “automation_scripts” folder.

2) Open the NEPI device’s RUI APPLICATION/AUTOMATION tab and start the ““_object_detect_and_3d_targeting_startup_script.py” script which automates starting the “ai_detector_config_script.py” and “ai_3d_targeting_process_scropt.py” scripts.

A screenshot of a computer program Description automatically generated

Together, these NEPI automation script starts an AI detector, then uses its detection outputs along with the depth data from the ZED camara’s depth_map image stream to estimate range and bearing of detected targets.  The script then publishes a TargetLocation ROS data stream and a new image with detection and range/bearing data to a new image topic named targeting_image which you can view in the RUI APPLICATIONS/IMAGERY tab

A person sitting at a desk using a computer Description automatically generated

The range, horizontal angle, and vertical targeting stream can be used by other scripts and can be viewed from an SSH terminal connected to your NEPI device using the folloing command:

rostopic echo /nepi/s2x/targeting/targeting_data

A screen shot of a computer Description automatically generated

Table of Contents