DeepStream-3D Custom Apps and Libs Tutorials — DeepStream documentation (original) (raw)

ds3d framework, interfaces and custom-libs are designed for DeepStream-3D processing. ds3d is agnostic from Gstreamer/Glib frameworks. These interfaces are capble of different types of data fusion. Developers can implement different types of custom libraries for dataloader, datafilter and datarender. The interface has ABI compatible layers and modern C++ interface. Developers only need to focus on the modern C++ interface for application or custom lib development.

DS3D dataloader is loaded by GstAppSrc, enabling its utilization for depth cameras like stereo cameras and Time-of-Flight cameras to capture image/depth data or load data from the file system. Additionally, it can be employed for capturing lidar data from sensors or lidar data files.datafilter is loaded by the nvds3dfilter Gst-plugin. It could be used for 2D depth data processing , 3D point-cloud data extraction from depth, other 2D-depth or 3D-points data filters and lidar or 3D data inference.datarender is loaded by GstAppSink. It could be used for 2D depth rendering and 3D point-cloud and lidar data rendering. It also could be used for file dump.

DS3D Application Examples#

The image below shows the overview of lidar 3D data inference and rendering pipeline in deepstream-lidar-inference.

DeepStream Lidar point cloud inference and rendering overview

See more details in the DeepStream 3D Lidar Inference App.

The image below shows the overview of depth to 3D point processing pipeline in deepstream-3d-depth-camera.

DeepStream Depth Camera for 3D point cloud processing overview

See more details in the DeepStream 3D Depth Camera App.

All the components are configured in YAML format. They are loaded by Gst-plugins.There are 3 major components, they may all be loaded into the deepstream pipeline.

DS3D data format ds3d/datamap#

ds3d/datamap recognizes the data format used in DS3D framework. Data buffers flowing between GStreamer plugins will be of this data-type.ds3d/datamap are key-value pairs where key is a string and value, a pointer, structure or tensor frame to data. All of the operations on the buffer is managed by GuardDataMap.

Examples:

#include <ds3d/common/hpp/datamap.hpp>
using namespace ds3d;
GuardDataMap datamap(NvDs3d_CreateDataHashMap(), true); // creat a empty datamap
TimeStamp ts{0};
datamap.setData("DS3D::Timestamp", ts); // set timestamp
float score = 0.1;
datamap.setData("DS3D::Score", score); // copy score into datamap

struct TimeStamp {
uint64_t t0 = 0;
uint64_t t1 = 0;
uint64_t t2 = 0;
REGISTER_TYPE_ID(DS3D_TYPEID_TIMESTAMP)
};

// Add the shared_ptr into datamap
std::shared_ptr timePtr(new TimeStamp);
datamap.setPtrData("DS3D::Timestamp0", timePtr);

// Add a copy of the TimeStamp into datamap
TimeStamp time1;
datamap.setData("DS3D::Timestamp1", time1);

Example 2, Instantiate template struct TpId<> { static constexpr TIdType __typeid(); }, this is helpful when adding a 3rdparty data structure into datamap and not able to modify the 3rd-party existing DataStructure.

// this is the 3rd-party structure
struct Existing3rdpartData {
float v0;
int v1;
};

// derive ds3d::__TypeID for any 3rdparty data structure.
#incude "ds3d/common/type_trait.h"
#define DS3D_TYPEID_EXISTING_3RDPART_DATA 0x80001
namespace ds3d {
template <>
struct TpId: __TypeID {};
}

// Add the shared_ptr into datamap
std::shared_ptr dataPtr(new Existing3rdpartData);
datamap.setPtrData("3rdpartyData0", dataPtr);

// Add a copy of the Existing3rdpartData into datamap
Existing3rdpartData data3rdparty{0.0f, 0};
datamap.setData("3rdpartyData1", data3rdparty);

const std::string keyName = "DS3D::dataarray"; // user define a key name.
// setGuardData holds a reference_count on the tensorFrame without deep-copy
datamap.setGuardData(keyName, tensorFrame);

DS3D DataMap interoperate with GstBuffer#

DS3D DataMap has ABI-compatible class ds3d::abiRefDataMap. With that, NvDs3DBuffer is defined for storing DS3D datamap along with GstBuffer. Header file is ds3d/gst/nvds3d_meta.h.

struct NvDs3DBuffer { uint32_t magicID; // must be 'DS3D' ds3d::abiRefDataMap* datamap; };

Warning

Do not use the datamap directly. The easy and safe way to access that is through GuardDataMap. see examples below

#include <ds3d/common/hpp/datamap.hpp> #include <ds3d/common/hpp/frame.hpp> using namespace ds3d;

GstBuffer gstBuf = ; // get the gstBuf from probe function or Gstreamer plugins if (NvDs3D_IsDs3DBuf(gstBuf)) { const abiRefDataMap refDataMap = nullptr; ErrCode c = NvDs3D_Find1stDataMap(gstBuf, refDataMap); if (refDataMap) { GuardDataMap dataMap(*refDataMap); FrameGuard lidarFrame; c = dataMap.getGuardData("DS3D::LidarXYZI", pointFrame); // get lidar points reference.

FrameGuard uvCoord; c = dataMap.getGuardData("DS3D::TextureCoordKey", uvCoord); // get 3D points UV coordinates reference.

Frame2DGuard depthFrame; c = dataMap.getGuardData("DS3D::DepthFrame", depthFrame); // get depth frame reference.

DepthScale scale; c = dataMap.getData("DS3D::DepthScaleUnit", scale); // copy depth scale }

}

#include <ds3d/common/hpp/datamap.hpp>
#include <ds3d/common/hpp/frame.hpp>
#include <ds3d/common/impl/impl_frames.h>
GuardDataMap datamap(NvDs3d_CreateDataHashMap(), true); // set true to take the reference ownership.
/* Create color image frame and store them into ds3d datamap. /
// Assume format is RGBA
{
Frame2DPlane colorPlane = {1920, 1080, 1920 * sizeof(uint8_t) , sizeof(uint8_t), 0};
uint32_t colorBytesPerFrame = colorPlane.pitchInBytes * colorPlane.height;
std::vector data(colorBytesPerFrame); // Image data
void dataPtr = &data[0];
// create color 2D frame
Frame2DGuard frame = Wrap2DFrame<uint8_t, FrameType::kColorRGBA>(
dataPtr, {_config.colorPlane}, bytesPerFrame, MemType::kCpu, 0,
data = std::move(data) {});
c = datamap.setGuardData(kColorFrame, colorFrame); // store colorFrame reference into datamap.
... // check error code
}
Once datamap is ready, you can create a new GstBuffer with DS3D datamap.
// GuardDataMap datamap is ready
GstBuffer* gstBuf = nullptr;
ErrCode c = NvDs3D_CreateGstBuf(gstBuf, datamap.abiRef(), false); // set false to increase reference count.
... // check error code

ds3d::dataloader - Load Custom Lib for Data Capture#

Load and Manage DS3D Dataloader#

Examples:

name: realsense_dataloader type: ds3d::dataloader out_caps: ds3d/datamap custom_lib_path: libnvds_3d_dataloader_realsense.so custom_create_function: createRealsenseDataloader config_body: streams: [color, depth]

A custom dataloader must have type: ds3d::dataloader. It is created by explicit call of NvDs3D_CreateDataLoaderSrc(srcConfig, loaderSrc, start) with the full compoment YAML content. During this call, the custom_lib_path is loaded and a specific data loader is created via custom_create_function. A GstAppsrc object is also created into loaderSrc.gstElement.

GstAppsrc manages the ds3d::dataloader dataflows. This ds3d::dataloader component could be started automatically by gst-pipeline or manually by the application call.

GuardDataLoader dataloader = loaderSrc.customProcessor; ErrCode c = dataloader.start();

To stop the dataloader, user can set GstAppsrc states to GST_STATE_READY or stop it manually.

GuardDataLoader dataloader = loaderSrc.customProcessor; ErrCode c = dataloader.stop();

DS3D Dataloader in DeepStream User Application#

DS3D Custom Dataloaders are agnostic from Gstreamer/Glib Framework. But it could be working with Gstramer as well. The ds3d::dataloader could interactively work with GstAppSrc together. They could be created by NvDs3D_CreateDataLoaderSrc.

Examples:

#include <ds3d/common/config.h> #include <ds3d/gst/nvds3d_gst_plugin.h>

std::string yamlStr = R"( name: ds3d_lidar_file_source type: ds3d::dataloader out_caps: ds3d/datamap custom_lib_path: libnvds_lidarfileread.so custom_create_function: createLidarFileLoader config_body: data_config_file: lidar_data_list.yaml points_num: 242180 mem_type: gpu # choose [cpu gpu] gpu_id: 0 mem_pool_size: 6 element_size: 4 output_datamap_key: DS3D::LidarXYZI )";

ErrCode c = ErrCode::kGood; ComponentConfig config; c = parseComponentConfig(yamlStr.c_str(), "./config_lidar_loader.yaml", config); DS_ASSERT(config.type == ComponentType::kDataLoader); DS_ASSERT(config.customLibPath == "libnvds_lidarfileread.so"); DS_ASSERT(config.customCreateFunction == "createLidarFileLoader");

gst::DataLoaderSrc appLoader; c = NvDs3D_CreateDataLoaderSrc(c, appLoader, true); // get DS3D custom dataloader. GuardDataLoader dataLoader = appLoader.customProcessor; // get GstAppSrc from this appLoader; GstElement* appsrc = appLoader.gstElement.get();

// gstreamer pipeline setup and running. // during each read_data from GstAppSrc callback. it would simultaneously reading data from dataloader // GuardDataMap datamap; // dataloader.read_data(datamap);

// DS3D dataloader would stop qutomatically when Gstpipeline is stopped. // But in the case if user want to stop it early or manually. // Obtain DS3D custom dataloader and stop manually. c = dataloader.flush(); c = dataloader.stop();

GuardDataLoader provides safe access to abiDataLoader. Once it’s created, it will maintain the reference pointer to the dataloader.

Implement a DS3D Custom Dataloader#

Examples:

#include <ds3d/common/impl/impl_dataloader.h> class TestTimeDataLoader : public ds3d::impl::SyncImplDataLoader { public: TestTimeDataLoader() = default;

protected: ErrCode startImpl(const std::string& content, const std::string& path) override { setOutputCaps("ds3d/datamap"); return ErrCode::kGood; } ErrCode readDataImpl(GuardDataMap& datamap) override { datamap.reset(NvDs3d_CreateDataHashMap()); static uint64_t iTime = 0; TimeStamp t{iTime++, 0, 0}; datamap.setData("time", t); emitError(ErrCode::kGood, "timstamp added"); return ErrCode::kGood; } ErrCode stopImpl() override { return ErrCode::kGood; } ErrCode flushImpl() override { return ErrCode::kGood; } };

DS3D_EXTERN_C_BEGIN DS3D_EXPORT_API abiRefDataLoader* createTestTimeDataloader() { return NewAbiRef(new TestTimeDataLoader); } DS3D_EXTERN_C_END

A shown in the example above, You’ll need to derive dataloader from the ds3d::impl::SyncImplDataLoader class, and implement interfaces for the following:

ErrCode startImpl(const std::string& content, const std::string& path) override; ErrCode readDataImpl(GuardDataMap& datamap) override; ErrCode stopImpl() override; ErrCode flushImpl() override;

ds3d::databridge - Loads Custom Lib for data conversion to and from DS3D.#

This plugin and custom lib helps convert data type to and from ds3d/datamap

More details: Gst-nvds3dbridge.

ds3d::datafilter- DS3D Custom DataFilter#

DS3D DataFilter is processing inputs from ds3d::datamap and producing outputs input new ds3d::datamap. Users can implement custom datafilter lib for their own use case.

Create And Manage DS3D Datafilter in DeepStream App#

Examples:

#include <ds3d/common/config.h> #include <ds3d/gst/nvds3d_gst_ptr.h> #include <ds3d/gst/nvds3d_gst_plugin.h>

std::string yamlStr = R"( name: fusion_inference type: ds3d::datafilter in_caps: ds3d/datamap out_caps: ds3d/datamap custom_lib_path: libnvds_tritoninferfilter.so custom_create_function: createLidarInferenceFilter config_body: mem_pool_size: 2 model_inputs: config_file: model_config_files/config_triton_bev_fusion_infer_grpc.pbtxt - name: input_image_0 datatype: UINT8 shape: [1, 900, 1600, 4] from: DS3D::ColorFrame_0+1 is_2d_frame: true - name: input_lidar datatype: FP32 shape: [242180, 4] from: DS3D::LidarXYZI+0 )";

gst::ElePtr gstPlugin = gst::elementMake("nvds3dfilter", "ds3d-custom-filter"); g_object_set(G_OBJECT(gstPlugin.get()), "config-content", yamlStr.c_str(), nullptr); GstElement* ele = gstPlugin.get();

A custom datafilter must have type: ds3d::datafilter. It is loaded through the nvds3dfilter Gst-plugin. It is started by gst_element_set_state(GST_STATE_READY). During this call, the custom_lib_path is loaded and a specific data filter is created by custom_create_function. nvds3dfilter Gst-plugin has config-content and config-file properties. One of them must be set to create a datafilter object.

Implement a Custom DS3D Datafilter#

Examples:

#include <ds3d/common/impl/impl_datafilter.h> class TestFakeDataFilter : public impl::BaseImplDataFilter { public: TestFakeDataFilter() = default;

protected: ErrCode startImpl(const std::string& content, const std::string& path) override { setInputCaps(kFakeCapsMetaName); setOutputCaps(kFakeCapsMetaName); return ErrCode::kGood; } ErrCode processImpl( GuardDataMap datamap, OnGuardDataCBImpl outputDataCb, OnGuardDataCBImpl inputConsumedCb) override { DS_ASSERT(datamap); TimeStamp t; ErrCode c = datamap.getData("time", t); if (!isGood(c)) { return c; } t.t0 += 1; inputConsumedCb(ErrCode::kGood, datamap); c = datamap.setData("time", t); if (!isGood(c)) { return c; } outputDataCb(ErrCode::kGood, datamap); return ErrCode::kGood; }

ErrCode flushImpl() override { return ErrCode::kGood; }
ErrCode stopImpl() override { return ErrCode::kGood; }

};

DS3D_EXTERN_C_BEGIN DS3D_EXPORT_API abiRefdatafilter* createTestFakeDatafilter() { return NewAbiRef(new TestFakeDataFilter); } DS3D_EXTERN_C_END

As shown in the example above, you’ll need to derive the datafilter from the ds3d::impl::BaseImplDataFilter class, and implement interfaces for the following:

ErrCode startImpl(const std::string& content, const std::string& path) override; ErrCode processImpl( GuardDataMap datamap, OnGuardDataCBImpl outputDataCb, OnGuardDataCBImpl inputConsumedCb) override; ErrCode stopImpl() override; ErrCode flushImpl() override;

To load this custom lib through nvds3dfilter Gst-plugin, you’ll also need to export a specific symbol createTestFakeDatafilter.

ds3d::datarender - Loads DS3D Custom DataRender#

DS3D Custom DataRenders are agnostic from Gstreamer/Glib Framework. But it could be working with Gstramer as well. The ds3d::datarender could interactively work with GstAppSink together. They could be created by NvDs3D_CreateDataRenderSink.

Examples:

Load And Manage DS3D Datarender#

Examples:

#include <ds3d/common/config.h> #include <ds3d/gst/nvds3d_gst_plugin.h>

std::string yamlStr = R"( name: lidar_render type: ds3d::datarender in_caps: ds3d/datamap custom_lib_path: libnvds_3d_gl_datarender.so custom_create_function: createLidarDataRender gst_properties: sync: True async: False drop: False config_body: title: ds3d-lidar-render streams: [lidardata] width: 1280 height: 720 block: True view_position: [0, 0, 60] view_target: [0, 0, 0] view_up: [0, 1, 0] near: 0.3 far: 100 fov: 50 lidar_color: [0, 255, 0] lidar_data_key: DS3D::LidarXYZI element_size: 4 lidar_bbox_key: DS3D::Lidar3DBboxRawData enable_label: True )";

ErrCode c = ErrCode::kGood; ComponentConfig config; c = parseComponentConfig(yamlStr.c_str(), "./config_lidar_loader.yaml", config); DS_ASSERT(config.type == ComponentType::kDataRender); DS_ASSERT(config.customLibPath == "libnvds_3d_gl_datarender.so"); DS_ASSERT(config.customCreateFunction == "createLidarDataRender");

gst::DataRenderSink appRender; c = NvDs3D_CreateDataRenderSink(config, appRender, true); // get DS3D custom datarender. GuardDataRender datarender = appRender.customProcessor; // get GstAppSink from this appRender; GstElement* appsink = appRender.gstElement.get();

// gstreamer pipeline setup and running. // during each render_data from GstAppSink callback. it would simultaneously reading data from datarender // datarender.render(datamap, consumed_callback);

// DS3D datarender would stop qutomatically when Gstpipeline is stopped. // But in the case if user want to stop it early or manually. // Obtain DS3D custom datarender and stop manually. c = datarender.stop();

A custom datarender must have type: ds3d::datarender. It is created by explicit call of NvDs3D_CreateDataRenderSink(sinkConfig, renderSink, start) with the full compoment YAML content. During this call, the custom_lib_path is loaded and a specific data loader is created via custom_create_function. A GstAppsink object is also created into renderSink.gstElement.

GstAppsink manages the ds3d::datarender dataflows. This ds3d::datarender component could be automatically started by the gst-pipeline, or manually by the application call.

GuardDataRender datarender = renderSink.customProcessor; ErrCode c = datarender.start();

To stop the datarender, you can set GstAppsink states to GST_STATE_READY, or stop manually. .. code-block:: text

GuardDataRender datarender = renderSink.customProcessor; ErrCode c = datarender.stop();

GuardDataRender provides safe access to abidatarender. Once it’s created, it will maintain the reference pointer to datarender. preroll is called only once to initialize some resources.

Implement a DS3D Custom Datarender#

Examples:

#include <ds3d/common/impl/impl_datarender.h> class TestFakeDataRender : public impl::BaseImplDataRender { public: TestFakeDataRender() = default;

protected: ErrCode startImpl(const std::string& content, const std::string& path) override { setInputCaps("ds3d/datamap"); return ErrCode::kGood; } ErrCode prerollImpl(GuardDataMap datamap) override { return ErrCode::kGood; } ErrCode renderImpl(GuardDataMap datamap, OnGuardDataCBImpl dataDoneCb) override { DS_ASSERT(datamap); emitError(ErrCode::kGood, "data rendered"); dataDoneCb(ErrCode::kGood, datamap); return ErrCode::kGood; } ErrCode flushImpl() override { return ErrCode::kGood; } ErrCode stopImpl() override { return ErrCode::kGood; } };

DS3D_EXTERN_C_BEGIN DS3D_EXPORT_API abiRefdatarender* createTestFakedatarender() { return NewAbiRef(new TestFakeDataRender()); } DS3D_EXTERN_C_END

As shown in the example above, you’ll need to derive datarender from the ds3d::impl::BaseImplDataRender class, and implement interfaces for the following:

ErrCode startImpl(const std::string& content, const std::string& path) override; ErrCode prerollImpl(GuardDataMap datamap) override; ErrCode renderImpl(GuardDataMap datamap, OnGuardDataCBImpl dataDoneCb) override; ErrCode stopImpl() override; ErrCode flushImpl() override;

To load this custom lib through NvDs3D_CreateDataRenderSink, you’ll also need to export a specific symbol createTestFakedatarender.

Custom Libs Configuration Specifications#

Components Common Configuration Specifications#

These custom libs are part of DeepStream release package.

Supported DS3D Custom Process libraries#

libnvds_tritoninferfilter Configuration Specifications#

Multi-modal tensors Triton inference with key-value pairs in ds3d::datamap, Supports user defined custom preprocess and postprocess.

model_inputs has all input layers information. If there is custom preprocess function/lib specfied inside config body. The lib would search from keyname of the input ds3d::datamap, and forward the frame/tensor data into Triton server’s input tensors. The from keyname must be a frame, 2D-frame or tensor inside the ds3d::datamap

If custom_preprocess_lib_path and custom_preprocess_func_name are specified, Custom processing will be loaded and parse the config body and get all common and user defined information. then process each input ds3d::datamap and generate batchArray for triton inference inputs.

Users can derive interface IInferCustomPreprocessor from sources/includes/ds3d/common/hpp/lidar_custom_process.hppSee examples in sources/libs/ds3d/inference_custom_lib/ds3d_v2x_infer_custom_preprocess/nvinferserver_custom_preprocess.cpp

#include <ds3d/common/hpp/datamap.hpp> #include <ds3d/common/hpp/frame.hpp> #include <ds3d/common/hpp/lidar_custom_process.hpp> using namespace ds3d; using namespace nvdsinferserver; class NvInferServerCustomPreProcess : public IInferCustomPreprocessor { public: // process key-values from datamap and generate into model inputs batchArray. NvDsInferStatus preproc(GuardDataMap &datamap, SharedIBatchArray batchArray, cudaStream_t stream) override {...} }; extern "C" { IInferCustomPreprocessor *CreateInferServerCustomPreprocess() { return new NvInferServerCustomPreProcess(); } }

Postprocessing is quite specific for each model. You’ll need to implement your own postprocessing functions based on inference output tensors results. Custom postprocess libs and functions are specfied from nvdsinferserver’s config. for example in apps/sample_apps/deepstream-3d-lidar-sensor-fusion/model_config_files/config_triton_bev_fusion_infer_grpc.pbtxt

infer_config { backend { triton { model_name: "bevfusion" grpc {...} } } extra { output_buffer_pool_size: 4 # specify custom postprocess function custom_process_funcion: "Nvds3d_CreateLidarDetectionPostprocess" } custom_lib { # specify custom postprocess library path: "libnvds_3d_infer_postprocess_lidar_detection.so" } }

An example of custom postprocess implementation is provided here: sources/libs/ds3d/inference_custom_lib/ds3d_lidar_detection_postprocess/ds3d_infer_postprocess_lidar_detection.cpp

#include "infer_custom_process.h" #include <ds3d/common/hpp/frame.hpp> #include <ds3d/common/hpp/datamap.hpp> using namespace ds3d; using namespace nvdsinferserver; class DS3DTritonLidarInferCustomPostProcess : public IInferCustomProcessor { public: // process key-values from datamap and generate into model inputs batchArray. NvDsInferStatus inferenceDone(const IBatchArray* batchArray, const IOptions* inOptions) override { ...

// get ``ds3d::datamap`` from ``inOptions``
abiRefDataMap* refDataMap = nullptr;
if (inOptions->hasValue(kLidarRefDataMap)) {
    INFER_ASSERT(inOptions->getObj(kLidarRefDataMap, refDataMap) == NVDSINFER_SUCCESS);
}
GuardDataMap dataMap(*refDataMap);
...
// parsing output tensors from batchArray
TensorMap outTensors;
for (uint32_t i = 0; i < batchArray->getSize(); ++i) {
    auto buf = batchArray->getSafeBuf(i);
    outTensors[buf->getBufDesc().name] = buf;
}
std::vector<Lidar3DBbox> bboxes;
ret = parseLidar3Dbbox(outTensors, bboxes);
// warp data into ds3d frame ``bboxFrame``
size_t bufBytes = sizeof(Lidar3DBbox) * bboxes.size();
void* bufBase = (void*)bboxes.data();
Shape shape{3, {1, (int)bboxes.size(), sizeof(Lidar3DBbox)}};
FrameGuard bboxFrame = impl::WrapFrame<uint8_t, FrameType::kCustom>(
    bufBase, bufBytes, shape, MemType::kCpu, 0, [outdata = std::move(bboxes)](void*) {});
// add key-value fram into ds3d::datamap
ErrCode code = dataMap.setGuardData(_3dBboxKey, bboxFrame);
...
return ret;

} }; extern "C" { IInferCustomProcessor *Nvds3d_CreateLidarDetectionPostprocess() { return new DS3DTritonLidarInferCustomPostProcess(); } }

Custom Datafilter libnvds_3d_alignment_datafilter Specifications#

Data alignment for lidar and video data can be done using a custom library provided with DeepStreamSDK.

More details on this is here: Custom ds3d::datafilter library: libnvds_3d_alignment_datafilter.so.

Custom Datafilter libnvds_3d_lidar_preprocess_datafilter Specifications#

LiDAR data preprocess into voxel data format before V2X model sensor fusion inference. Source files located at /opt/nvidia/deepstream/deepstream/sources/libs/ds3d/datafilter/lidar_preprocess

For example:

config_body: mem_pool_size: 4 filter_input_datamap_key: DS3D::LidarXYZI_0 model_inputs:

Custom Dataloader libnvds_lidarfileread Configuration Specifications#

The lib reads lidar data file frame by frame, it creates a new ds3d::datamap per frame and deliver to next compoment. Source files located at /opt/nvidia/deepstream/deepstream/sources/libs/ds3d/dataloader/lidarsource

For example:

config_body: data_config_file: lidar_nuscene_data_list.yaml points_num: 242180 fixed_points_num: True lidar_datatype: FP32 mem_type: gpu gpu_id: 0 mem_pool_size: 6 element_size: 4 output_datamap_key: DS3D::LidarXYZI file_loop: True

The lib reads frames from data_config_file which is a seperate file containers multiple lidar files. each lidar file is a single lidar frame file which are listed inside of source-list. Each item’s keyname is the timestamp(millisecond) of the lidar frame.

An example of data_config_file.

source-list:

Custom Dataloader libnvds_3d_dataloader_realsense Configuration Specifications#

Configuration for Realsense Dataloader Header:

name: realsense_dataloader type: ds3d::dataloader out_caps: ds3d/datamap custom_lib_path: libnvds_3d_dataloader_realsense.so custom_create_function: createRealsenseDataloader

libnvds_3d_dataloader_realsense.so requires you to install librealsense2 SDK. For x86, follow the instructions from IntelRealSense/librealsense. For Jetson platform, follow the instructions from IntelRealSense/librealsense.

Custom datafilter libnvds_3d_depth2point_datafilter Configuration Specifications#

Convert 2D depth data into 3D point-cloud(XYZ) data into ds3d::datamap

name: depth2points type: ds3d::datafilter in_caps: ds3d/datamap out_caps: ds3d/datamap custom_lib_path: libnvds_3d_depth2point_datafilter.so custom_create_function: createDepth2PointFilter

Custom datarender libnvds_3d_gles_ensemble_render Configuration Specifications#

Renders 3D scene using OpenGL ES (GLES) with various elements (textures, LiDAR points, bounding boxes) within a single window, allowing for flexible layout customization. Users can split the windows into multiple views, and project each tensor data or frame from ds3d::datamap into a seperate view or into multiple views at the same time. It could also support overlay if multi frames rendered into a single view positon overlap, it depends on the render graph order.

name: ds3d_sensor_fusion_render type: ds3d::datarender in_caps: ds3d/datamap custom_lib_path: libnvds_3d_gles_ensemble_render.so custom_create_function: NvDs3D_CreateGlesEnsembleRender

An example of a multi-view config with render_graph. There are 2 views rendered with different frames inside the same ds3d::datamap. The 2D color image is rendered by texture3d_render into view area [0, 0, 640, 360]; The Lidar data is rendered by lidar3d_render into view area [640, 0, 1280, 360].

config_body: window_width: 1280 # window size window_height: 360 # window size color_clear: true window_title: DS3D-Lidar-6-Cameras-BEVFusion render_graph: - texture3d_render: # 2D texture view layout: [0, 0, 640, 360] # layout [x0, y0, x1, y1] max_vertex_num: 6 color_clear: false texture_frame_key: DS3D::ColorFrame_2 # image data key - lidar3d_render: # lidar top view layout: [640, 0, 1280, 360] # layout [x0, y0, x1, y1] color_clear: false view_position: [0, 0, 30] view_target: [0, 0, 0] view_up: [0, 1, 0] lidar_color: [0, 0, 255] lidar_data_key: DS3D::LidarXYZI lidar_bbox_key: DS3D::Lidar3DBboxRawData element_size: 4

It supports 2 different modes of views

Each mode could be configured multiple times in the same render_graph. Multiple camera data inside a single ds3d::datamap could be configured into different view area. Similarly, multiple lidar data insdie the same ds3d::datamap could also be configured at different viewpoint(topview, front-view, side-view) into different view area.

Custom datarender libnvds_3d_gl_datarender Configuration Specifications#

Configuration Common header for libnvds_3d_gl_datarender:

name: depth-point-render type: ds3d::datarender in_caps: ds3d/datamap custom_lib_path: libnvds_3d_gl_datarender.so

Configuration Body for Common Part:

name: point-3D-render type: ds3d::datarender in_caps: ds3d/datamap custom_lib_path: libnvds_3d_gl_datarender.so custom_create_function: createPointCloudDataRender # specific function for 3D point rendering

name: lidar-data-render type: ds3d::datarender in_caps: ds3d/datamap custom_lib_path: libnvds_3d_gl_datarender.so custom_create_function: createLidarDataRender # specific function for Lidar point cloud rendering

For more details on 3D coordinate system, refer to https://learnopengl.com/Getting-started/Coordinate-Systems. To know the value meanings for view_position, view_target and view_up,refer to the gluLookAt here: https://www.khronos.org/registry/OpenGL-Refpages/gl2.1/xhtml/gluLookAt.xml. To know the value meanings for near, far and fov, refer to the gluPerspective here: https://www.khronos.org/registry/OpenGL-Refpages/gl2.1/xhtml/gluPerspective.xml.

name: depth-2D-render type: ds3d::datarender in_caps: ds3d/datamap custom_lib_path: libnvds_3d_gl_datarender.so custom_create_function: createDepthStreamDataRender # specific function for 2D depth rendering

libnvds_3d_depth_datasource Depth file source Specific Configuration Specifications#

Configuration header:

name: depthfilesource type: ds3d::dataloader out_caps: ds3d/datamap, framerate=30/1 custom_lib_path: libnvds_3d_depth_datasource.so custom_create_function: createDepthColorLoader

Configuration body:

Configuration Body for Intrinsic Parameters :

Configuration Body for Extrinsic Parameters: