GitHub - kesai-labs/py123d: 123D: A Unified Library for Multi-Modal Autonomous Driving Data (original) (raw)
123D: Unifying Multi-Modal Autonomous Driving Data at Scale
Paper | Video | Documentation
One library for autonomous driving datasets. 123D converts raw data from Argoverse 2, nuScenes, nuPlan, KITTI-360, PandaSet, and Waymo into a unified Apache Arrow format, and then gives you a single API to read cameras, lidar, HD maps, and labels across all of them.
📰 News
- 2026-05-28: Released NAV123D, a reimplementation of NAVSIM on top of 123D.
- 2026-05-08: Our paper 123D: Unifying Multi-Modal Autonomous Driving Data at Scale is out on arXiv.
✨ Features
- Dataset download: Fetch supported datasets from their official sources via the CLI, and optionally convert directly into the unified format.
- Hydra-based conversion CLI: YAML configs to manage your data pipelines.
- Apache Arrow storage: columnar, memory-mapped, zero-copy reads. Fast and memory efficient.
- Multiple sensor codecs: MP4/JPEG/PNG for cameras; LAZ/Draco/Arrow IPC for lidar.
- No sensor duplication: Converted logs reference source camera/lidar files via relative paths, so you don't store sensors twice.
- Unified API: Read cameras, lidar, maps, and labels through a single interface, regardless of the source dataset.
- Built-in visualization: interactive 3D viewer (Viser), and matplotlib plotting.
📦 Installation
Per-dataset extras (e.g. py123d[av2], py123d[nuscenes], py123d[waymo]) install the parser dependencies for each dataset on demand. See the Demo below for an example.
🚀 Demo
Demo using the Argoverse 2 Sensor dataset, which is publicly readable from S3 and requires no cloud authentication.
The av2-sensor-stream config downloads the requested logs/maps into a managed temp directory, converts them into our self-contained Arrow format, and cleans up the source files afterwards. PY123D_DATA_ROOT controls where the converted logs/maps are written. The script below installs the Av2 extra, converts the first 3 validation logs (~250 MB each), and launches the Viser viewer:
1. Install
pip install py123d[av2]
export PY123D_DATA_ROOT=/path/to/py123d_data
2. Download + Convert
py123d-conversion dataset=av2-sensor-stream
dataset.parser.splits='[av2-sensor_val]'
dataset.parser.downloader.num_logs=3
3. Launch Viewer
py123d-viser scene_filter=av2-sensor
Open http://localhost:8080 to browse the converted scenes interactively.
🖼️ Viewer
📊 Supported Datasets
| Scale | Sensors [#/Hz] | Annotations [✓/Hz] | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | Year | Dur. [h] | Dist. [km] | Logs [#] | Cam. | LiDAR | 3D Box | Tls. | Map | |
| Manual | nuScenes | 2020 | 5.6 | 100.9 | 1,000 | 6 / 12 | 1 / 20 | ✓ / 2 | ✗ | ✓ |
| WOD-Perc. | 2020 | 6.4 | 154.0 | 1,150 | 5 / 10 | 5 / 10 | ✓ / 10 | ✗ | ✓ | |
| AV2-Sens. | 2021 | 4.4 | 87.5 | 1,000 | 9 / 20 | 2 / 10 | ✓ / 10 | ✗ | ✓ | |
| PandaSet | 2021 | 0.2 | 8.3 | 103 | 6 / 10 | 2 / 10 | ✓ / 10 | ✗ | ✗ | |
| KITTI-360 | 2022 | 2.7 | 73.7 | 9 | 4 / 10 | 1 / 10 | ✓ / 10 | ✗ | ✓ | |
| Auto-labeled | WOD-Mot. | 2021 | 574.1 | 10,323.5* | 103,354 | ✗ | ✗ | ✓ / 10 | ✓ / 10 | ✓ |
| nuPlan | 2024 | 1,174.3 | 17,808.6 | 15,910 | 8 / 10† | 5 / 20† | ✓ / 20 | ✓ / 20 | ✓ | |
| – nuPlan-mini | 2024 | 7.2 | 103.0 | 64 | 8 / 10† | 5 / 20† | ✓ / 20 | ✓ / 20 | ✓ | |
| PAI-AV | 2025 | 1,707.0 | 69,265.7 | 307,332 | 7 / 30 | 1 / 10 | ✓ / 10 | ✗ | ✗ | |
| – NCore | 2026 | 6.3 | 167.6 | 1,147 | 7 / 30 | 1 / 10 | ✓ / 10 | ✗ | ✗ | |
| Synth. | CARLA | 2017 | var. | var. | var. | var. | var. | ✓ | ✓ | ✓ |
| – L3AD | 2026 | 7.3 | 138.7 | 789 | 6 / 10 | 2 / 10 | ✓ / 10 | ✓ / 10 | ✓ |
* Computed only from the non-overlapping 20 s training files. † Released for a 120 h subset; full coverage on mini.
📝 Changelog
v0.5.0 (2026-05-28)
- New
Polyline2D/PolylineSE2/PolylineSE3.subline()to extract a sub-polyline between two distances along a path. - New
MapAPI.get_layer_graph()returning anetworkx.DiGraphof lane / lane-group predecessor–successor topology. - Added
BoxDetectionsSE3.box_detections_se2for projecting SE3 detections to SE2. - Global runtime settings consolidated into
common.runtime, configured via Hydra, with configurable LRU cache sizes (e.g.max_lru_cached_tables). - Added a minimal PyTorch dataloader example (
examples/01_torch_dataloader.py). - Fixes: nuPlan map predecessor/successor naming for lanes and lane connectors;
np.nanlane speed limits; scene-sampling stride withtarget_iteration_stride; matplotlib observation rendering.
Breaking changes: OccupancyMap2D.contains_vectorized renamed to contains_points_2d (now shape-preserving); dataset_paths moved from py123d.common to py123d.common.runtime.
v0.4.0 (2026-05-19)
- Added L3AD dataset support (CARLA-derived) with a HuggingFace-hosted downloader that fetches pre-converted Arrow logs directly into
$PY123D_DATA_ROOT. - New
SceneAPI.get_modality_between_timestamps()for time-windowed retrieval. - nuPlan route and lidar tokens exposed through the custom-modality interface.
- Visualization: expanded viser color and rendering options, with configurable output resolution and frame rate.
- Parser fixes for PAI-AV and ncore: bounding box alignment and file-derived vehicle dimensions.
No breaking changes to the public API, Arrow schema, or CLI entry points.
v0.3.0 (2026-04-28)
- Refactored dataset download interface, with new download/stream options for nuScenes, PandaSet (HF mirror), AV2-sensor, WOD-perception, WOMD (#126), and nuPlan.
- Added ncore dataset support with parser, downloader, and on-the-fly conversion (#125).
- Waymo Open Motion: remaining WOMD splits, WOMD-specific fields via custom modalities, and skip-logs / skip-map options for storage-constrained runs.
- Map improvements: speed bumps as surface map objects in Waymo (#130);
align_road_edges_to_trafficfor traffic-aligned road edges (#123). - Parser fixes across PandaSet (lidar/ego poses, extrinsics), nuPlan (#128), KITTI-360 labels, nuScenes map path in stream mode, WOD motion streaming, and
pai-av/ncorelabels. - Runtime and packaging: corrected sync-table entries when inferring dynamics; Ray executor compatibility;
google-cloud-storagemoved to the Waymo extra.
Includes all fixes from v0.2.1 and v0.2.2. No breaking changes to the public API, Arrow schema, or CLI entry points.
v0.2.0 (2026-04-14)
- Transferred repository to KE:SAI.
- Aligned ego and agent dynamics to a unified global/ego-frame convention, with velocity/acceleration inference in
LogWriterfrom poses forEgoStateandBoxDetectionsSE3(#119, #120). - Improved OpenDRIVE maps: 3D road-edge lifting, lane-boundary reconstruction, and cleaner map-metadata location handling (#121).
- Parser and visualization fixes: NuScenes interpolated parser defaults to 10 Hz sync with camera-pose interpolation; PandaSet extrinsic/undistortion fixes; viser fixes (#117); new matplotlib camera-rig and lidar-reprojection utilities.
No breaking changes to the public API, Arrow schema, or CLI entry points.
v0.1.0 (2026-03-22)
- Asynchronous (native-rate) data storage: modalities are now written at their original capture rate, not just at the a frame-wise rate.
- New parser architecture with
BaseLogParser.iter_modalities_asyncfor native-rate iteration alongside the existing synchronized path. - Added NVIDIA Physical AI AV dataset support (experimental).
- Added standalone OpenDRIVE / CARLA map parser.
- Refactored
conversion/module intoparser/with consistent naming across all dataset parsers. - Refactored Viser 3D viewer. Adds more control and dark mode.
- Added
LaneType,IntersectionType,StopZoneTypeto map data structure. - Replaced Waymo heavy dependencies with lightweight protobufs.
- Various fixes to camera-to-global transforms across all datasets. v0.0.9 (2026-02-09)
- Added Waymo Open Motion Dataset support.
- Replaced gpkg map implementation with Arrow-based format for improved performance.
- Added sensor names and timestamps to camera and Lidar data across all datasets.
- Added ego-to-camera transforms in static metadata.
- Implemented geometry builders for PoseSE2/PoseSE3 from arbitrary rotation/translation representations.
- Added support for loading merged point clouds in API.
- Improved map querying speed and OpenDrive lane connectivity handling.
- Added recommended conversion options to dataset YAML configuration files.
- Fixed PandaSet static extrinsics and KITTI-360 timestamp handling.
- Fixed memory issues when converting large datasets (e.g., nuPlan). v0.0.8 (2025-11-21)
- Release of package and documentation.
- Demo data for tutorials.
📚 Citation
@article{Dauner2026ARXIV, title={123D: Unifying Multi-Modal Autonomous Driving Data at Scale}, author={Dauner, Daniel and Charraut, Valentin and Berle, Bastian and Li, Tianyu and Nguyen, Long and Wang, Jiabao and Jing, Changhui and Igl, Maximilian and Caesar, Holger and Ivanovic, Boris and Geiger, Andreas and Chitta, Kashyap}, journal={arXiv preprint arXiv:2605.08084}, year={2026} }
⚖️ License
123D is released under the Apache License 2.0.