Add nuScenes Dataset Support for 2D Object Detection

[tejasstanley]

PR: Add nuScenes Dataset Support for 2D Object Detection

Summary

This PR adds support for the nuScenes dataset for 2D object detection by introducing the NuScenesDetectionDataset class. It parses 3D bounding boxes from nuScenes into 2D camera views, keeps canonical class labels, and works with the existing tutorials (tutorial_image_detection.ipynb).

Changes Included

• New dataset class: NuScenesDetectionDataset inheriting from ImageDetectionDataset.
• Added dependency on nuscenes-devkit in pyproject.toml.
• Notebooks/examples updated for nuScenes integration.
• Code formatted with Black, type annotations, and Sphinx-style docstrings included.

Notes / Pending Tasks

• Dataset viewer tab integration for nuScenes is in progress and will be added in a follow-up PR.
• Occluded projected boxes are kept as-is for now.
• Working on adding nuScenes LiDAR support and a LiDAR segmentation tutorial; will update once ready.

Testing

• Verified dataset creation, reading annotations, and tutorial execution on nuScenes v1.0-mini.
• Visual inspection of bounding boxes works correctly with current pipeline.

References

• Related discussion: [Add support for nuScenes dataset #393 ]

[dpascualhe]

Hi @tejasstanley . Thanks for your contribution! It will take me a couple of days to review it, but at first sight, the docstrings don't follow the Sphinx style (reference: https://sphinx-rtd-tutorial.readthedocs.io/en/latest/docstrings.html). Can you update that? Also, can you share links so that I can download the dataset myself?

[tejasstanley]

Hi @dpascualhe, thanks for the feedback, I will update the docstrings.
nuScenes can be downloaded by logging in this website https://www.nuscenes.org/nuscenes#download, and selecting the US version of the mini dataset. Direct link can't be copied it seems.

[dpascualhe]

Hi @tejasstanley , great work! It is working for me. Regarding the notebook, can you update it to contain only relative paths so that your home directory is not in it (if you could also fix it for COCO, that would be great!). Also, why are some classes dropped in the dataset? Beyond, and more importantly, for the particular case of 2D object detection, after going through nuScenes, maybe we should pivot towards nuImages (https://www.nuscenes.org/nuimages), so that we don't have problems with projections, occlusions, etc. What do you think? Would you be interested in checking if integrating nuImages for object detection instead of nuScenes makes more sense?

[tejasstanley]

Thanks for the feedback @dpascualhe. Yes I will update the relative paths for the datasets. Regarding the class drops, there were many classes like traffic barriers and etc, which were hampering the visualization. If needed the set can be kept empty also; all classes would be restored.
Regarding nuImages, yes I think nuImages would be a better fit for both the image detection and image segmentation tasks(since masks are available), also since it is derived from nuScenes the lidar segmentation should also work.
Should I start working on replacing the current nuScenes setup with nuImages, or do we want to maintain both setups for now?

[dpascualhe]

Should I start working on replacing the current nuScenes setup with nuImages, or do we want to maintain both setups for now?

I think supporting nuImages for image object detection makes more sense, sorry for not identifying this earlier, but structure-wise your dataset class can be reused I hope!

[tejasstanley]

Hi @dpascualhe, apologies for the delay. You were right, nuImages seems like a much better fit for both segmentation and BBOX detection.
I had to introduce a separate file for nuImages since there are some differences compared to nuScenes, (how key sample tokens are handled). Let me know if the nuScenes support needs to be removed.
Since nuImages provides both segmentation and 2D bounding box annotations, I added support for both tutorials. I also made a small modification in torch_segmentation.py.
Both Rellis3D and nuImages should now work together without any errors. You can test this using the mini dataset available here: https://www.nuscenes.org/nuimages#download.

[dpascualhe]

Awesome! Looks good. I'll download the dataset and test it thoroughly to give you a proper review asap. Thanks!

[dpascualhe]

Still in progress my review, but can you rebase latest changes in master? They had a fix so that you don't have to add that small change in torch_segmentation. Also, can you look for models pretrained with nuImages? It would be nice to have models to properly test it. Regarding nuScenes, I'd get rid of it yes. nuImages is the way to go 😄

[dpascualhe]

It's looking good! I'd get rid of the nuscenes-related changes, and build a new tutorial specific for nuimages (detection and segmentation, while keeping the previous tutorials on image detection and segmentation unchanged. The implementation of the dataset itself is looking fine beyond the minor comments I raised. Make sure to update with the latest changes in master and resolve any conflicts. Upon solving these issues, we can merge.

Good job! 😄

[tejasstanley]

Hi @dpascualhe,
Apologies for the delay in making the changes, and thank you for the review. I have incorporated the suggested updates and synced the branch with the latest commits from master.
Regarding pretrained models, I was not able to find any models trained directly on nuImages.
I was thinking of training a detection model(Faster R-CNN or DETR) on nuImages and sharing the checkpoint I will explore this further and let you know if it is feasible, as I do have access to the required compute resources.