IEEE Big Data 2026 - Proposal: UCF UrbanTwin Sim2Real LiDAR Challenge

Proposed by: Muhammad Shahbaz & Shaurya Agarwal
University of Central Florida
Orlando, Florida.

1. Executive summary

UCF Urbanity Lab proposes a Big Data Cup challenge titled UCF UrbanTwin Sim2Real LiDAR Challenge, run as two parallel tracks against two complementary public roadside-LiDAR datasets:

This challenge asks a scientific question: can synthetic LiDAR data be made realistic enough that a deep perception models trained on it generalises to real LiDAR frames? Participants generate their own synthetic LiDAR data targeting the track's real-data distribution, train a 3D object detector on that synthetic data only, and submit (i) 50 of their synthetic frames, (ii) detections on 50 real held-out frames, and (iii) a signed honor declaration. We score on a 0.6 x detection + 0.4 x distributional-realism combined metric.

The challenge directly addresses two IEEE Big Data 2026 topics of interest: a) Transportation and autonomous driving and b) Smart City and Community, and a third overlapping with Internet of Things Data Analytics (roadside LiDAR is the canonical ITS-IoT sensor).

The data, scoring program, starting kit, declaration template, baselines, and reference resources are hosted at Codabench, so the challenge is ready to release on the Big Data Cup's June 1, 2026 data-release date.

Both tracks are privately available at for you review:

• LUMPI Tracks: https://www.codabench.org/competitions/16684/?secret_key=14163c77-2b76-475a-bfa1-a5d3eee0d5dd
• V2X-Real Track: https://www.codabench.org/competitions/16685/?secret_key=c78db4c1-d51c-459d-ac8e-1aa518b62319

2. Required questions (per the CFP)

2.1 Do you require source code with submissions?

No source code is required at submission time. Each submission to the leaderboard is a single submission.zip containing:

• synthetic/0001.bin - 0050.bin. These are 50 synthetic LiDAR frames (raw float32 N x 3 or N x 4)
• predictions.json. This file contains 3D detections on the 50 released real test frames
• declaration.pdf. It is a signed honor declaration that no held-out real frame was used for training, validation, or hyperparameter tuning.

This light submission keeps the barrier to entry low and lets the scoring program run in under five minutes per submission.

Top-3 teams per track must, however, release source code along with their 6-page challenge report before the IEEE BigData 2026 workshop. The report itself is a separate IEEE Big Data Cup deliverable; the source-code release is a condition of prize eligibility and is needed to verify the integrity of the winning systems.

2.2 What prizes will be offered?

Total prize budget: USD 2,000. Following the CFP convention that cash prizes are shared between competitors and BigData to defray demonstration-hosting costs, the budget is allocated as:

• USD 1,000 - contribution to IEEE BigData 2026 demonstration-hosting expenses.
• USD 1,000 - distributed across winners, divided per track using the standard 50/30/20 split:

The current proposal is not yet sponsor-backed. Organizers will actively seek a corporate or research-foundation sponsor between proposal acceptance and the Development phase opening, and will commit institutional funds as a floor so that the announced prize structure stands regardless of sponsorship outcome. The sponsor list (or "supported by the organizers' institution" fallback) will be confirmed on the challenge page before the Development phase opens.

In addition to cash prizes, every team that submits a valid final-phase entry is offered:

• a 6-page challenge-report slot at the IEEE BigData 2026 workshop session associated with this cup, with top-3 teams given a longer presentation slot, and
• Top-5 reports considered for workshop proceedings and archival.
• an invitation to extend their work into a full paper for a follow-on venue.

2.3 What is the dataset to be released?

We release two complete challenge bundles (one per track) and rely on the two underlying real datasets being already publicly available under their original licenses. We do not redistribute the real data; participants obtain it directly from the source repositories.

(A) Underlying real datasets:

(B) Released by us as the challenge bundle (per track):

• detection_test_frames - 50 real LiDAR frames (point clouds only, labels withheld) that participants run their sim-only detector against.
• detection_test_frame_ids.json - Canonical frame-ID ordering for predictions.json.
• forbidden_frames.txt - 100 frame IDs (the 50 detection-test frames plus 50 held-out realism-reference frames) that participants must not train on, validate on, or hyperparameter-tune on.
• local_eval.py - the exact scoring program that runs on the Codabench worker. Participants can reproduce server scores bit-for-bit locally.
• make_dummy_submission.py - generator of a known-bad submission for format sanity-checking.
• submission_format.md - full submission spec.
• declaration_template.pdf - one-page honor declaration with signature line.
• config.json - public normalization endpoints, point-cloud range, IoU thresholds, seed.

(C) Held out (organizer-only, never released):

• 50 real frames per track used as the realism reference set.
• Ground-truth 3D boxes for the 50 detection-test frames per track.

Reference resources:

• UrbanTwin: Synthetic Roadside LiDAR Datasets (IEEE OJ-ITS 2026),
• High-Fidelity Digital Twins for Bridging the Sim2Real Gap in LiDAR-Based ITS Perception (IEEE T-ITS 2026), and
• UrbanTwin: Building High-Fidelity Digital Twins for Sim2Real LiDAR Perception and Evaluation

Our prior work above describe a complete reference synthesis pipeline. The synthetic datasets are available on: Harvard Dataverse (https://dataverse.harvard.edu/dataverse/ucf-ut). The open-source LiGuard LiDAR processing framework (IEEE T-IV 2026) is publicly released. These are documented starting points for participants who want one; they are not required.

2.4 Which sector does the proposed Data Challenge belong to?

Listed CFP sectors, in decreasing order of fit:

1. Transportation and autonomous driving (primary)
2. Smart City and Community
3. Internet of Things Data Analytics (roadside LiDAR is the canonical infrastructure IoT sensor)
4. Scientific Discovery (the challenge probes whether high-fidelity simulation can substitute for expensive labeled real data, a methodological contribution applicable beyond ITS)

2.5 Who are the organizers?

Lead organizer

• Muhammad Shahbaz, Ph.D. - Postdoctoral Researcher, University of Central Florida
  • Affiliation: Urbanity Lab, Department of Civil, Environmental, and Construction Engineering, University of Central Florida, Orlando, FL, USA.
  • Contact: Muhammad.Shahbaz@ucf.edu
  • Bio: Post-Doctoral Scholar focusing interdisciplinary research across advanced Computer Vision, sensor fusion, and AI, and their applications in the field of Intelligent Transportation Systems and intelligent robotics. He is the lead researcher on the UrbanTwin paper series and datasets, and maintains the open-source LiDAR processing framework LiGuard. He received the B.S. in computer science degree from Pir Mehr Ali Shah Arid Agriculture University Rawalpindi, M.S. degree in computer science from the Pakistan Institute of Engineering and Applied Sciences, Islamabad, Pakistan, and Ph.D. degree in Civil Engineering from University of Central Florida, USA.

Co-organizer

• Shaurya Agarwal, Ph.D. - Associate Professor, University of Central Florida
  • Affiliation: Department of Civil, Environmental, and Construction Engineering; Director, Future City Initiative; Coordinator, Smart City MS Program; Director, Urbanity Lab - University of Central Florida, Orlando, FL, USA.
  • Contact: Shaurya.Agarwal@ucf.edu
  • Bio: Senior Member IEEE, currently an Associate Professor in the Civil, Environmental, and Construction Engineering Department at the University of Central Florida. He is the founding director of the Urban Intelligence and Smart City (URBANITY) Lab, Director of the Future City Initiative at UCF, and serves as the coordinator for Smart Cities Masters program at UCF. He was previously (2016-18) an Assistant Professor in the Electrical and Computer Engineering Department at California State University, Los Angeles. He completed his post-doctoral research at New York University (2016) and his Ph.D. in Electrical Engineering from the University of Nevada, Las Vegas (2015). His B.Tech. degree is in ECE from the Indian Institute of Technology (IIT), Guwahati. His research focuses on interdisciplinary areas of cyber-physical systems, smart and connected transportation, and connected and autonomous vehicles. Passionate about cross-disciplinary research, he integrates control theory, information science, data-driven techniques, and mathematical modeling in his work. As of May 2025, he has published a book, over 37 peer-reviewed journal publications, and multiple conference papers. His work has been funded by several private and government agencies. He is a senior member of IEEE and serves as an Associate Editor of IEEE Transactions on Intelligent Transportation Systems.

2.6 What competition infrastructure will be used?

Codabench (https://www.codabench.org). We request to deviate from the CFP-default Kaggle for the following concrete reasons:

• Both tracks are tested on Codabench. The challenges are tested under our organizer's account (Muhammad.Shahbaz@ucf.edu), with phase dates, leaderboards, scoring programs, reference datasets, and Docker image (codalab/codalab-legacy:py312) already provisioned. UUIDs for the tasks, reference data, and scoring programs are also baked for IEEE BigData 2026.
• The scoring program is non-trivial and not Kaggle-portable as-is. It computes four distributional-realism metrics over masked, seeded subsamples (Chamfer Distance, Maximum Mean Discrepancy with median-heuristic bandwidth, exact Earth Mover's Distance via POT's network simplex, and a Fréchet Point-cloud Distance) and a KITTI-style 3D / BEV mAP at IoU 0.5 and 0.7 over OpenPCDet-formatted oriented boxes, then linearly combines them under min-max normalization endpoints published in config.json so that organizer and participant compute identical scores. Reproducing this exactly on Kaggle is possible but adds engineering work that is configured for Codabench.
• No per-submission fees. Codabench is free for non-commercial academic challenges, which keeps the prize budget allocated to participants and to IEEE BigData rather than to platform fees.
• Custom honor-declaration upload and per-frame submission auditing. Codabench's bundle model lets us require, accept, and inspect the per-submission declaration.pdf and synthetic frames; Kaggle's submission model is geared to single-prediction-file uploads.

Resource commitment by the organizers:

• Compute: scoring runs on Codabench's free public worker queue, which we have profiled at < 5 min per submission for valid inputs. We commit organizer compute as a backstop if queue contention arises.
• Storage: The datasets for both tracks are public and are hosted by their official institutions, UrbanTwin is managed by Urbanity Lab; we commit to maintaining it.
• Moderation and integrity: organizers will manually review the synthetic-frame and declaration artifacts of all top-10 final-phase submissions per track, plus any submission flagged by the leaderboard-leakage heuristics (rules-tab list).

2.7 What are the task and the evaluation metrics?

Task (identical across both tracks, parameterized per real dataset). Given (i) a public real roadside-LiDAR dataset and (ii) 50 unlabeled real test frames from that dataset, participants must:

1. Generate synthetic LiDAR frames targeting the real-data distribution using any approach they like including physics-based simulation (CARLA, OpenScene, etc.), generative models (diffusion, flow-based, etc.), neural-radiance-field renders, or hybrid pipelines.
2. Train a 3D object detector of any architecture on their synthetic data only. No real labels from any roadside-LiDAR dataset.
3. Run that detector on the 50 released real test frames.
4. Submit submission.zip = 50 synthetic frames + predictions on the 50 real test frames + signed declaration.

Evaluation metric (combined score on the leaderboard):

score_combined = 0.6 × score_detection + 0.4 × score_realism

where every component is min-max normalized into [0, 1] using per-track endpoints published in config.json.

• Detection component. score_detection = norm(AP_{3D IoU=0.5}), where AP_3D is the KITTI 40-point-interpolated mean Average Precision computed across the track's classes:
  • LUMPI track: 8 classes (Person, Car, Bicycle, Motorcycle, Bus, Truck, Van, Unknown).
  • V2X-Real track: 3 classes (vehicle, pedestrian, truck).
  Per-class AP is reported in the detailed-results view; classes with zero ground-truth instances in the held-out 50-frame set are excluded from the mean. We also report 3D AP @ IoU 0.7, BEV AP @ IoU 0.5, and 11-point AP variants, all under the OpenPCDet 7-DoF box convention [x, y, z, dx, dy, dz, heading]. IoU is 3D oriented-BEV × Z-overlap.
• Realism component. Concatenate the 50 submitted synthetic frames into P_syn and the 50 held-out real reference frames into P_real, mask both to the per-track point-cloud range (LUMPI: [-40, -40, -5, 40, 40, 5]; V2X-Real: [-40, -40, -8, 40, 40, 2]), then uniform-randomly subsample each side to N = 10,000 points with fixed RNG seed 1234.

Compute four distributional metrics on those two 10,000-point sets:

• Chamfer Distance (CD) - symmetric squared-L2 nearest-neighbor cost. Lower is better.
• Maximum Mean Discrepancy (MMD) - unbiased off-diagonal estimator with RBF kernel; bandwidth γ set by the median heuristic on a 1,000-point joint subsample. Lower is better.
• Earth Mover's Distance (EMD) - Wasserstein-1 with Euclidean ground cost, solved exactly by POT's network simplex; reported in metres. Lower is better.
• Fréchet Point-cloud Distance (FPD) - FID-style 2-Wasserstein between fitted 3-variate Gaussians over the points. Lower is better.

Each is normalized into [0, 1] with the "lower is better -> 1 - clip" sign, then averaged:

score_realism = 1/4 [norm(CD) + norm(MMD) + norm(EMD) + norm(FPD)].

The normalization endpoints are publicly committed in config.json so that the participant's local evaluator and the Codabench worker compute bit-identical scores. Frame ordering does not affect realism (it is a set-level metric); the 50-file count is enforced.

Leaderboard columns (visible to participants): Combined, 3D mAP @ 0.5, 3D mAP @ 0.7, BEV mAP @ 0.5, CD, MMD, EMD, FPD, plus hidden normalized component scores.

Submission rule: Force_Best: only each participant's best submission is shown.

3. Why this challenge belongs at IEEE BigData 2026

• Big data, by every measure that matters in this space. Each underlying real dataset is in the tens-of-thousands of multi-class-annotated LiDAR frames, and a competitive synthetic generator will produce orders of magnitude more frames during development - the participants are doing big-data engineering whether they realize it or not.
• Real, measurable obstacle. Roadside-LiDAR perception for ITS is genuinely bottlenecked by labeled-data cost. A challenge whose top entries demonstrably narrow the sim-to-real gap moves the field; existing baselines (Section 2.7) show the gap is real but not closed.
• Two complementary tracks. LUMPI is multi-perspective European-intersection data with 8 classes including small vulnerable road users; V2X-Real is North-American infrastructure-V2X with 3 broad classes and very different sensor geometry. A method that wins both has demonstrated cross-distribution robustness, which is the actual research question.
• Lowered barrier to entry. A working reference synthesis pipeline (UrbanTwin), ready-to-use synthetic datasets (Harvard Dataverse), and an open-source LiDAR processing tool (LiGuard) are publicly available. Participants can spend their time on the interesting part - closing the gap - instead of on plumbing.
• Workshop integration. Top-3 teams per track are committed to a 4-page challenge report and a presentation at the cup workshop; aggregated analysis of the submission population gives the workshop audience a unique cross-method view of what works in sim-to-real LiDAR today.

4. References

• Shahbaz, M. and Agarwal, S. UrbanTwin: Synthetic Roadside LiDAR Datasets. IEEE Open Journal of Intelligent Transportation Systems, 2026.
• Shahbaz, M. and Agarwal, S. High-Fidelity Digital Twins for Bridging the Sim2Real Gap in LiDAR-Based ITS Perception. IEEE Transactions on Intelligent Transportation Systems, 2026.
• Shahbaz, M. and Agarwal, S. UrbanTwin: Building High-Fidelity Digital Twins for Sim2Real LiDAR Perception and Evaluation. arXiv preprint arXiv:2509.02903, 2025.
• Shahbaz, M. and Agarwal, S. LiGuard. IEEE Transactions on Intelligent Vehicles, 2026 (companion short in Journal of Open Source Software).
• Busch, S., Koetsier, C., Axmann, J., Brenner, C. LUMPI: The Leibniz University Multi-Perspective Intersection Dataset. IEEE Intelligent Vehicles Symposium, 2022.
• Xiang, H., et al. V2X-Real: A Large-Scale Dataset for Vehicle-to-Everything Cooperative Perception. ECCV, 2024.
• OpenPCDet Development Team. OpenPCDet: An Open-Source Toolbox for 3D Object Detection from Point Clouds. 2020.
• Achlioptas, P., Diamanti, O., Mitliagkas, I., Guibas, L. Learning Representations and Generative Models for 3D Point Clouds. ICML, 2018.
• Flamary, R., Courty, N., et al. POT: Python Optimal Transport. JMLR, 2021.