Methodology
How we built Dream, and why.
Dream is the first test of one idea: what makes Indian roads hard for driving software can be measured from dashcam footage, and once it is measured, software can practise against it before it meets the real road. This page sets out that idea, what we tried, what we found and what we have not shown yet.
Our thesis
The hard part is how often things happen.
Most driving software learns on orderly roads. Indian roads are not orderly, but they are not random either. Two-wheelers squeeze past every minute or so, people cross wherever they need to, autos cut in, and now and then a vehicle comes the wrong way. Each of these happens at some rate, and a rate is something you can count.
We collect dashcam footage on Indian roads for a living. So we asked whether that footage alone could tell us how often each of these things happens, and whether a simulator built on those numbers would make driving software face Indian roads more honestly than the ones it is tested on today.
What we set out to test
Four questions, and what we found.
- 1. Can dashcam footage tell us how often things happen?
- Yes, with people doing the labelling. Our team labelled 100 random 4-second clips by hand, each with its main event. We also tried off-the-shelf AI video models on the same clips. The best of them agreed with our people on only about half, so Dream uses the people's labels. Because the sample is small, every rate we give is a range, not a single number.
- 2. Can a simulator reproduce those rates?
- Yes, for all 12 kinds of event we have measured. Our traffic model produces 6 of them at the right rate on its own. An event generator adds the rest, and every event Dream records says whether the traffic produced it or the generator added it.
- 3. Does it change how driving software does?
- Yes. We drove the same textbook planner over the same 100 routes twice. It crashed on 14 of them in orderly traffic and on 27 in Gurgaon traffic.
- 4. Can other teams use it as it is?
- That was the test for everything we added. Dream follows the conventions AV teams already use for training environments and for scoring planners, its map exports to the standard format other simulators load, and its code, map and rates are open.
How it works
From footage to a city you can drive.
- Footage. Our dashcam drives in Delhi NCR, day and night.
- Labels. Random 4-second clips, each given its main event by a person. When several things start in one clip, a fixed priority list decides which counts.
- Rates. How often each event happens per hour of driving, with the range our sample allows.
- Map. Every road, junction and signal in a stretch of Gurgaon about 5 km by 2 km, from OpenStreetMap.
- Traffic. Vehicles that follow, change lanes and filter through gaps the way Indian traffic does, with speed breakers, potholes and bus stops on the roads.
- Events. A generator that makes up whatever the traffic does not produce on its own, up to the measured rates.
- Checks. The simulator counted the same way the clips were labelled and compared with the footage (the realism check), a planner benchmark, and the standard tests for training environments.
Measured and assumed
Where every number comes from.
We would rather say a number is assumed than let it pass as measured. The code marks each one the same way.
Measured or mapped
- How often each of the 12 measured kinds of event happens
- How dense the traffic is, by kind of road user
- How fast our own car drove, from its GPS (the reference for the speed check)
- The roads, junctions, signals and bus stops, from OpenStreetMap
Assumed, until we can measure them
- Lane counts on roads OpenStreetMap leaves untagged
- Most speed breakers, and all potholes
- How often cows stand in the road, dogs run across and vehicles make U-turns
- How fast drivers like to go on each kind of road
What Dream does not show yet
The limits, plainly.
- Driving software here sees positions, not pictures. In Dream, the software gets an exact list of every road user around it. A real car sees camera images. Dream tests the traffic, not the eyes.
- The sample is small. 100 labelled clips give wide ranges. More labelled footage will narrow them.
- One stretch of one city. The rates come from our Delhi NCR footage and the map is a part of Gurgaon.
- Simulated cars stand still too often, partly because of the cautious textbook driver we tested with and partly because of how unsignalled junctions clear.
What comes next
Driving software that sees what a car sees.
Our next step closes the biggest gap above: an environment where the software receives camera images from our footage, as it would on the road. First, it watches real frames from our four-camera rig, plans the next few seconds, and we score that plan against what actually happened. Then a world model paints the camera view of Dream's scenes, so the software can drive in closed loop and the traffic reacts to it.
Dream is where we will put this work as it lands: training environments, world models and the benchmarks behind them. If you would like to help label footage, test your own planner, or need data from the roads you ship into, we would like to hear from you.