In 2026, the Waymo research team released the WOD-E2E (also known as Rare-E2E) end-to-end driving dataset at CVPR 2026. This dataset extracts 4,021 driving clips, totaling approximately 12 hours, from about 6.4 million miles of real-world road testing. It specifically curates long-tail scenarios that occur with a frequency of less than 0.03% in daily driving, and introduces the RFS (Rater Feedback Score) evaluation metric to measure whether driving behaviors are reasonable and stable. In fact, once an end-to-end autonomous driving system can stably handle a large number of common driving scenarios, what truly exposes the boundaries of the model's capabilities are not those repeatedly occurring normal roads, but these rare, complex, and error-prone scenarios.
01. Why Do More Mature Systems Need to Proactively Seek Out Errors?
End-to-end autonomous driving requires learning the relationship between environmental changes and vehicle behaviors from massive amounts of driving data. High-frequency scenarios such as car-following, lane-changing, and passing through regular intersections can yield abundant training data through continuous accumulation. However, there is another category of data on real roads: they occur infrequently but are more likely to expose system vulnerabilities. Examples include a lead vehicle suddenly cutting in, pedestrians emerging from occluded areas, construction altering the original road structure, or multiple traffic participants making unexpected maneuvers simultaneously.
These scenarios account for a very small proportion of massive normal driving data, yet they can directly determine the model's performance in complex environments. This leads to an urgent problem to be solved: the more common the scenario, the easier it is to obtain abundant data, whereas the rarer the scenario, the more data-deficient it is. Therefore, once common scenarios have been accumulated to a certain scale, continuously adding a large amount of highly similar normal driving data may yield diminishing marginal value in discovering the model's insufficient long-tail capabilities. At this point, greater attention should be paid to data that can expose model flaws.
02. Truly Valuable Data Does Not Necessarily Involve Actual Mistakes
If only data involving collisions or obvious erroneous vehicle decisions are extracted for training, it is still far from sufficient. What autonomous driving truly needs to focus on is identifying the model's weak points as much as possible before errors cause severe consequences. For instance, a vehicle might eventually pass through a complex intersection safely, but when faced with the simultaneous occurrence of occlusions, jaywalking pedestrians, and oncoming vehicles, the driving behavior output by the model is noticeably unstable. Although this data did not result in an accident, it can be highly valuable because it exposes the model's instability when facing specific combinations of scenarios.
There is also other data where the vehicle ultimately completes the driving task, but with abrupt braking, frequent trajectory adjustments, or noticeably different driving behaviors in similar scenarios. These situations are equally worth mining and analyzing. From the perspective of a data closed-loop, such data is more valuable than simply recording what happened.
At this point, it is necessary to review this type of data: is it an isolated incident, or a recurring problem in a certain category of scenarios? Is it due to insufficient perception information, or an inadequate understanding of scenario relationships by the model? Is it due to insufficient coverage of training data, or a generalization issue of the model under certain conditions?
In other words, what is truly worth mining is not just data where mistakes have already been made, but data that can help engineering teams figure out why those mistakes occurred. Edge cases are precisely an important category among them. Research related to Safety of the Intended Functionality (SOTIF) also points out that although these scenarios occur infrequently, they can expose the functional limitations of autonomous driving systems and pose greater challenges to system safety. The RFS metric proposed by WOD-E2E is designed to address this very issue. In long-tail scenarios where traditional distance error metrics fail to capture the multimodality of driving behaviors, RFS introduces human preference scores for driving behaviors, shifting the evaluation from "whether the trajectory looks similar" to "whether the driving is reasonable and stable."
03. How Should Data Be Utilized After Identifying Errors?
For end-to-end autonomous driving, finding an edge case does not mean it can simply be thrown into the training set. Data generated from real roads is highly complex, and the same abnormal performance may correspond to completely different underlying issues. After data mining, it is necessary to analyze the data in combination with vehicle behavior, model outputs, and scenario features to determine whether the sample is an isolated incident or follows a recurring pattern. If the same problem occurs across a large number of similar scenarios, its value will significantly increase. Engineering teams can supplement data for such scenarios, refine annotations, and incorporate them into the training and testing processes.
After the model is updated, it is necessary to re-verify whether the original problem has been improved, while also checking if any new issues have been introduced. Only then can a complete data closed-loop be formed: real-world road operation → anomaly discovery → hard data mining → problem analysis → training and testing → re-verification. The most difficult part of this closed-loop is not actually collecting data, but finding the parts truly worth learning from massive amounts of data. This is also why, as end-to-end autonomous driving enters a more mature stage, simply pursuing the scale of the training set is no longer the primary focus. This does not mean data scale is unimportant; rather, while continuously accumulating data, there is an increasing need to proactively seek out data that can expose the model's shortcomings. Because for a system that can already handle a large number of routine scenarios, what truly determines the room for the next improvement is not how much it already knows, but in what situations it is still prone to making errors, and whether these errors can be turned into data for the next round of model improvement.
#AutonomousDriving #EndToEndLargeModels #EdgeCases