Modern autonomous vehicles are equipped with an increasing number of cameras! Why do several cameras need to repeatedly monitor the same area when a vehicle is already fitted with so many? The reason lies in the fact that what autonomous driving truly needs to solve is not just whether it can see, but also whether the system can still obtain sufficient environmental information when a certain field of view is unclear, invisible, or even fails. Visual redundancy means ensuring that critical areas have more than one source of visual observation.
01. Visual Redundancy: What Exactly Is Being Redundant?
When it comes to visual redundancy, many people's first reaction is to increase the number of cameras. However, quantity is merely the outcome; what truly matters is whether different cameras form reasonable overlapping coverage and complementary capabilities. For instance, cameras with different fields of view (FOV) can be used at the front of the vehicle.
Cameras with a narrower FOV are more suitable for observing long-distance areas, while wide-angle cameras can cover a larger range, and side cameras are responsible for supplementing the areas on both sides and at close range of the vehicle. The scenes captured by these cameras do not need to be completely identical; instead, overlapping FOVs can be allowed in some critical areas. In this way, the same target may enter the fields of view of multiple cameras from different positions and angles, which also reduces the system's reliance on a single perspective.
For example, when the vehicle turns right, a pedestrian may appear simultaneously in the fields of view of both the front and right-side cameras. The two perspectives do not mean the system simply identifies the target twice or chooses between the two views; rather, these two perspectives provide complementary image information and spatial constraints, offering more basis for target detection, localization, and motion judgment.
Therefore, visual redundancy is not merely about repeatedly capturing a single target, but about ensuring that critical areas have multiple sources of visual observation. This is also the difference between multi-camera setups and visual redundancy. Multi-camera setups can be achieved by expanding the perception coverage range, whereas visual redundancy places more emphasis on overlapping observation of critical areas and retaining other observation sources even after some visual information fails.
02. Why Is It Difficult for a Single Camera to Independently Handle All Visual Perception?
Cameras can provide rich visual information such as color, texture, road markings, traffic signs, and target appearance, but their imaging quality is highly susceptible to actual environmental factors like lighting, weather, and occlusions. Backlighting may cause local overexposure, nighttime reduces effective visual information, rain and dirt may affect imaging, and large vehicles ahead may directly occlude small targets. For long-distance targets such as pedestrians and traffic cones, even if they are not occluded, their effective pixels in the image may be very limited.
Furthermore, cameras acquire 2D image information, while the vehicle needs to further estimate the spatial position and motion state of targets from images, temporal changes, multiple perspectives, and other sensor information. Adding cameras at different positions can provide more observation angles and spatial constraints.
Thus, visual redundancy solves not only the issue of seeing wider but also reduces the uncertainty brought by a single perspective. The redundancy here can be understood in two dimensions. One is redundancy at the information level. Different cameras observe the environment from different directions, providing complementary information so that the system does not have to rely entirely on a single perspective. The other is redundancy at the fault level. When a certain camera is affected by occlusion, contamination, or partial failure, if other visual channels also cover the relevant areas, the system may still retain some effective observations.
However, this does not mean that if one camera fails, another will definitely take over. If two cameras are simultaneously affected by the same rain, dirt, or strong light, or if the target does not enter the effective FOV of the other camera at all, adding cameras may not necessarily form effective redundancy. Therefore, true redundancy depends not only on whether there is a second camera but also on whether there is sufficient independence between different visual paths.
03. Why Can't the Number of Cameras Be Increased Infinitely?
If more cameras mean safer, then the simplest solution to improve autonomous driving safety would be to continuously increase the number of cameras. But the reality is not so. Each additional camera means more image data to be processed, which simultaneously increases data transmission and computational pressure. The data volume brought by high-resolution cameras is particularly significant. Meanwhile, multiple cameras also require time synchronization and extrinsic calibration; the system needs to know the position and corresponding time of each image stream, as well as the spatial relationships between different cameras.
More importantly, the capabilities of different cameras are not exactly the same. Cameras with a narrower FOV can concentrate more pixels on long-distance areas, but their coverage range is limited; wide-angle cameras can cover larger areas, but they are not necessarily suitable for small targets at long distances.
Therefore, the actual perception architecture is not a simple replication of identical cameras, but a combination based on different areas and tasks. Here, what is truly worth discussing is not how many cameras should be installed on the vehicle, but which areas require redundancy, and to what extent the redundancy is worth paying the extra computing power, bandwidth, power consumption, and hardware costs. Redundancy design pursues reasonable coverage and fault tolerance, rather than infinitely increasing the number of sensors.
04. Two Cameras Do Not Necessarily Mean True Redundancy
There is another easily overlooked issue, which is the risk of common-source failure. Suppose two cameras are both installed behind the windshield. If a water film forms on the glass surface or a large amount of dirt adheres to it, both cameras may be affected simultaneously.
For another example, although two cameras are physically two independent sensors, if they rely on the same critical computing module, power supply link, or data processing stage, then when this common node fails, both visual channels may be affected simultaneously. Therefore, redundancy is not just about having two in quantity; the independence between different perception paths must also be considered. For redundant signals, attention must also be paid to related faults and fault detection capabilities, rather than assuming that the two data streams are inherently independent.
This is why true visual redundancy involves sensor placement, FOV, hardware links, computing resources, and software fault handling, rather than merely adding a few cameras. NIO's currently publicized Aquila perception system is a relatively typical industry case. NIO officially describes it as having 360° visual redundancy, with public configurations including seven 8MP high-resolution cameras, four 3MP surround-view cameras, as well as perception units such as LiDAR, millimeter-wave radar, and ultrasonic sensors. It should be noted that this is merely NIO's architectural design concept for visual redundancy and does not mean that all models adopt exactly the same sensor configuration under any circumstances; the actual vehicle configuration is still subject to the specific model.
05. Why Is Visual Redundancy Still Important in the End-to-End Era?
End-to-end autonomous driving changes the information processing methods among perception, prediction, and decision-making, but it does not change the physical limitations of the sensors themselves. Models can directly process multi-camera data and learn the motion patterns of targets from continuous images, but if a certain area lacks effective observations due to occlusion, dirt, or camera failure, the model cannot obtain real-world information out of thin air.
Therefore, in the end-to-end era, visual redundancy remains important. It is also necessary to distinguish between visual redundancy and multi-sensor complementarity. The front and side cameras forming a certain degree of overlap on the same area belong to the redundancy and complementarity within the visual system; cameras and LiDAR or millimeter-wave radar, on the other hand, mostly utilize different perception principles to provide complementary information. The latter can certainly participate in the redundancy design of the overall perception system, but cannot be simply equated with visual redundancy.
Thus, what visual redundancy truly pursues is not that more cameras mean safer, but to ensure reasonable overlapping observations in critical areas and minimize the impact of a single visual path on the overall perception results. For autonomous driving, reliable visual perception is not just about seeing clearly under normal conditions, but also considering whether the system still has other observation sources to utilize when one perspective cannot see.
#AutonomousDriving #VisualRedundancy