Such systems carry additional burdens that are foreign to more consumer/business-level #MachineLearning systems - in particular, the need to exhaustively quantify "the unseen" through objective analysis.
It is something that, most notably, #Tesla fails to recognize with respect to their #FSDBeta program, likely by design.
This is an untrained human driver operating a vehicle with #FSDBeta active.
The partial automated vehicle, to the excitement of the human driver, appears to successfully negotiate a "Michigan Left" turn at almost the same time that the traffic light turns from red to green.
One of the major, ahem, "selling points" of #Tesla's #FSDBeta program is that untrained human drivers are providing massive, rich "training data" to enhance the reliability of the total Tesla fleet.
These untrained human drivers are able to provide feedback, both manually via on-screen buttons and recently by voice note if the human driver "disengages" the automated system.
Additionally, according to Tesla, the system can send data back to Tesla automatically after a disengagement event.
That is the importance of, #Tesla in this case, maintaining a robust systems safety lifecycle and, for the human test drivers, a robust Safety Management System.
That is, the human test drivers are continuously read into the broader systems aspects of the vehicle under test.
Maintaining such a system converts would-be subjective opinions like the above (which have no technical or safety value) into objective analyses that creates quantifiable reliability progress.
But maintaining such a process is extremely costly and cannot be productized - both being priorities for #Musk and #Tesla.
Therefore, it is not done.
Besides the clear lack of technical value, the danger here is obvious.
With partial automated driving systems (like #FSDBeta), where the human driver is the fallback, the experience shown in the clip above constructs complacency.
That is, a past, possibly coincidental maneuver that appeared successful is not so "lucky" the next time.
What if the furthest oncoming traffic lane (the lane closest the curb) had high-speed traffic instead of slow-moving traffic in this scenario?
Would the FSD Beta-active vehicle have been able to sufficiently capture the necessary physical situation before committing to the left hand turn?
Note how the vehicle in the furthest lane only appears on the visualization on the vehicle HMI after the vehicles in the closest lane move out of the way...
The term "safety" is tossed around quite a bit by Musk, by Tesla and by these untrained human drivers.
But a complete assessment of safety is not only what is seen (the #FSDBeta-active vehicle did not appear to collide with anything), but of the potential, future unhandled failure modes that are unseen.
That is the only pathway of progress, the only pathway towards a continuously safe system.
Anything else, like what is happening with the FSD Beta program, is just goofing around.
Add comment