The "behind-the-scenes workers" of Tesla's autonomous driving: monotonous work under surveillance, dare not go to the toilet

Phoenix Technology (ID: ifeng_tech), Editor: Xiao Yu, Original Title: "Unveiling the 'Behind-the-Scenes Workers' of Tesla's Autopilot: Monotonous Work Under Surveillance, Dare Not Go to the Bathroom", Image Source: AI Generation

For Tesla vehicles, when it is snowing, when should the car start braking at the parking sign? When should the turn signal be turned on? How to distinguish between traffic lights and a full moon?

These are just some of the daily issues handled by Tesla's Autopilot team. Tesla's assisted driving system relies on an army of data annotators who review thousands of hours of videos from Tesla owners and internal test drivers. These annotators gradually teach Tesla's artificial intelligence (AI) how to drive like a human driver, teaching only a 30-second video each time.

Tesla has hired many annotators who work full-time at the company, earning an hourly wage of about $20 (approximately 142 RMB).

The US magazine "Business Insider" interviewed 17 current and former employees of Tesla's data annotation team to understand the working conditions of this team that supports Tesla's Autopilot and Full Self-Driving (FSD) systems behind the scenes. The team is distributed across three different Tesla facilities in Buffalo, New York, Palo Alto, California, and Draper, Utah.

The projects assigned to the team have different deadlines, some may take months, while others only require a few days. The work processes also vary, with some requiring staff to annotate short video clips, while others require tracking static images or overlaying satellite data.

"Sometimes the work is very monotonous," a former employee said, "You might spend 8 hours a day for several months annotating lane lines and curbs in thousands of videos."

Peeking into Others' Lives

These video clips provide a unique window into understanding the daily lives of Tesla owners. Five annotators mentioned that once, a project required them to annotate data obtained from some owners' garages through Tesla's Sentry Mode feature.

According to two annotators, another project called "Selfie" required some annotators to annotate data captured from Tesla's in-car cameras. Four other annotators stated that they were aware of this project, where the Selfie program aims to teach Tesla's system how to recognize when drivers are not paying attention to the road while using Autopilot.

Tesla states in the owner's manual that the in-car cameras "will share short video clips with Tesla to help us develop future safety enhancements and continuously improve the intelligence of functions relying on in-car cameras." Tesla emphasizes that **owners must first choose to share their data for Tesla annotators to access these videos **

"Business Insider" previously reported that in other cases, annotators may find themselves annotating data related to YouTube influencers or even Elon Musk's travel routes.

"Looking at others' daily driving feels strange," a current Tesla annotator said. "It feels weird to peek into someone else's life, but it is also an important part of correcting and improving the program."

Fifteen annotators said that these videos come from various locations in the United States, as well as some areas in Europe and South America. Two annotators recalled annotating videos from Ukrainian car owners, coinciding with Russia's military actions against Ukraine.

"Business Insider" reached out to Tesla, Musk, and his legal team for comment, but did not receive a response before publication.

Annotators in a workflow may encounter data from any country, which means they must constantly understand the different road rules in each region. Seven former and current annotators said that Tesla sometimes seems to take a more lenient approach to these rules. For example, some annotators said they were told to ignore signs like "No Right Turn on Red" or "No U-Turn," meaning they are not training the system to obey these signs.

"It's a customer-centric mindset," a former annotator said. "I think our idea is that we want to train cars to drive like humans, not like robots that only follow rules."

Sometimes, annotators need to annotate videos from car accidents and near-accident scenes. Seven annotators recalled annotating accidents involving Tesla or neighboring vehicles. Four annotators said that once, an annotator shared a video among colleagues showing a young boy on a bicycle being hit by a Tesla, which became one of the many videos and emojis they used for communication.

Last year, Reuters first reported on the potential privacy issues of this bicycle video and Tesla's annotation site. Nine annotators said that shortly after the article was published, Tesla began restricting annotators' access to videos outside of designated projects and added watermarks to some videos and images to easily track which employees were sharing images.

Employee Monitoring System

Tesla has set up a fairly strict employee monitoring system in the Buffalo factory. Eleven annotators told "Business Insider" that there is a row of surveillance cameras overlooking the entire work area.

Tesla also uses two different software systems to closely monitor employees. Four annotators said that one of the systems is called HuMans, used to evaluate how much time they should spend on each video clip. Annotators who work continuously for too long within the specified time frame may receive a poor rating or be included in a Performance Improvement Plan (PIP). The software was originally designed to assist U.S. Air Force pilots and also has the ability to track employees' eye movements and recordings, but it is unclear if Tesla uses the software to track employees' eye movements According to 17 annotators, Tesla also uses a time metric called "Flide Time" to track annotators' active time on the annotation software. It can track the number of keystrokes by annotators and the time spent on the open annotation software, but it does not track the time annotators spend using other tools on the computer. Depending on the level of annotators, they need to record 5 to 7.5 hours of Flide Time, which means they must remain active on the software for at least that long.

Six annotators stated that even if their active time is five minutes less than the specified time, they may face disciplinary action. If they fail to meet the standard three times in six months, they may be dismissed. Some annotators have tried to oppose Tesla's set assessment metrics, but with little success.

In February 2023, some employees at Tesla's Buffalo factory attempted to form a union. Union organizers at the Buffalo factory told Bloomberg that Tesla tracks the time employees spend on each task by monitoring keystrokes, as well as the time they actively work each day, leading some people to refrain from using the restroom. "People are tired of being treated like robots," said Al Celli, a member of the union organizing committee.

In the same month, Tesla fired dozens of workers at the Buffalo factory. At that time, the National Labor Relations Board (NLRB) filed a lawsuit accusing Tesla of illegally firing some employees "in retaliation for union activities and to discourage union activities." However, Tesla denied these allegations, stating that these employees were dismissed due to poor performance. The NLRB has not responded to the current status of this lawsuit.

According to Reuters, when Tesla began building its assisted driving program in 2016, the company outsourced its data annotation business to a company based in California with an office in Kenya, but Tesla brought the project back in-house in 2019.

Recently, the Tesla Autopilot team was affected by company-wide layoffs in April. According to the Worker Adjustment and Retraining Notification Act (WARN) notice, Tesla laid off nearly 300 people in Buffalo.

Tesla has stated that the company's neural network will one day be able to self-train, but for now, it relies on human power.

This work is crucial for Musk to achieve the vision he has set for Tesla. Over the years, he has repeatedly emphasized the importance of Tesla's efforts to achieve autonomous driving. In 2022, Musk stated that the value of Tesla is whether it is worth a lot of money or basically worthless, depending on autonomous driving technology.

Tesla plans to launch a self-driving taxi service later this year, which is expected to be built on the same autonomous driving software as the company, along with the lengthy, line-by-line analysis of video clips by its annotators