Amazon Web Services made major changes to its data deletion process after Apple alerted the cloud giant to potential security risks, according to an internal document obtained by Business Insider.
In early 2023, Apple detected unusual activity around data and content associated with its retired cloud accounts on AWS. By April 2023, Apple had escalated its concerns to the AWS security team, asking the cloud giant to investigate whether data and content had been deleted from nearly 2,200 of its AWS accounts that had been locked for more than 90 days. AWS customers expect their data to be permanently deleted 90 days after accounts are closed.
An internal investigation into AWS found that it failed to remove nearly 2,000 pieces of content or metadata associated with terminated Apple accounts, according to the document. It's not clear what specific Apple data was left undelete. However, AWS cloud services include storing information such as software, text, audio, video, images, resource IDs, metadata tags, and permissions.
The oldest content retained was from an account that was terminated in October 2020. Amazon told Apple that it would delete all relevant data by the end of June 2023. The document added that there was no “unauthorized access” to Apple's data.
AWS “bypassed” the deletion of data for some of these Apple cloud accounts and internal “data deletion systems or processes” for some services were disrupted at the time, according to the internal document.
As a result, the AWS Responsible Security Team recommended establishing clear guidelines around “end-to-end data deletion” and designating a director or vice president-level executive responsible for the “correctness of data deletion at AWS,” the document said.
“AWS services retained customer content and resources for longer than 90 days,” the document said. “Today, there is no central monitoring of timely deletion of resources.”
“Fruitstand” = apple
The document doesn't mention Apple specifically. But a separate email obtained by BI refers to this case for a customer named “Fruitstand,” an internal Apple code name, according to people familiar with the matter. These people requested anonymity while discussing a sensitive cloud security issue.
The incident gives a rare behind-the-scenes look at how one of AWS's largest customers pushed security improvements for the cloud giant. It's also worth noting that Apple's marketing places a lot of emphasis on privacy and keeping data secure.
The 23-page document dating back to last June is what Amazon internally calls a “bug correction” report, an in-depth analysis of an incident that the company wants to prevent from happening again. The employees responsible for the report must explain why the incident occurred and how the team plans to remedy it. This particular report, which was marked “privileged and confidential,” was shared by more than a dozen Amazon managers and employees and was updated at least 30 times.
“Data not accessed”
Before publication, AWS spokesperson Patrick Neghorn told BI that “this story has a number of inaccuracies.” He also wrote in an email that Amazon's internal document had “some misconceptions.”
“Our deletions are thorough and have worked almost 100% of the time,” Neghorn added. “In a very small number of cases in the past, deleting a small amount of data – mostly metadata such as configuration data – took longer than intended. There was no data accessed, and we quickly fixed these edge use cases as soon as they were identified.” “
An Apple spokesperson did not respond to requests for comment. Cloud security is becoming more important for Apple. Next week, the iPhone maker is expected to unveil new AI features at its WWDC event, and at least some of those new offerings will have to run in the cloud.
Difficult problem
Cybersecurity experts say cloud vendors in general may have difficulty deleting data associated with terminated accounts.
Customer data is usually scattered around many servers around the world, and it is difficult for cloud providers to track every piece of data in real time. Sometimes they are backed up in multiple data centers, and some applications may not have systems for automatically deleting data, these people said.
“It's a difficult problem,” Justin Capus, a computer science professor at New York University, told BI. “Cloud providers need to treat this as a serious benefit they offer customers because customers should have the right to control their data, where it is distributed, how long it remains, and most importantly, when it is removed.”
“Nightmare” “Shadow Data”
The biggest concern is that this data may be exposed to a third party. Fabrice Delhoste, CTO at security startup Mindflow, said deleting data from expired cloud accounts can be a “nightmare” due to its complexity, and it is the cloud provider's responsibility to ensure all data is safely removed.
“Abandoned cloud accounts and misconfigured services create unmanaged ‘shadow data,’ which can leave sensitive information exposed,” Ken Elefant, managing director of Sorenson Ventures, told BI. “Once abandoned, a lack of oversight and updates increases the likelihood of exploitation over time.
Security is a top priority for AWS
Amazon Web Services CEO Matt Jarman Amazon
For AWS, security has always been a top priority. In a recent blog post, Amazon Cloud's new CEO, Matt Jarman, wrote that ensuring security for customers is “mission zero.”
As more companies and government agencies rely on cloud infrastructure, large vendors have become a prime target for cyberattacks in recent years.
Most notably, a hacking group linked to the Chinese government hacked into Microsoft's cloud service last year. A review backed by the US Department of Homeland Security, published earlier this year, blamed Microsoft for allowing a series of “avoidable errors.”
Microsoft President Brad Smith is scheduled to testify before the House Committee on Cyber Breach next week.
“Canary-like environment”
According to the Amazon document, AWS employees took Apple's inquiry seriously and made several recommendations and changes to improve the internal data deletion process.
The AWS security team suggested taking action to “improve” the quality of the data deletion process and “define clear guidelines” around it. It also has plans for new tracking and escalation mechanisms, a periodic review plan, and a new data deletion form for all types of accounts. Up until that point, AWS had relied on a largely manual audit process.
The team wrote that “failures to delete data after shutdown” raise “customer concerns,” and that the company has not invested sufficiently in the right solutions. She added that having a “canary-like environment that constantly tests whether service teams are deleting their data” would provide the “highest quality signal” to prevent such issues.
The document added that even with these changes, there are still “open risks” regarding complete deletion of all content or metadata, and AWS will need to invest in “additional systems” that continually detect data deletion for each AWS service team.
AWS now continually conducts “automated audits of every AWS service for data deletion compliance,” and any compliance issue receives “immediate attention,” said Nyghorn, the AWS spokesperson. He added that AWS “has always had comprehensive guidelines for the data deletion process,” and has recently made improvements to make it easier for engineers to follow.
Neighorn also said that the few AWS customers who asked about data deletion were “satisfied with the decision we made.” AWS completed all of these fixes and has since implemented “several improvements,” such as automatically detecting whether resources exist after an account is closed, he said.
“As with any other AWS service or function, we are constantly looking for ways to make our guidance more useful,” Neghorn said.
Do you work at Amazon? Got a tip?
Contact reporter Eugene Kim via encrypted messaging apps Signal, Telegram (+1-650-942-3061) or email (ekim@businessinsider.com). Communicate using an unwieldy device. See the Business Insider source guide for other tips on sharing information securely.
