Why Apple’s child safety updates are so controversial?

Last week, Apple foreseen a series of updates designed to reinforce child safety features on their devices. Among them: a new technology that can scan the photos on the users’ devices to detect child sexual abuse material (CSAM). Although the change was widely praised by some legislators and child safety defenders, it caused an immediate envelope of many security and privacy experts, who say that Apple’s update amounts walk their commitment to put the privacy of the user above all.

Apple has played that characterization, saying that its approach balances both privacy and the need to do more to protect children by preventing some of the most abominable content from extending more widely.

What announced Apple?

Apple announced three separate updates, all of which fall under the umbrella of “Child Safety”. The most significant, and the one who has received most of the attention, is a feature that will scan the photos of iCloud to know the known CSAM. The characteristic, which is integrated into the photos of iCloud, compares the photos of a user against a previously identified material database. If a certain number of these images is detected, triggered a review process. If the images are verified by human reviewers, Apple will suspend that the ICLOD account and inform the National Center for Fald and Exploited Children (NCMEC).

Apple also previews the new “Communication Security” functions for the Message application. That update allows the message message to detect when children send or receive sexually explicit photos. It is important to note that this feature is only available for children who are part of a family account, and depends on parents to opt for them.

If parents enter the function, they will be alertified if a child under 13 sees one of these photos. For children over 13, the messages application will show a warning when receiving an explicit image, but it will not alert your parents. Although the function is part of the application of messages, and separated from the detection of CSAM, Apple has indicated that the function could still play a role in stopping child exploitation, since it could interrupt predatory messages.

Finally, Apple is updating SIRI and its search capabilities so that it can “intervene” in CSAM consultations. If someone asks how to report abuse material, for example, Siri will provide links to resources to do so. If you detect that someone might be looking for CSAM, it will show a warning and surface resources to provide help.

When is this happening and you can choose to leave?

The changes will be part of iOS 15, which will be deployed later this year. Users can choose to separate effectively by disabling the photos of iCloud (instructions to do so can be found here). However, anyone who disables icloud photos should bear in mind that it could affect it to access the photos on various devices.

So, how does this image work the scan?

Apple is far from the only company that scans photos to look for CSAM. Apple’s approach to do so, however, is unique. The detection of CSAM is based on a database of known material, maintained by NCMEC and other safety organizations. These images are “hashed” (the official name of Apple for this is Neuralhash): a process that converts images to a numerical code that allows them to identify themselves, even if they are modified in some way, such as cutting or making other visual editions. As mentioned above, the detection of CSAM only works if the iCloud photos are enabled. What has been remarkable about Apple’s approach is that instead of matching the images once they have been sent to the cloud, since most of the cloud platforms do: Apple has transferred that process to the devices of the users.

This is how it works: The hashes of the known CSAM are stored on the device, and the photos on the device are compared with those hashes. The iOS device generates an encrypted “security voucher” that is sent to iCloud along with the image. If a device reaches a certain threshold of CSAM, Apple can decrypt the security vouchers and perform a manual review of those images. Apple is not saying what the threshold is, but it has made it clear a single image would not be any action.

Apple also published a detailed technical explanation of the process here.

Why is it so controversial?

Privacy advocates and security researchers have raised a series of concerns. One of these is that this feels like an important investment for Apple, which was five years ago rejected the FBI request to unlock a phone and has placed posters indicating “what happens on your iPhone stays on your iPhone”. For many, the fact that Apple created a system that can proactively verify their images by illegal material and refer them to the police, feels as a betrayal of that promise.

In a statement, the Electronic Foundation called it “an impact on the face for users who have been based on the company’s leadership in Privacy and Safety.” In the same way, Facebook, which has been years, taking Apple’s warmth on your privacy errors, has taken itself with the focus of the CSAM iPhone manufacturer. The WHATSAPP chief, Will Cathcart, described it as “a built and operated surveillance system of Apple”.

More specifically, there are real concerns that once this system is created, Apple could be pressed, either by the application of the Law or Governments, to look for other types of material. While the detection of CSAM will only be in the US. UU To begin with, Apple has suggested that it can eventually expand to other countries and work with other organizations. It is not difficult to imagine scenarios where Apple could be pressed to start looking for other types of content that is illegal in some countries. The company’s concessions in China, where, according to reports, Apple “gave up the control” of its data centers to the Chinese Government, are cited as proof that the company is not immune to the demands of less democratic governments.

There are other questions as well. As if it is possible that someone abuse this process, getting maliciously someone’s device to activate them, losing access to their iCloud account. Or if there could be a false positive, or some other scenario that results in that someone is incorrectly marked by the company’s algorithms.

What does Apple say about this?

Apple has reluctantly denied that privacy is degrading or returning its previous commitments. The company published a second document in which it tries to address many of these affirmations.

On the subject of false positives, Apple has repeatedly emphasized that it is only comparing the photos of users against a collection of known child exploitation material, so images, so the images, say their own children will not activate a report. In addition, Apple has said that the chances of a false positive are around one by one trillion when it stops at the fact that a certain number of images should be detected to even activate a review. However, crucially, Apple is basically saying, we only have to take his word about that. As the former Facebook security manager, Alex Stamos and the security researcher, Matthew Green wrote at a set set of New York Times Op-Ed, Apple has not provided external researchers with a lot of visibility in the way all this it works.

Apple also says that its manual review, which is based on human reviewers, could detect if the CSAM was on a device as a result of some kind of malicious attack.

When it comes to the pressure of governments or police agencies, the company has basically said it would refuse to cooperate with such applications. “We have faced the demands of building and implementing compulsory government changes that degrade the privacy of users before, and have strongly rejected those demands,” he writes. “We will continue rejecting them in the future. Let’s be clear, this technology is limited to detecting the CSAM stored in iCloud and we will not access the request of any government to expand it.” Although, once again, we just have to take Apple in his word here.

If it’s so controversial, why does the apple do it?

The short answer is because the company believes that this is to find the right balance between the increase in child safety and privacy protection. The CSAM is illegal and, in the US UU., Companies are obliged to inform it when they find it. As a result, CSAM’s detection features have been baked in popular services for years. But unlike other companies, Apple has not checked the CSAM in the photos of the users, to a large extent due to their position on privacy. Without being surprisingly, this has been an important source of frustration for child safety organizations and the application of the law.

To put this in perspective, in 2019 Facebook reported 65 million instances of CSAM on its platform, according to the New York Times. Google reported 3.5 million photos and videos, while Twitter and Snap reported “more than 100,000”, on the other hand, the apple, on the other hand, reported 3,000 photos.

That is not because children predators do not use Apple’s services, but Apple has not been as aggressive as some other platforms to look for this material, and their privacy characteristics have made it difficult to do so. What has changed is now that Apple says it is created with a technical means to detect CSAM collections known in iCloud’s photo libraries that still respect the privacy of users. Obviously, there is a lot of disagreement about the details and if any type of detection system can be really “private”. But Apple has calculated that the compensation is worth it. “If you are storing a collection of CSAM material, yes, this is bad for you,” said Apple’s Privacy Chief to New York Times. “But for the rest of you, this is not different.”