Why Facebook Can’t Fix Itself (original) (raw)

There’s a brief pause in the conference room. “Is everyone O.K. with that?” the trainer says.

“No, not O.K.,” a moderator responds. The other moderators laugh uneasily, and the scene ends.

After the footage became public, a Facebook spokesperson claimed that the trainer had made a mistake. “I know for a fact that that’s a lie,” Chris Gray told me. “When I was there, I got multiple tickets with that exact meme in it, and I was always told to ignore. You go, ‘C’mon, we all know exactly what this means,’ but you’re told, ‘Don’t make your own judgments.’ ”

A former moderator from Phoenix told me, “If it was what they say it is—‘You’re here to clean up this platform so everyone else can use it safely’—then there’s some nobility in that. But, when you start, you immediately realize we’re in no way expected or equipped to fix the problem.” He provided me with dozens of examples of hate speech—some of which require a good amount of cultural fluency to decode, others as clear-cut as open praise for Hitler—that he says were reviewed by moderators but not removed, “either because they could not understand why it was hateful, or because they assumed that the best way to stay out of trouble with their bosses was to leave borderline stuff up.”

Recently, I talked to two current moderators, who asked me to call them Kate and John. They live and work in a non-Anglophone European country; both spoke in accented, erudite English. “If you’re Mark Zuckerberg, then I’m sure applying one minimal set of standards everywhere in the world seems like a form of universalism,” Kate said. “To me, it seems like a kind of libertarian imperialism, especially if there’s no way for the standards to be strengthened, no matter how many people complain.”

They listed several ways in which, in their opinion, their supervisors’ interpretations of the Implementation Standards conflicted with common sense and basic decency. “I just reviewed an Instagram profile with the username KillAllFags, and the profile pic was a rainbow flag being crossed out,” Kate said. “The implied threat is pretty clear, I think, but I couldn’t take it down.”

“Our supervisors insist that L.G.B.T. is a concept,” John explained.

“So if I see someone posting ‘Kill L.G.B.T.,’ unless they refer to a person or use pronouns, I have to assume they’re talking about killing an idea,” Kate said.

Cartoon by Jason Adam Katzenstein

“Facebook could change that rule tomorrow, and a lot of people’s lives would improve, but they refuse,” John said.

“Why?” I said.

“We can ask, but our questions have no impact,” John said. “We just do what they say, or we leave.”

Around the time the Grays were hired, Britain First, a white-nationalist political party in the U.K., had a Facebook page with about two million followers. (By contrast, Theresa May, then the Prime Minister, had fewer than five hundred thousand followers.) Offline, Britain First engaged in scare tactics: driving around Muslim parts of London in combat jeeps, barging into mosques wearing green paramilitary-style uniforms. On Facebook, Chris Gray said, members of Britain First would sometimes post videos featuring “a bunch of thugs moving through London, going, ‘Look, there’s a halal butcher. There’s a mosque. We need to reclaim our streets.’ ” A moderator who was empowered to consider the context—the fact that “Britain First” echoes “America First,” a slogan once used by Nazi sympathizers in the U.S.; the ominous connotation of the word “reclaim”—could have made the judgment that the Party members’ words and actions, taken together, were a call for violence. “But you’re not allowed to look at the context,” Gray said. “You can only look at what’s right in front of you.” Britain First’s posts, he said, were “constantly getting reported, but the posts that ended up in my queue never quite went over the line to where I could delete them. The wording would always be just vague enough.”

Tommy Robinson, a British Islamophobe and one of Britain First’s most abrasive allies, often gave interviews in which he was open about his agenda. “It’s a Muslim invasion of Europe,” he told Newsweek. On Facebook, though, he was apparently more coy, avoiding explicit “calls for exclusion” and other formulations that the company would recognize as hate speech. At times, Gray had the uncanny sense that he and the other moderators were acting as unwitting coaches, showing the purveyors of hate speech just how far they could go. “That’s what I’d do, anyway, if I were them,” he said. “Learn to color within the lines.” When Robinson or a Britain First representative posted something unmistakably threatening, a Facebook moderator would often flag the post for removal. Sometimes a “quality auditor” would reverse the decision. The moderator would then see a deduction in his or her “quality score,” which had to remain at ninety-eight per cent or above for the moderator to be in good standing.

Normally, after a Facebook page violates the rules multiple times, the page is banned. But, in the case of Britain First and Tommy Robinson, the bans never came. Apparently, those two pages were “shielded,” which meant that the power to delete them was restricted to Facebook’s headquarters in Menlo Park. No one explained to the moderators why Facebook decided to shield some pages and not others, but, in practice, the shielded pages tended to be those with sizable follower counts, or with significant cultural or political clout—pages whose removal might interrupt a meaningful flow of revenue.

There is little recourse for a content moderator who has qualms about the Implementation Standards. Full-time Facebook employees are given more dispensation to question almost any aspect of company policy, as long as they do so internally. On Workplace, a custom version of the network that only Facebook staffers can access, their disagreements are often candid, even confrontational. The former Dublin employee who worked on Middle East policy believes that Facebook’s management tolerates internal dissent in order to keep it from spilling into public view: “Your average tech bro—Todd in Menlo Park, or whatever—has to continually be made to feel like he’s part of a force for good. So whenever Todd notices anything about Facebook that he finds disturbing there has to be some way for his critiques to be heard. Whether anything actually changes as a result of those critiques is a separate question.”

On December 18, 2017, on a Workplace message board called Community Standards Feedback, a recruiter in Facebook’s London office posted a Guardian article about Britain First. “They are pretty much a hate group,” he wrote. He noted that “today YouTube and Twitter banned them,” and asked whether Facebook would do the same.

Neil Potts, Facebook’s public-policy director for trust and safety, responded, “Thanks for flagging, and we are monitoring this situation closely.” However, he continued, “while Britain First shares many of the common tenets of alt-right groups, e.g., ultra-nationalism,” Facebook did not consider it a hate organization. “We define hate orgs as those that advance hatred as one of their primary objectives, or that they have leaders who have been convicted of hate-related offenses.”

Another Facebook employee, a Muslim woman, noted that Jayda Fransen, a leader of Britain First, had been convicted of hate crimes against British Muslims. “If the situation is being monitored closely,” she asked, “how was this missed?”

“Thanks for flagging,” Potts responded. “I’ll make sure our hate org SMEs”—subject-matter experts—“are aware of this conviction.”

A month later, in January of 2018, the female employee revived the Workplace thread. “Happy new year!” she wrote. “The Britain First account is still up and running, even though as per above discussion it clearly violates our community standards. Is anything being done about this?”

“Thanks for circling back,” Potts responded, adding that a “team is monitoring and evaluating the situation and discussing next steps forward.” After that, the thread went dormant.

A few weeks later, Darren Osborne, a white Briton, was convicted of murder. Osborne had driven a van into a crowd near a London mosque, killing a Muslim man named Makram Ali and injuring at least nine other people. Prosecutors introduced evidence suggesting that Osborne had been inspired to kill, at least in part, by a BBC miniseries and by following Britain First and Tommy Robinson on social media. The judge deemed the killing “a terrorist act” by a man who’d been “rapidly radicalized over the Internet.” Within six weeks, Britain First and Tommy Robinson had been banned from Facebook. (Pusateri, the Facebook spokesperson, noted that the company has “banned more than 250 white supremacist organizations.”)

“It’s an open secret,” Sophie Zhang, a former data scientist for the company, recently wrote, “that Facebook’s short-term decisions are largely motivated by PR and the potential for negative attention.” Zhang left Facebook in September. Before she did, she posted a scathing memo on Workplace. In the memo, which was obtained by BuzzFeed News, she alleged that she had witnessed “multiple blatant attempts by foreign national governments to abuse our platform on vast scales”; in some cases, however, “we simply didn’t care enough to stop them.” She suggested that this was because the abuses were occurring in countries that American news outlets were unlikely to cover.

When Facebook is receiving an unusual amount of bad press for a particularly egregious piece of content, this is referred to within the company as a “press fire,” or a “#PRFire.” Often, the content has been flagged repeatedly, to no avail, but, in the context of a press fire, it receives prompt attention. A Facebook moderator who currently works in a European city shared with me a full record of the internal software system as it appeared on a recent day. There were dozens of press fires in progress. Facebook was being criticized—by Facebook users, primarily—for allowing widespread bullying against Greta Thunberg, the teen-age climate activist, who has Asperger’s syndrome. The content moderators were instructed to apply an ad-hoc exemption: “Remove all instances of attacks aimed at Greta Thunberg using the terms or hashtag: ‘Gretarded’, ‘Retard’ or ‘Retarded.’ ” No similar protections were extended to other young activists, including those whose bullying was unlikely to inspire such a public backlash. A woman who worked as a content-moderation supervisor in the U.S. told me, “You can ask for a meeting, present your bosses with bullet points of evidence, tell them you’ve got team members who are depressed and suicidal—doesn’t help. Pretty much the only language Facebook understands is public embarrassment.”

Facebook moderators have scant workplace protections and little job security. The closest they have to a labor organizer is Cori Crider, a lawyer and an activist based in London. Crider grew up in rural Texas and left as soon as she could, going first to Austin, for college; then to Harvard, for law school; and then to London, where she worked for a decade at a small human-rights organization, representing Guantánamo detainees and the relatives of drone-strike victims in Yemen and Pakistan. “It was through worrying about drones that I came to worry about technology,” she said. “I started to feel like, While we’re all focussed on the surveillance tactics of the Pentagon, a handful of companies out of California are collecting data on a scale that would honestly be the envy of any state.”

Last year, she co-founded a not-for-profit called Foxglove, where she is one of two employees. The foxglove, a wildflower also known as digitalis, can be either medicinal or toxic to humans, depending on how it’s ingested. The group’s mission is to empower the tech industry’s most vulnerable workers—to help them “clean up their factory floor,” as Crider often puts it. Her more ambitious goal is to redesign the factory. She was influenced by Shoshana Zuboff, the author of “The Age of Surveillance Capitalism,” who argues that “instrumentarian” behemoths such as Facebook pose an existential threat to democracy. In Crider’s analysis, not even the most ingenious technocratic fix to Facebook’s guidelines can address the core problem: its content-moderation priorities won’t change until its algorithms stop amplifying whatever content is most enthralling or emotionally manipulative. This might require a new business model, perhaps even a less profitable one, which is why Crider isn’t hopeful that it will happen voluntarily. “I wish I could file a global injunction against the monetization of attention,” she said. “In the meantime, you find more specific ways to create pressure.”

“Can’t say I agree with his methods, but I’ll be damned if there’s a horse that man can’t break.”

Cartoon by Lars Kenseth

In July, 2019, Crider was introduced to a Facebook moderator in Europe who was able to map out how the whole system worked. This moderator put her in touch with other moderators, who put her in touch with still others. Sometimes, while courting a moderator as a potential source, Crider arranged a meeting and flew to wherever the person lived, only to be stood up. Those who did talk to her were almost always unwilling to go on the record. The process reminded Crider of the months she’d spent in Yemen and Pakistan, trying to gain people’s trust. “They often have very little reason to talk, and every reason in the world not to,” she said. The content moderators were not yet ready to form a union—“not even close,” Crider told me—but she hoped to inculcate in them a kind of latent class consciousness, an awareness of themselves as a collective workforce.

Last October, Crider met Chris Gray at a conference in London. She started introducing him to journalists and activists, helping to spread his story. Two months later, Gray hired a local law firm and sued Facebook in Irish High Court, alleging that his “repeated and unrelenting exposure to extremely disturbing, graphic and violent content” had caused him lasting psychological trauma. Shortly thereafter, about twenty more former Facebook moderators in Dublin contacted the law firm representing Gray to ask about possible lawsuits against the company.

Soon after Gray left Facebook, he wrote a fifty-three-hundred-word memo to Zuckerberg and Sheryl Sandberg, the company’s chief operating officer, laying out his critiques of the content-moderation process. He proposed a few fixes, which ranged from the granular (“The hotkey system is a mess”) to the sweeping (“Rewrite . . . all of your user privacy policies”). None, however, addressed what he considered to be the underlying issue. On the whole, he concluded, Facebook “is not committed to content moderation, does not have a clear strategy or even a good handle on how to do it, and the people trying to do the actual work are under immense pressure to shovel shit uphill without proper tools or direction. . . . There is no leadership, no clear moral compass.” He e-mailed the memo to Zuckerberg and Sandberg from an anonymous address: shovellingshituphill@gmail.com.

Jair Bolsonaro, the autocratic Brazilian politician, ran for President, in 2018, on a shoestring budget. To get his message out, he relied heavily on Facebook, whose social-media apps are the most popular in his country. Since taking office, in 2019, he has delivered a weekly Presidential address on Facebook Live. Earlier this year, during one speech, he said of Brazil’s indigenous population, “The Indian has changed. He is evolving and becoming, more and more, a human being like us.” Bolsonaro’s racism was not exactly a surprise, but his comments caused an uproar nonetheless. Facebook did not remove the video, even though its guidelines prohibit “dehumanizing speech,” including any insinuations of “subhumanity.”

David Thiel, a cybersecurity expert at Facebook’s headquarters in Menlo Park, read about the controversy. (He is unrelated to Peter Thiel, the venture capitalist and Trump donor who sits on Facebook’s board.) After searching for the speech on Facebook, he was shocked to find that it was still up. He wrote on Workplace, “I assume we’ll be removing this video from our platform?” His question was passed on to subject-matter experts, including one in Brasília and one in Dublin. Both ruled that the video did not violate the guidelines. “President Bolsonaro is known by his controversial and ‘politically incorrect’ speeches,” the expert in Brasília wrote. “He is actually referring to indigenous people becoming more integrated to the society (as opposed to isolated in their own tribes).” This did not satisfy Thiel, in part because the expert, who has worked for at least one pro-Bolsonaro politician, did not strike him as an objective source. Also, given that Facebook’s local sales representatives had surely encouraged Bolsonaro to use their products, there may have been a conflict of interest. “It’s awkward for a business to go from a posture of ‘Please, sir, use our product,’ to ‘Actually, sir, you’re now in trouble for using our product,’ ” he said.