Tag Archive for: Moderation

Content Moderation Best Practices for Startups

/in

To say content moderation has become a hot topic over the past few years would be an understatement. The conversation has quickly shifted from how to best deal with pesky trolls and spammers  —  straight into the world of intensely serious topics like genocide and destabilization of democracies.

While this discussion often centers around global platforms like Facebook and Twitter, even the smallest of communities can struggle with content moderation. Just a limited number of toxic members can have an outsize effect on a community’s behavioral norms.

That’s why the issue of content moderation needs to be treated as a priority for all digital communities, large and small. As evidenced by its leap from lower-order concern to front-page news, content moderation is deserving of more attention and care than most are giving it today. As I see it, it’s a first-class engineering problem that calls for a first-class solution. In practical terms, that means providing:

  1. accessible, flexible policies and procedures that account for the shades of gray moderators see day to day; and

  2. technology that makes those policies and procedures feasible, affordable, and effective.

Fortunately, this doesn’t have to be a daunting task. I’ve spent years having conversations with platforms that are homes to tens to hundreds of millions of monthly active users, along with advisors spanning the commercial, academic, and non-profit sectors. From these conversations, I’ve created this collection of content moderation and community building best practices for platforms of all sizes.

Content policies

  1. Use understandable policies.

This applies to both the policies you publish externally and the more detailed, execution-focused version of these policies that help your moderators make informed and consistent decisions. While the decisions and trade-offs underlying these policies are likely complex, once resolution is reached the policies themselves need to be expressed in simple terms so that users can easily understand community guidelines and moderators can more easily recognize violations.

When the rules aren’t clear, two problems arise: (i) moderators may have to rely on gut instincts rather than process, which can lead to inconsistency; and (ii) users lose trust because policies appear arbitrary. Consider providing examples of acceptable and unacceptable behaviors to help both users and moderators see the application of your policies in action (many examples will be more clarifying than just a few). Again, this is not to say that creating policies is an easy process, there will be many edge cases that make this process challenging. We touch more on this below.

  1. Publicize policies and changes.

Don’t pull the rug out from under your users. Post policies in an easy-to-find place, and notify users when they change. How to accomplish the latter will depend on your audience, but you should make a good faith effort to reach them. For some, this may mean emailing; for others, a post pinned to the top of a message board will suffice.

  1. Build policies on top of data.

When your policies are called into question, you want to be able to present a thoughtful approach to their creation and maintenance. Policies based on intuition or haphazard responses to problems will likely cause more issues in the long run. Grounding your content policies on solid facts will make your community a healthier, more equitable place for users.

  1. Iterate.

Times change, and what works when you start your community won’t necessarily work as it grows. For instance, new vocabulary may come into play, and slurs can be reappropriated by marginalized groups as counterspeech. This can be a great opportunity to solicit feedback from your community to both inform changes and more deeply engage users. Keep in mind that change need not be disruptive  —  communities can absorb lots of small, incremental changes or clarifications to policies.

Harassment and abuse detection

  1. Be proactive.

Addressing abusive content after it’s been posted generally only serves to highlight flaws and omissions in your policies, and puts the onus of moderation on users. Proactive moderation can make use of automated initial detection and human moderators working in concert. Automated systems can flag potentially abusive content, after which human moderators with a more nuanced understanding of your community can jump in to make a final call.

  1. Factor in context.

Words or phrases that are harmful in one setting, may not be in another. Simple mechanisms like word filters and pattern matching are inadequate for this task, as they tend to under-censor harmful content and over-censor non-abusive content. Having policies and systems that can negotiate these kinds of nuances is critical to maintaining a platform’s health.

  1. Create a scalable foundation.

Relying on human moderation and sparse policies may work when your goal is to get up and running, but can create problems down the road. As communities grow, the complexity of expression and behavior grows. Establishing policies that can handle increased scale and complexity over time can save time and money  —  and prevent harassment  —  in the long term.

  1. Brace for abuse.

There’s always the danger of persistent bad actors poisoning the well for an entire community. They may repeatedly test keyword dictionaries to find gaps, or manipulate naive machine learning-based systems to “pollute the well.” Investing in industrial-grade detection tooling early on is the most effective way to head off these kinds of attacks.

  1. Assess effectiveness.

No system is infallible, so you’ll need to build regular evaluations of your moderation system into your processes. Doing so will help you understand whether a given type of content is being identified correctly or incorrectly  —  or missed entirely. That last part is perhaps the biggest problem you’ll face. I recommend using production data to build evaluation sets, allowing you to track performance over time.

Moderation actions

  1. Act swiftly.

Time is of the essence.  The longer an offensive post remains, the more harm can come to your users and your community’s reputation. Inaction or delayed response can create the perception that your platform tolerates hateful or harassing content, which can lead to a deterioration of user trust.

  1. Give the benefit of the doubt.

From time to time, even “good” community members may unintentionally post hurtful content. That’s why it’s important to provide ample notice of disciplinary actions like suspensions. Doing so will allow well-intentioned users to course-correct, and, in the case of malicious users, provide a solid basis for more aggressive measures in the future.

  1. Embrace transparency.

One of the biggest risks in taking action against a community member is the chance you’ll come across as capricious or unjustified. Regularly reporting anonymized, aggregated moderation actions will foster a feeling of safety among your user base.

  1. Prepare for edge cases.

Just as you can’t always anticipate new terminology, there will likely be incidents your policies don’t clearly cover. One recommendation for handling these types of hiccups is a process that triggers the use of an arbiter that holds final authority.

Another method is to imagine the content or behavior to be 10,000 times as common as it is today. The action you would take in that scenario can inform the action you take today. Regardless of the system you develop, be sure to document all conversations, debates, and decisions. And once you’ve reached a decision, formalize it by updating your content policy.

  1. Respond appropriately.

Typically, only a small portion of toxic content comes from persistent, determined bad actors. The majority of incidents are due to regular users having an off-day. That’s why it’s important to not apply draconian measures like permanent bans at the drop of a hat. Lighter measures like email or in-app warnings, content removal, and temporary bans send a clear signal about unacceptable behavior while allowing users to learn from their mistakes.

  1. Target remedies.

Depending on the depth of your community, a violation may be limited to a subgroup within a larger group. Be sure to focus on the problematic subgroup to avoid disrupting the higher-level group.

  1. Create an appeals process.

In order to establish and build trust, it’s important to create an equitable structure that allows users to appeal when they believe they’ve been wrongly moderated. As with other parts of your policies, transparency plays a big role. The more effort you put into explaining and publicizing your appeals policy up front, the safer and stronger your community will be in the long run.

  1. Protect moderators.

While online moderation is a relatively new field, the stresses it causes are very real. Focusing on the worst parts of a platform can be taxing psychologically and emotionally. Support for your moderators in the form of removing daily quotas, enforcing break times, and providing counseling is good for your community — and the ethical thing to do.

And if you’re considering opening a direct channel for users to communicate with Trust & Safety agents, be aware of the risks. While it can help dissipate heightened user reactions, protecting moderators here is also critical. Use shared, monitored inboxes for inbound messages and anonymized handles for employee accounts. Use data to understand which moderators are exposed to certain categories or critical levels of abusive content. Lastly, provide employees with personal online privacy-protecting solutions such as DeleteMe.

Measurement

  1. Maintain logs.

Paper trails serve as invaluable reference material. Be sure to keep complete records of flagged content including the content under consideration, associated user or forum data, justification for the flag, moderation decisions, and post mortem notes, when available. This information can help inform future moderation debates and identify inconsistencies in the application of your policies.

  1. Use metrics.

Moderation is possibly the single most impactful determinant of a community user’s experience. Measurement of its effectiveness should be subject to the same rigor you’d apply to any other part of your product. By evaluating your moderation process with both quantitative and qualitative data, you’ll gain insight into user engagement, community health, and the impact of toxic behavior.

  1. Use feedback loops.

A final decision on a content incident need not be the end of the line. Don’t let the data you’ve collected through the process go to waste. Make it a part of regular re-evaluations and updates of content policies to not only save effort on similar incidents, but also to reinforce consistency.

Most importantly, though, your number one content moderation concern should be strategic in nature. As important as all of these recommendations are for maintaining a healthy community, they’re nothing without an overarching vision. Before you define your policies, think through what your community is, who it serves, and how you’d like it to grow. A strong sense of purpose will help guide you through the decisions that don’t have obvious answers — and, of course, help attract the audience you want.

This collection of best practices is by no means the be-all and end-all of content moderation, but rather a starting point. This industry is constantly evolving and we’ll all need to work together to keep best practices at the frontier. If you have any comments or suggestions, feel free to share on this Gitlab repo.

Let’s help make the internet a safer, more respectful place for everyone.

Taylor Rhyne is co-founder and Chief Operating Officer of Sentropy, an internet security company building machine learning products to detect and fight online abuse. Rhyne was previously an Engineering Project Manager at Apple on the Siri team where he helped develop and deploy advanced Natural Language Understanding initiatives.

Techdirt.

Content Moderation And Human Nature

/in

It should go without saying that communication technologies don’t conjure up unfathomable evils all by themselves. They are a convenience-enhancer, a conduit, and a magnifying lens amplifying something that’s already there: our deeply flawed humanity. Try as we might to tame it (and boy have we tried), human nature will always rear its ugly head. Debates about governing these technologies should start by making the inherent tradeoffs more explicit.

Institutions

First, a little philosophizing. From the social contract onwards, a significant amount of resources have been allocated to attempting to subdue human nature’s predilection for self-preservation at all costs. Modern society is geared towards improving the human condition by striving to unlearn — or at least overpower — our more primitive responses.

One such attempt is the creation of institutions, with norms, rules, cultures and, on paper, inherently stronger principles than those rooted deep inside people.

It’s difficult to find ideologies that don’t allow for some need for institutions. Even the most ardent of free market capitalists acquiesce to the — limited, in their mindset — benefits of certain institutions. Beyond order and a sense of impartiality, institutions help minimize humans’ unchecked power in consequential choices that can impact wider society.

One ideal posits that institutions (corporations, parties, governments) given unfettered control over society could rid us of the aspects of our humanity that we’ve so intently tried to escape, bringing forth prosperity, equality, innovation, and progress. The fundamental flaw in that reasoning is that institutions are still intrinsically connected to humanity; created, implemented, and staffed by fallible human beings.

However strict the boundaries in which humans are expected to operate, the potential for partial or even total capture is very high. The boundaries are rarely entirely solid, and even if they were, humans always have the option to not comply. Bucking the system is not just an anomaly, it’s revered in a large portion of non-totalitarian regimes as a sign of independence, strong individuality, and as a characteristic of those lauded as mavericks.

The power of institutional norms tasked with guarding against the worst of what humans can offer is proven to be useless when challenged by people for whom self-preservation is paramount. A current and facile example is the rise to power of Donald Trump and his relentless destruction of society-defining unwritten rules.

Even without challenging the institution, a turn towards self-indulgence is easily achievable, forging a path to a reshaping in its image. The most obvious example is that of communism, wherein the lofty goal of equality is operationalized through a party-state apparatus to ostensibly distribute equally the spoils of society’s labor. As history has shown, this is contingent on the sadly unlikely situation wherein all those populating institutions are genuinely altruistic. Invariably, the best-case scenario dissipates, if it ever materialized, and inequality deepens — the opposite of the desired goal.

This is not a tacit endorsement of a rule-less, institution-less dystopia simply because rules and institutions are not adept at a practically impossible task. Instead, this should be read as a cautionary tale for overextending critical aspects of society and treating them as panacea, rather than a suitable and mostly successful palliative.

Artificial Intelligence

Armed with the continuous failure of institutions to overcome human nature, you’d think we would stop trying to remove our imperfect selves from the equation.

But what we’ve seen for more than a decade now has been technology that directly and distinctly promises to remove our worst impulses, if not humans entirely, from thinking, acting, or doing practically anything of consequence. AI, the ultimate and literal deus ex machina, is advertised as a solution for a large number of much smaller concerns. Fundamentally, its solution to these problems is ostensibly removing the human element.

Years of research, experiments, blunders, mistakes and downright evil deeds have led us to safely conclude that artificial intelligence is as successful at eliminating the imperfect human as the “you wouldn’t steal a car” anti-piracy campaign was at stopping copyright infringement. This is not to denigrate the important and beneficial work scientists and engineers have put into building intelligent automation tasked with solving complex problems.

Technology, and artificial intelligence in particular, is created, run and maintained by human beings with perspectives, goals, and inherent biases. Just like institutions, once a glimpse of positive change or success is evident, we extrapolate it far beyond its limits and task it with the unachievable and unenviable goal of fixing humanity — by removing it from the equation.

Platforms

Communication technology is not directly tasked with solving society, it simply is meant as a tool to connect us all. Much like AI, it has seemingly elegant solutions for messy problems. It’s easy to see that thanks to tech platforms, be they bulletin boards or TikTok, distance becomes trivial in maintaining connection. Community can be built and fostered online, otherwise marginalized voices can be heard, and businesses can be set up and grow digitally. Even loneliness can be alleviated.

With such a slew of real and potential benefits, it’s no wonder that we started to ascribe them with increasingly more consequential roles for society; roles these technologies were never built for and are far beyond their technical and ethical capabilities.

The Arab Spring in the early 2010s wasn’t just a liberation movement by oppressed and energized populations. It was also an opportunity for free PR for now tech-giants Twitter and Facebook, as various outlets and pundits branded revolutions with their names. It didn’t help that CEOs and tech executives seized on this narrative and, in typical Silicon Valley fashion, took to promising things akin to a politician trying to get elected.

When you set the bar that high, expectations understandably follow. The aura of tech solutionism implies such earth-shattering advancements as ordinary.

Nearly everyone can picture the potential good for society these technologies can do. And while we may all believe in that potential, the reality is that, so far, communication technologies have mostly provided convenience. Sometimes this convenience is in fact live-saving, but mostly it’s just an added benefit.

Convenience doesn’t alter our core. It doesn’t magically make us better humans or create entirely different societies. It simply lifts a few barriers from our path. This article may be seen as an attempt to minimize the perceived role of technology in society, in order to subsequently deny it and its makers any blame for how society uses it. But that is not what I am arguing.

An honest debate about responsibility has to fundamentally start with a clear understanding of the actual task something accomplishes, the perceived task others attribute to it, and its societal and historical context. A technology that provides convenience should not be fundamental to the functioning of a society. Convenience can easily become so commonplace that it ceases to be an added benefit but an integral part of life where the prospect of it being taken away is met with screams of bloody murder.

Responsibility has to be assigned to the makers, maintainers and users of communication technology, by examining which barriers are being lifted and why. There is plenty of responsibility there to be had, and I am involved in a couple of projects that try to untangle this complex mess. However, these platforms are not the reason for the negative parts of life, they are merely the conduit.

Yes, a sentient conduit can tighten or loosen its grip, divert, amplify, temporarily block messages, but it isn’t the originator of those messages, or of the intent behind it. It can surely be extremely inviting for messages of hate and division, maybe because of business models, maybe because of engineering decisions, or maybe simply because growth and scale never actually happened in a proper way. But that hate and division is endemic to human nature, and to assume that platforms can do what institutions have persistently failed to do, namely entirely eradicate it, is nonsensical.

Regulation

It is clear that platforms, reaching the size and ubiquity that they have, require updated and smart regulations in order to properly balance their benefits and the risks. But the push (and counter-push) to regulate has to start from a perspective that understands both fundamental leaps: platforms are to human nature what section 230 (or any other national-level intermediary liability law) is to the First Amendment (or any national level text that inscribes the social consensus on free speech).

If your issue is with hate and hate speech, the main thing you have to contend with are human nature and the First Amendment, not just the platforms and section 230. Without a doubt, both the platforms and section 230 are choices and explicit actions built on top of the other two, and are not fundamentally the only or best form of what they could be.

But a lot of the issues that bubble up within the content moderation and intermediary liability space come from a concern over the boundaries. That concern is entirely related to the broader contexts rather than the platforms or the specific legislation.

Regulating platforms has to start from the understanding that tradeoffs, most of which are cultural in nature, are inevitable. To be clear: there is no way to completely stop evil from happening on these platforms without making them useless.

If we were to simply ignore hate speech, we’d eliminate convenience and in some instances invalidate the very existence of these platforms. That should not be an issue if these platforms were still seen as simple conveyors of convenience, but they are currently being seen as much more than that.

Tech executives and CEOs have moved into the fascinating space wherein they have to protect their market power to assuage their shareholders, treat their products as mind-meltingly amazing to gain and keep users, yet imply their role in society is transient and insignificant in order to mollify policy-makers all at the same time.

The convenience afforded by these technologies is allowing nefarious actors to cause substantial harm to a substantial number of people. Some users get death threats, or even have their life end tragically because of interactions on these platforms. Others will have their most private information or documents exposed, or experience sexual abuse or trauma through a variety of ways.

Unfortunately, these things happen in the offline world as well, and they are fundamentally predicated on the regulatory/institutional context and the tools that allow them to manifest. The tools are not off the hook. Their propensity to not minimize harm, online and off, are due for important conversations. But they are not the cause. They are the conduit.

Thus, the ultimate goal of “platforms existing without hate or violence” is very sadly not realistic. Neither are tradeoffs such as being ok with stripping fundamental rights in exchange for a safer environment, or being ok with some people suffering immense trauma and pain simply because one believes in the concept of open speech.

Maybe the solution is to not have these platforms at all, or ask them to change substantially. or maybe it’s to calibrate our expectations, or maybe yet, to address the underlying issues in our society. Once we see what the boundaries truly are, any debate becomes infinitely more productive.

This article is not advancing any new or groundbreaking ideas. What it does is identify crucial and seemingly misunderstood pieces of the subtext and spell it out. Sadly, the fact that these more or less evident issues needed to be said in plain text should be the biggest take-away.

As a qualitative researcher, I learned that there is no way to “de-bias” my work. Trying to remove myself from the equation results in a bland “view from nowhere” that is ignorant of the underlying power dynamics and inherent mechanisms of whatever I am studying. However, that doesn’t mean we take off our glasses when trying to see for fear of the glasses influencing what we see, because that would actually make us blind. We remedy that by acknowledging our glasses as well.

A communication platform (company, tech, product) that doesn’t have inherent biases is impossible. But that shouldn’t mean that we can’t try to ask it to be better, either through regulation, collaboration or hostile action. We just have to be cognizant of the place we’re standing when asking, the context, potential consequences and as this piece hopefully shows, what it can’t actually do.

The conversation surrounding platform governance would benefit immensely from these tradeoffs being made explicit. It would certainly dial down the rhetoric and (genuine) visceral attitudes towards debate as it would force those directly involved or invested in one outcome to carefully assess the context and general tradeoffs.

David Morar, PhD is an academic with the mind of a practitioner and currently a Fellow at the Digital Interests Lab and a Visiting Scholar at GWU’s Elliott School of International Affairs.

Techdirt.

Content Moderation Knowledge Sharing Shouldn’t Be A Backdoor To Cross-Platform Censorship

/in

Ten thousand moderators at YouTube. Fifteen thousand moderators at Facebook. Billions of users, millions of decisions a day. These are the kinds of numbers that dominate most discussions of content moderation today. But we should also be talking about 10, 5, or even 1: the numbers of moderators at sites like Automattic (WordPress), Pinterest, Medium, and JustPasteIt—sites that host millions of user-generated posts but have far fewer resources than the social media giants.

There are a plethora of smaller services on the web that host videos, images, blogs, discussion fora, product reviews, comments sections, and private file storage. And they face many of the same difficult decisions about the user-generated content (UGC) they host, be it removing child sexual abuse material (CSAM), fighting terrorist abuse of their services, addressing hate speech and harassment, or responding to allegations of copyright infringement. While they may not see the same scale of abuse that Facebook or YouTube does, they also have vastly smaller teams. Even Twitter, often spoken of in the same breath as a “social media giant,” has an order of magnitude fewer moderators at around 1,500.

One response to this resource disparity has been to focus on knowledge and technology sharing across different sites. Smaller sites, the theory goes, can benefit from the lessons learned (and the R&D dollars spent) by the biggest companies as they’ve tried to tackle the practical challenges of content moderation. These challenges include both responding to illegal material and enforcing content policies that govern lawful-but-awful (and mere lawful-but-off-topic) posts.

Some of the earliest efforts at cross-platform information-sharing tackled spam and malware such as the Mail Abuse Prevention System (MAPS) — which maintains blacklists of IP addresses associated with sending spam. Employees at different companies have also informally shared information about emerging trends and threats, and the recently launched Trust & Safety Professional Association is intended to provide people working in content moderation with access to “best practices” and “knowledge sharing” across the field.

There have also been organized efforts to share specific technical approaches to blocking content across different services, namely, hash-matching tools that enable an operator to compare uploaded files to a pre-existing list of content. Microsoft, for example, made its PhotoDNA tool freely available to other sites to use in detecting previously reported images of CSAM. Facebook adopted the tool in May 2011, and by 2016 it was being used by over 50 companies.

Hash-sharing also sits at the center of the Global Internet Forum to Counter Terrorism (GIFCT), an industry-led initiative that includes knowledge-sharing and capacity-building across the industry as one of its 4 main goals. GIFCT works with Tech Against Terrorism, a public-private partnership launched by the UN Counter-Terrrorism Executive Directorate, to “shar[e] best practices and tools between the GIFCT companies and small tech companies and startups.” Thirteen companies (including GIFCT founding companies Facebook, Google, Microsoft, and Twitter) now participate in the hash-sharing consortium.

There are many potential upsides to sharing tools, techniques, and information about threats across different sites. Content moderation is still a relatively new field, and it requires content hosts to consider an enormous range of issues, from the unimaginably atrocious to the benignly absurd. Smaller sites face resource constraints in the number of staff they can devote to moderation, and thus in the range of language fluency, subject matter expertise, and cultural backgrounds that they can apply to the task. They may not have access to — or the resources to develop — technology that can facilitate moderation.

When people who work in moderation share their best practices, and especially their failures, it can help small moderation teams avoid pitfalls and prevent abuse on their sites. And cross-site information-sharing is likely essential to combating cross-site abuse. As scholar evelyn douek discusses (with a strong note of caution) in her Content Cartels paper, there’s currently a focus among major services in sharing information about “coordinated inauthentic behavior” and election interference.

There are also potential downsides to sites coordinating their approaches to content moderation. If sites are sharing their practices for defining prohibited content, it risks creating a de facto standard of acceptable speech across the Internet. This undermines site operators’ ability to set the specific content standards that best enable their communities to thrive — one of the key ways that the Internet can support people’s freedom of expression. And company-to-company technology transfer can give smaller players a leg up, but if that technology comes with a specific definition of “acceptable speech” baked in, it can end up homogenizing the speech available online.

Cross-site knowledge-sharing could also suppress the diversity of approaches to content moderation, especially if knowledge-sharing is viewed as a one-way street, from giant companies to small ones. Smaller services can and do experiment with different ways of grappling with UGC that don’t necessarily rely on a centralized content moderation team, such as Reddit’s moderation powers for subreddits, Wikipedia’s extensive community-run moderation system, or Periscope’s use of “juries” of users to help moderate comments on live video streams. And differences in the business model and core functionality of a site can significantly affect the kind of moderation that actually works for them.

There’s also the risk that policymakers will take nascent “industry best practices” and convert them into new legal mandates. That risk is especially high in the current legislative environment, as policymakers on both sides of the Atlantic are actively debating all sorts of revisions and additions to intermediary liability frameworks.

Early versions of the EU’s Terrorist Content Regulation, for example, would have required intermediaries to adopt “proactive measures” to detect and remove terrorist propaganda, and pointed to the GIFCT’s hash database as an example of what that could look like (CDT joined a coalition of 16 human rights organizations recently in highlighting a number of concerns about the structure of GIFCT and the opacity of the hash database). And the EARN-IT Act in the US is aimed at effectively requiring intermediaries to use tools like PhotoDNA—and not to implement end-to-end encryption.

Potential policymaker overreach is not a reason for content moderators to stop talking to and learning from each other. But it does mean that knowledge-sharing initiatives, especially formalized ones like the GIFCT, need to be attuned to the risks of cross-site censorship and eliminating diversity among online fora. These initiatives should proceed with a clear articulation of what they are able to accomplish (useful exchange of problem-solving strategies, issue-spotting, and instructive failures) and also what they aren’t (creating one standard for prohibited — much less illegal— speech that can be operationalized across the entire Internet).

Crucially, this information exchange needs to be a two-way street. The resource constraints faced by smaller platforms can also lead to innovative ways to tackle abuse and specific techniques that work well for specific communities and use-cases. Different approaches should be explored and examined for their merit, not viewed with suspicion as a deviation from the “standard” way of moderating. Any recommendations and best practices should be flexible enough to be incorporated into different services’ unique approaches to content moderation, rather than act as a forcing function to standardize towards one top-down, centralized model. As much as there is to be gained from sharing knowledge, insights, and technology across different services, there’s no-one-size-fits-all approach to content moderation.

Emma Llansó is the Director of CDT’s Free Expression Project, which works to promote law and policy that support Internet users’ free expression rights in the United States and around the world. Emma also serves on the Board of the Global Network Initiative, a multistakeholder organization that works to advance individuals’ privacy and free expression rights in the ICT sector around the world. She is also a member of the multistakeholder Freedom Online Coalition Advisory Network, which provides advice to FOC member governments aimed at advancing human rights online.

Techdirt.

Gizmodo: Why Can’t YouTube Do ‘Good’ Content Moderation? Answer: Because It’s Fucking Impossible

/in

We’ve had something of a long-running series of posts on the topic of content moderation, with our stance generally being that any attempt to do this at scale is laughably difficult. Like, to the point of being functionally impossible. This becomes all the more difficult when the content in question is not universally considered objectionable.

Tech firms tend to find themselves in the most trouble when they try to bow to this demand for content moderation, rather than simply declaring it to be impossible and moving on. The largest platforms have found themselves in this mess, namely Facebook and YouTube. YouTube, for instance, has released new moderation policies over the past two months or so that seek to give it broad powers to eliminate content that it deems to be hate speech, or speech centered on demographic supremacy. Wanting to eliminate that sort of thing is understandable, even if you still think it’s problematic. Actually eliminating it at scale, and in a way that doesn’t sweep up collateral damage and garners wide support, is impossible.

Which makes it frustrating to read headlines such as Gizmodo’s recent piece on how YouTube is doing with all of this.

YouTube Said It Was Getting Serious About Hate Speech. Why Is It Still Full of Extremists?

Because it’s fucking impossible, that’s why. There is simply no world in which YouTube both successfully eliminates all, or even the majority, of speech that some large group or another considers hate speech or “extreme.” That’s never going to happen. YouTube never should have suggested it would happen. The screw up here is YouTube not properly setting the public’s expectations as to what its policy would achieve. Yeah, there is still a good deal of extremist content on YouTube. Whipping up anger at content that’s available at this moment is trivially easy.

Making it more frustrating is Gizmodo’s assertion, with a sinister connotation, that all of this is “part of YouTube’s plan.”

Strangely, this isn’t a simple oversight by YouTube’s parent company, Google. In fact, it’s the policy working as planned. YouTube hosts more than 23 million channels, making it impossible to identify each and every one that is involved with the hate movement—especially since one person’s unacceptable hate speech is another person’s reasonable argument. With that in mind, we used lists of organizations promoting hate from the Southern Poverty Law Center, Hope Not Hate, the Canadian Anti-Hate Network, and the Counter Extremism Project, in addition to channels recommended on the white supremacist forum Stormfront, to create a compendium of 226 extremist YouTube channels earlier this year.

While less than scientific (and suffering from a definite selection bias), this list of channels provided a hazy window to watch what YouTube’s promises to counteract hate looked like in practice. And since June 5th, just 31 channels from our list of more than 200 have been terminated for hate speech. (Eight others were either banned before this date or went offline for unspecified reasons.)

Before publishing this story, we shared our list with Google, which told us almost 60 percent of the channels on it have had at least one video removed, with more than 3,000 individual videos removed from them in total. The company also emphasized it was still ramping up enforcement. These numbers, however, suggest YouTube is aware of many of the hate speech issues concerning the remaining 187 channels—and has allowed them to stay active.

I would suggest that these numbers actually likely represent YouTube blocking too much content, rather than not enough. In a politically divided country like ours getting some significant number of people to state that even a relatively innocuous video is “extreme” would be pretty easy. Add to that fact that the selection bias mentioned above is way understated in this article, and the problem deepens. Layer on top of that the simple fact that some of the sources for this list of “extremist” content — namely the SPLC — have been caught quite recently being rather cavalier about the labels they throw around, and this whole experiment begins to look bunk.

Making Gizmodo’s analysis all the worse is that it seems to complain that YouTube is only policing the content that appears on its platform, rather than banning all content from uploaders who take nefarious actions off of YouTube’s platform.

To understand why these channels continue to operate, it’s important to know how YouTube polices its platform. YouTube’s enforcement actions are largely confined to what happens directly on its website. There are some exceptions—like when a channel’s content is connected to outside criminality—but YouTube generally doesn’t consider the external behavior of a group or individual behind an account. It just determines whether a specific video violated a specific policy.

Heidi Beirich, who runs the Southern Poverty Law Center’s Intelligence Project, charges that YouTube’s approach puts it far behind peers like Facebook, which takes a more holistic view of who is allowed to post on its site, prohibiting hate groups and their leaders from using the social network.

“Because YouTube only deals with the content posted, it allows serious white supremacists like Richard Spencer and David Duke to keep content up,” Beirich said. “In general, our feeling is that YouTube has got to get serious about removing radicalizing materials given the impact these videos can have on young, white men.”

It’s an insane request. Because a person or group says some things that are obviously objectionable, we want their voices silenced on YouTube, even when the content there isn’t objectionable? That’s fairly antithetical to how our country operates. YouTube is of course not governed by the First Amendment and can take down whatever content it chooses, but the concept of free speech and the free exchange of ideas in America is much more global as an ideal than the specific prescriptions outlined in the Constitution. Silencing all potential speech from a party simply because some of that speech is objectionable is quite plainly un-American.

Gizmodo then complains about the inconsistencies in enforcing this impossible policy.

The apparent inconsistencies go on: The channel of South African neo-Nazigroup AWB was terminated. Two others dedicated to violent Greek neo-Naziparty Golden Dawn remain active. The channel of white nationalist group American Identity Movement, famous for distributing fliers on college campuses, is still up. As is a channel for the white nationalist group VDARE. And, notably, none of the 33 channels on our list run by organizations designated by the Southern Poverty Law Center as anti-LGBTQ hate groups have been removed from the platform.

In addition to giving many hateful channels a pass, this agnosticism to uploaders’ motives means that some channels with no interest in promoting white supremacy have been punished as YouTube enforces its policies.

Unlike what Gizmodo — and even YouTube — says, this is a bug, not a feature. We cannot say this enough: there is no good way to do this. Frankly, save for criminal content, YouTube probably shouldn’t even be trying. Alternatively, if it does want to try, it probably would be more satisfying if YouTube’s public stance was something like: “We’ll block whatever we want, because we’re allowed to. If those blocks don’t seem to make sense to you, deal with it.” At least that would set the proper expectations with the public.

And then maybe there would be less consternation as to why YouTube hasn’t yet achieved the impossible. Impossible, in this case, being both doing content moderation at scale and simultaneously making everybody happy.

Permalink | Comments | Email This Story

Techdirt.