Moderation at scale is impossible. And yet, you’d still hope we’d get better moderation than this, despite all the problems inherent in policing millions of transactions.

Archie McPhee — seller of all things weird and wonderful — recently tried promoting its “tardigrade” line of goods only to find out PayPal users couldn’t purchase them. Tardigrades are the official name for microscopic creatures known colloquially as “water bears.” Harmless enough, except PayPal blocked the transaction and sent this unhelpful response:

If you can’t read/see the tweet and the screenshot, here’s what it says:

Just an FYI that @PayPal is currently blocking all transactions containing the word “tardigrade” in the product name or description. We’ve contacted them and they told us we should just stop using the word tardigrade.

And PayPal’s response:

Every transaction that goes through our system, is reviewed by our internal security team. Certain words can trigger our security system. Unfortunately, this cannot be overridden. I would advise you to change the wording on your website to prevent this from happening.

PayPal’s size demands the use of automated moderation. But this outcome seems inexplicable. It says the “internal security team” manually reviewed the block… and decided to keep it in place anyway. What’s the point of having a “security team” if they can’t override the algorithm’s decision?

Then there’s the question as to why “tardigrade” is blocked in the first place. It’s the official name for a particularly hardy micro-animal found all over the world. Early speculation centered on the Scunthorpe Problem, suggesting PayPal blocks transactions involving forms of the word “retarded.”

But it appears to be even more ridiculous than that. Tim Ellis at GeekWire received this explanation from PayPal:

A PayPal representative put the blame on the US government’s Office of Foreign Assets Control (OFAC) sanctions, which contain an entry for an industrial supply company called “Tardigrade Limited” located in the country of Cyprus. According to PayPal, the word “tardigrade” triggered a manual review process because their system determined that the payments “may potentially violate US sanction laws.”

Customers have a Balkan arms dealer to blame for their inability to purchase tardigrade goods.

Slobodan Tesic (Tesic) was identified in the annex of E.O. 13818 on December 21, 2017. At the time of his designation, Tesic was among the biggest dealers of arms and munitions in the Balkans, spending nearly a decade on the United Nations (UN) Travel Ban List for violating UN sanctions against arms exports to Liberia.

[…]

Tesic also utilized Cyprus-based Tardigrade Limited (Tardigrade) to conduct business in third-party countries, particularly Arab and African countries. Tesic has also used his Serbian companies to sign contracts with Tardigrade before selling the goods to a final buyer.

So, “tardigrade” is flagged by the system as indicative of sanctions violations. But there’s that term again: “manual review.” Is it impossible for reviewers to distinguish between arms sales through third parties and these?

Now, it could be the manual review team didn’t want to end up on the wrong side of sanctions and felt safer blocking transactions than possibly allowing an arms dealer to launder money through the sale of adorable water bear products. Or it could be the manual “review” consists of scrolling through a list of flagged items as quickly as possible and hitting the “approve all” button. Whatever it is, it ain’t working. And Archie McPhee isn’t the first retailer to run into this problem. Two months ago, Two Photon Art noted it had to rename its Tardigrade pin to “Water Bear Enamel Pin” to allow PayPal users to purchase it.

Erring on the side of caution seems like the smart thing to do. But when the term “manual review” accompanies “automated process,” you’d think manual reviewers would see these errors for what they are, rather than allow the blocking to continue. It appears PayPal is doing a little more manual review for tardigrade-related purchases now that it’s gone a bit viral, with customers experiencing delays rather than being hit with warnings their purchases have violated PayPal policies.

The upshot is stuff like this will only become more common as time goes on. The more pressure that’s placed on tech companies to aggressively police content, the greater the chance harmless content will be rendered inaccessible. It’s not that companies shouldn’t make efforts to keep their sites free of illegal content and whatever the companies would rather not see on their sites, but automated moderation will always create issues like these. And there just aren’t enough manual reviewers available to clean up algorithmic mistakes.

Techdirt.