In 2018, Liz O’Sullivan and her colleagues at a well-known artificial intelligence start-up began working on a system that could automatically remove nudity and other explicit images from the Internet.
They sent millions of online photos to workers in India who spent weeks tagging unique material. The data paired with the photos would be used for teaching AI software how to spot indecent pictures. However, once the photos were tagged, Ms. O’Sullivan and her team noticed a problem: the Indian workers had labeled all pictures of same-sex couples as indecent.
For Ms. O’Sullivan, the moment showed how easily – and often – bias could penetrate artificial intelligence. It was a “cruel game of Whac-a-Mole,” she said.
This month, Ms. O’Sullivan, a 36-year-old New Yorker, was named General Manager of the new Parity company. The start-up is one of many organizations, including more than a dozen start-ups and some of the biggest names in technology, that offer tools and services designed to identify and eliminate bias in AI systems.
Companies will soon need this help. In April, the Federal Trade Commission warned Against the sale of AI systems that were racially biased or that could prevent individuals from obtaining employment, housing, insurance or other benefits. A week later, the European Union unveiled draft regulations that Punish companies for offering such technologies.
It is unclear how regulators could monitor bias. Last week the National Institute of Standards and Technology, a government research laboratory whose work often influences policy, released one suggestion details how businesses can combat bias in AI, including changes in the way technology is designed and built.
Many in the tech industry believe companies need to prepare for a crackdown. “Some legislation or regulation is inevitable,” said Christian Troncoso, senior director of legal policy for Software Alliance, a trading group that represents some of the largest and oldest software companies. “Every time there is one of those horrific stories about AI, it breaks the public’s trust and belief.”
In recent years, studies have shown that Facial recognition services, Health systems and even talking digital assistants may be biased towards women, colored people and other marginalized groups. Amid a growing number of complaints on this issue, some local regulators have already taken action.
State regulators in New York at the end of 2019 launched an investigation of the UnitedHealth Group after a study found that an algorithm used by a hospital gave white patient care priority over black patients, even if the white patients were healthier. Last year the state did has investigated the Apple Card credit service after claims it discriminates against women. The regulators decided that Goldman Sachs, who operated the card, did not discriminate, while the status of the UnitedHealth investigation is unclear.
A UnitedHealth spokesman Tyler Mason said the company’s algorithm was abused by one of its partners and was not racially biased. Apple declined to comment.
According to PitchBook, a research firm that tracks financial activities, more than $ 100 million has been invested in companies investigating ethical issues related to artificial intelligence in the past six months, up from $ 186 million last year.
However, efforts to address the problem reached a tipping point this month when the Software Alliance offered a detailed framework to combat bias in AI, including the recognition that some automated technologies require regular human monitoring. The trade group believes the document can help companies change their behavior and show regulators and lawmakers how to control the problem.
Although criticized for bias in their own systems, Amazon, IBM, Google, and Microsoft also offer tools to combat them.
Ms. O’Sullivan said there was no easy solution to bias in AI. A sensitive issue is that some in the industry are wondering if the problem is as widespread or as harmful as they think it is.
“Changes in mentality don’t happen overnight – and that is even more true when it comes to large companies,” she said. “You are trying to change not just one person’s opinion, but many.”
When she began advising companies on AI bias more than two years ago, Ms. O’Sullivan often met with skepticism. Many leaders and engineers advocated what they called “fairness through ignorance,” arguing that the best way to develop equitable technology is to ignore issues such as race and gender.
More and more companies developed systems that learned tasks by analyzing huge amounts of data such as photos, sounds, texts and statistics. The belief was that if a system learned from as much data as possible, fairness would follow.
But as Ms. O’Sullivan discovered after tagging in India, bias can creep into a system when designers select the wrong data or search it incorrectly. Studies show that facial recognition services can be biased against women and people of color when trained on photo collections dominated by white men.
Designers can be blind to these problems. Workers in India – where gay relationships were illegal at the time and where attitudes towards gays and lesbians were very different from those in the United States – classified the photos at their own discretion.
Ms. O’Sullivan realized the flaws and pitfalls of artificial intelligence while working for Clarifai, the company that ran the tagging project. She said she left the company after realizing it was building systems for the military that she believed could at some point be used for killing. Clarifai did not respond to a request for comment.
She now believes that after years of public complaints about bias in AI – not to mention the threat of regulation – attitudes are changing. In its new framework for curbing harmful biases, the Software Alliance warned against fairness through ignorance, saying the argument does not hold up.
“You acknowledge that you need to turn the rocks over and see what’s underneath,” said Ms. O’Sullivan.
Still, there is resistance. She said a recent crash on Google where two ethics researchers were ousted, was characteristic of the situation for many companies. Efforts to combat prejudice often collide with corporate culture and the relentless urge to develop new technologies, bring them to market, and make money.
It’s also still hard to tell how serious the problem is. “We have very little data needed to model the broader societal security problems with these systems, including biases,” said Jack Clark, one of the authors of the AI Index, which researches AI technology and policy around the world to pursue. “Many of the things that are important to the average person – like fairness – are not yet disciplined or large-scale.”
Ms. O’Sullivan, a college philosophy student and a member of the American Civil Liberties Union, is building her business around a tool developed by Rumman Chowdhury, a well-known AI ethics researcher who worked for Accenture consulting for years before joining on Twitter .
While other startups like Fiddler AI and Weights and Biases offer tools to monitor AI services and identify potentially biased behaviors, Parity’s technology aims to analyze the data, technologies, and methods a business uses to build its Services used, and then identify risk areas and propose changes.
The tool uses artificial intelligence technology that in itself can be skewedshowing the double-edged nature of AI – and the difficulty of Ms. O’Sullivan’s task.
Tools that can detect bias in AI are imperfect, just as AI is imperfect. But the power of such a tool, she said, is to pinpoint potential problems – to get people to take a closer look at the problem.
Ultimately, she said, the goal is to create a broader dialogue between people with a wide range of opinions. The problem arises when the problem is ignored – or when those who are discussing the problems share the same point of view.
“You need diverse perspectives. But can you really get different perspectives at one company? ”Asked Ms. O’Sullivan. “It’s a very important question that I can’t answer.”