In some ways, artificial intelligence acts like a mirror. Machine learning tools are designed to detect patterns, and they often reflect back the same biases we already know exist in our culture. Algorithms can be sexist, racist, and perpetuate other structural inequalities found in society. But unlike humans, algorithms aren’t under any obligation to explain themselves. In fact, even the people who build them aren’t always capable of describing how they work.
That means people are sometimes left unable to grasp why they lost their health care benefits, were declined a loan, rejected from a job, or denied bail—all decisions increasingly made in part by automated systems. Worse, they have no way to determine whether bias played a role.
In response to the problem of AI bias and so-called “black box” algorithms, many machine learning experts, technology companies, and governments have called for more fairness, accountability, and transparency in AI. The research arm of the Department of Defense has taken an interest in developing machine learning models that can more easily account for how they make decisions, for example. And companies like Alphabet, IBM, and the auditing firm KPMG are also creating or have already built tools for explaining how their AI products come to conclusions.
But that doesn’t mean everyone agrees on what constitutes a fair explanation. There’s no common standard for what level of transparency is sufficient. Does a bank need to publicly release the computer code behind its loan algorithm to be truly transparent? What percentage of defendants need to understand the explanation given for how a recidivism AI works?
“Algorithmic transparency isn’t an end in and of itself,” says Madeleine Clare Elish, a researcher who leads the Intelligence & Autonomy Initiative at Data & Society. “It’s necessary to ask: Transparent to whom and for what purpose? Transparency for the sake of transparency is not enough.”
By and large, lawmakers haven’t decided what rights citizens should have when it comes to transparency in algorithmic decision-making. In the US, there are some regulations designed to protect consumers, including the Fair Credit Reporting Act, which requires individuals be notified of the main reason they were denied credit. But there isn’t a broad “right to explanation” for how a machine came to a conclusion about your life. The term appears in the European Union's General Data Protection Regulation (GDPR), a privacy law meant to give users more control over how companies collect and retain their personal data, but only in the non-binding portion. Which means it doesn't really exist in Europe, either, says Sandra Wachter, a lawyer and assistant professor in data ethics and internet regulation at the Oxford Internet Institute.
GDPR’s shortcomings haven’t stopped Wachter from exploring what the right to explanation might look like in the future, though. In an article published in the Harvard Journal of Law & Technology earlier this year, Wachter, along with Brent Mittelstadt and Chris Russell, argue that algorithms should offer people “counterfactual explanations,” or disclose how they came to their decision and provide the smallest change “that can be made to obtain a desirable outcome.”
For example, an algorithm that calculates loan approvals should explain not only why you were denied credit, but also what you can do to reverse the decision. It should say that you were denied the loan for having too little in savings, and provide the minimum amount you would need to additionally save to be approved. Offering counterfactual explanations doesn’t require the researchers who designed an algorithm release the code that runs it. That’s because you don’t necessarily need to understand how a machine learning system works to know why it reached a certain decision.
“The industry fear is that [companies] will have to disclose their code,” says Wachter. “But if you think about the person who is actually affected by [the algorithm’s decision], they probably don’t think about the code. They’re more interested in the particular reasons for the decision.”
Counterfactual explanations could potentially be used to help conclude whether a machine learning tool is biased. For example, it would be easy to tell a recidivism algorithm was prejudiced if it indicated factors like a defendant’s race or zip code in explanations. Wachter’s paper has been cited by Google AI researchers and also by what is now called the European Data Protection Board, the EU body that works on GDPR.
A group of computer scientists has developed a variation on Wachter’s counterfactual explanations proposal, which was presented at the International Conference for Machine Learning’s Fairness, Accountability and Transparency conference this summer. They argue that rather offering explanations, AI should be built to provide "recourse," or the ability for people to feasibly modify the outcome of an algorithmic decision. This would be the difference, for example, between a job application that only recommends you obtain a college degree to get the position, versus one that says you need to change your gender or age.
“No one agrees on what an ‘explanation’ is, and explanations aren’t always useful,” says Berk Ustun, the lead author of the paper and a postdoctoral fellow at Harvard University. Recourse, as they define it, is something researchers can actually test.
As part of their work, Ustun and his colleagues created a toolkit computer scientists and policymakers can use to calculate whether or not a linear algorithm provides recourse. For example, a health care company could see if their AI uses things like marital status or race as deciding factors—things people can’t easily modify. The researchers’ work has already garnered attention from Canadian government officials.
Simply because an algorithm offers recourse, however, doesn’t mean it’s fair. It’s possible an algorithm offers more achievable recourse to wealthier people, or to younger people, or to men. A woman might need to lose far more weight for a health care AI to offer her a lower premium rate than a man would, for example. Or a loan algorithm might require black applicants have more in savings to be approved than white applicants.
“The goal of creating a more inclusive and elastic society can actually be stymied by algorithms that make it harder for people to gain access to social resources,” says Alex Spangher, a PhD student at Carnegie Mellon University and an author on the paper.
There are other ways for AI to be unfair that explanations or recourse alone wouldn't solve. That’s because providing explanations doesn’t do anything to address which variables automated systems take into consideration in the first place. As a society we still need to decide what data should be allowed for algorithms to use to make inferences. In some cases, discrimination laws may prevent using categories like race or gender, but it's possible that proxies for those same categories are still utilized, like zip codes.
Corporations collect lots of types of data, some of which may strike consumers as invasive or unreasonable. For example, should a furniture retailer be allowed to take into consideration what type of smartphone you have when determining whether you receive a loan? Should Facebook be able to automatically detect when it thinks you’re feeling suicidal? In addition to arguing for a right to explanation, Wachter has also written that we need a “right to reasonable inferences.”
Building a fair algorithm also doesn’t do anything to address a wider system or society that may be unjust. In June, for example, Reuters reported that ICE altered a computer algorithm used since 2013 to recommend whether an immigrant facing deportation should be detained or released while awaiting their court date. The federal agency removed the “release” recommendation entirely—though staff could still override the computer if they chose—which contributed to a surge in the number of detained immigrants. Even if the algorithm had been designed fairly in the first place (and researchers found it wasn’t), that wouldn't have prevented it from being modified.
“The question of ‘What it means for an algorithm to be fair?’ does not have a technical answer alone,” says Elish. “It matters what social processes are in place around that algorithm.”
- How the US fought China's cybertheft—with a Chinese spy
- Turning California’s weed into the champagne of cannabis
- Inside the secret conference plotting to launch flying cars
- Cities team up to offer broadband and the FCC is mad
- PHOTOS: The space shuttle program's golden age
- Get even more of our inside scoops with our weekly Backchannel newsletter
