A.I. and tackling the risk of “digital redlining”

As more and more insurers rely on A.I.-driven models, they face a critical issue on applying fair coverage.

This is the web version of Eye on A.I., Fortune’s weekly newsletter on artificial intelligence and machine learning. To get it delivered weekly to your in-box, sign up here.

Last week, a Dutch court ordered the government in the Netherlands to stop using a machine-learning algorithm for detecting welfare fraud, citing human rights violations.

The system, called System Risk Indicator (SyRI) in English, was being used by four Dutch cities to spot individuals whose benefits applications should receive extra scrutiny. It gathered information from 17 different government data sources, including tax records, vehicle registrations and land registries.

But the cities using SyRI did not run every application through the system—they only deployed it in poor neighborhoods where many residents are immigrants, often from Muslim countries.

The court ruled that SyRI violated the “right to private life” enshrined in European human rights law. The application of SyRI, it said, could lead to discrimination against individuals based on their socio-economic status, ethnicity or religion. It also said SyRI did not seem consistent with the requirements of Europe’s stringent data privacy law, GDPR.

Although the judgment only came from a district court and is subject to possible appeal, the decision is likely to set an important precedent within European Union—and it ought to reverberate elsewhere too, as societies around the world come to grips with how to apply fairness in a world of A.I.-driven risk models.

Nowhere is this more relevant than in the insurance sector, which is turning to machine-learning algorithms more and more in order to improve underwriting. Last week, I had a fascinating conversation with Daniel Schreiber, the co-founder and CEO of the New York-based insurance startup Lemonade. He shares concerns that the increased use of machine-learning algorithms, if mishandled, could lead to “digital redlining,” as some consumer and privacy right advocates fear.

But done right—and with the right measure of fairness—he thinks machine learning has the potential to increase access to financial services and decrease cost.

To ensure that an A.I.-led underwriting process is fair, Schreiber promotes the use of a “uniform loss ratio.” If a company is engaging in fair underwriting practices, its loss ratio, or the amount it pays out in claims divided by the amount it collects in premiums, should be constant across race, gender, sexual orientation, religion and ethnicity.

He admits that this means it is entirely possible that some categories of people—Schreiber, who is Jewish, uses the example of Jews—could be charged more on average for property insurance, because, for instance, their religious practice involves lighting candles in the home for certain holidays, and lighting candles might be correlated with a higher risk of house fire.

But, he says, no individual should be charged more because he or she is Jewish. It might turn out that a particular customer isn’t religious and doesn’t light candles. That’s why it is important not to ask people about their religious affiliation—that would be discriminatory. The key is for the insurance company to gather data that actually equates to risk: Do you light candles in your home?

In order for it to work properly, insurance companies will need to gather more data about customers, not less. Right now, Schreiber admits, the regulatory winds seem to be blowing in the opposite direction (especially in Europe, as the SyRI case shows). Most insurance regulators don’t understand machine learning. “That creates a fear of the unknown,” he says. What’s more, scandals such as Cambridge Analytica make people reluctant to share more data.

But Schreiber says customers might be willing to share more information if the insurers were transparent about why they needed to collect this data, how it was being used, and that it might result in customers paying a lower premium.

I wasn’t entirely convinced by Schreiber’s argument. If insurers become that much better at pricing risk, won’t many more people simply become uninsurable? (This is what happens in health insurance if companies are allowed to cherry-pick customers, excluding those with pre-existing conditions.)

Also, won’t people who live in impoverished neighborhoods still be forced to pay more for coverage, even though they may have little choice over where they can afford to live? Many poorer areas have higher risk of crime and fire, leading to higher home insurance premiums. (In fact, U.S. law prohibits policies that have a “disparate impact” on a protected class of people, unless a company can prove a legitimate business necessity for the policy.)

Schreiber told me that governments could mandate charging those who live in wealthy areas or who have high household incomes slightly more in premiums, and then using this excess to subsidize the premiums of those who live in poorer neighborhoods. But, he said, this was a discussion separate from the one about whether the underwriting model itself is fair.

What do you think? Feel free to write in and let us know your views.

Jeremy Kahn 
@jeremyakahn
jeremy.kahn@fortune.com

This story was originally featured on Fortune.com