The game Watch Dogs 2 is actually quite on target regarding this. In the game, the TPTB puts a non-scientific bias into predictive algorithms for certain neighbourhoods making the population of those areas pay more in insurance and be targeted by police a lot more often.
It's a dangerous path to take as suddenly you can be labeled a criminal just for living in the wrong place or even worse if the police put their bias into the algorithms.
History would be repeating itself. This was one of the factors in the burning down of much of the south Bronx in the 1970s. The city was broke and asked RAND Corporation to scientifically determine the least necessary firehouses to close.
Their methods were in retrospect biased and just plain inaccurate, as firehouses and police precincts in poor neighborhoods were closed just as crime and fires were spiking, creating some sick sort of positive feedback loop. There are zip codes in the south Bronx that lost 90%+ of their housing. It's incredible.
Beware of those who would use the word "science" to justify their preferences without explaining in detail.
The systems also encode in existing discrimination.
For example, women are less likely to be investigated for a crime, charged with a crime, convicted of a crime, and receive lower sentences when convicted. In some cases (especially involving sex crimes) there is explicit sex based bias written into the law itself. This means a lot of data on who commits crimes has a strong sexist bias, so any automated algorithms have a strong possibility of reinforcing the existing bias. The same will happen for race, class, and many other factors.
There are also cases where society wants an incorrect bias put into place. For example, parole risk assessment software vastly underrates the threat of certain classes of criminals compared to what society and police think it should because there are major myths about rehabilitation and recidivism that are as popular as they are wrong.
Perhaps the worst part is a total lack of transparency in the existing algorithms that determine risk. With enough data you can reverse engineer it, but that doesn't give the same impact as seeing the rules themselves. For example, one parole group I developed software for had a risk rating that appeared to automatically rate women a risk level lower for the same crime. Imagine if the same was done based on race.
"...Imagine if the same was done based on race...."
Well, the same is done based on race, they just don't need to write that into the software.
The point I'm trying to make is that it's going to be impossible to eliminate all bias in the system, but people are justified in their aversion to systems that automate bias into our system. Palantir certainly has the potential to do so, and the total lack of transparency definitely doesn't agitate in its favor.
While there are many parole officers who use race to make unfair judgments, this is considered wrong by the system and work is taken to stop it (one of the reasons systems like the one I mentioned was being adopted). Compare this to the system I was talking about where it was recognized official policy and considered good, right, and just. Imagine if some police department came out and openly admitting that their official policy is to use race to punish someone harsher for the exact same crime.
It's a dangerous path to take as suddenly you can be labeled a criminal just for living in the wrong place or even worse if the police put their bias into the algorithms.