Yes, trying to solve this problem by creating a definition of "misinformation" sufficiently precise to act on it for all possible articles, then trying to remove it somehow is probably an AI-complete problem. If it's not AI-complete it's probably just plain an ill-formed question. That's before we ask questions about the biases that get embedded into the misinformation detector.
This can't be fixed at Facebook scales by building a platform that so highly incentivizes low-quality content of all kinds, then trying to stop the content so late in the cycle. The entire incentive structure has to be rethought. Unfortunately, that's probably a problem that Facebook is literally incapable of solving without going out of business, because Facebook the corporate entity is that incentive structure. To fix Facebook requires Facebook to become not-Facebook.
This is a problem with human wetware - there are several cognitive biases [1][2][3][4] that make people engage more strongly with ideas that fit their preconceptions and discount ideas that don't. Add information-sharing to the mix and the aggregate effect you get is large information cascades that fracture the population into tribes that believe things not necessarily because they're true, but because they are stated early, vehemently, and frequently.
Remember that evolution favors individuals who survive & gain status, not those whose representation of the world is true. Someone who is right but killed for it (eg. Copernicus) is still dead.
The fix is for people to be aware of these cognitive biases in themselves, to actively seek out information that differs from their preconceptions, and to challenge inaccurate information (with evidence!) when presented with it. You can't outsource this to a communication platform, because it's inherently less comfortable and more effort for the reader, and the reader will just go to a competing communication platform that makes them feel better.
Maybe people confuse Copernicus with the advocate of related theories Giordano Bruno, who was killed in 1600 for his beliefs and advocacy (though not only for his beliefs about astronomy).
> The fix is for people to be aware of these cognitive biases in themselves, to actively seek out information that differs from their preconceptions, and to challenge inaccurate information (with evidence!) when presented with it.
You are right, but I hope you are aware that this would require a giant leap to an essentially superhuman state of rationality and clear thinking for a vast majority of people.
Just stop for a moment and look around you. Is that how most people actually function in this world?
Separate ideology (or whatever happens to be the object of bias) from identity, and you're halfway there. People are much more amenable to changing their minds if their identities aren't at stake. For example, most people can discuss the pros and cons of F-150s and Silverados fairly rationally; those with the Calvin pissing logos, not so much.
On a societal level, your guess is good as mine. Perhaps increased infrastructure spending so that high-skill, high-pay jobs are more evenly spread out and ideological segregation is reduced?
On an individual level, being active in (non-political) hobbies and volunteer groups can help, especially ones that attract both people from the cities and the countryside.
Who am I stereotyping? People with "Calvin pissing on Ford/Chevy/Ram logo" stickers? The vast majority of truck drivers don't really care about brand--they just want the most appropriate truck for their situation.
None of us function in this way. While part of the solution does have to involve people becoming better at vetting their information-sources, it is equally important to acknowledge that for everyone to do this individually would be absolutely impossible given time constraints.
What we need, fundamentally, is a media environment that can harness the collective capacity of humans to vet and debate new information in an organized fashion that can be trusted. This site is a good example where a knowledgeable community, and the simple mechanism of votes does some automatic, crowd-based content curation. But the Reddit-like model is very basic - so much more could possibly be done to allow individual users to contribute to a collective process of knowledge-building.
I think we need more development and discussion devoted to what kind of systems could be designed to encourage transparency, quality, and critical analysis in our news-making.
As a kid I learnt this and tried to be as rational and aware of my fallibalities as possible.
That makes you an alien to normal human beings. Conversation becomes impossible, processing fast enough becomes impossible, and the penalties for acting weird and communicating weird are huge.
Instead you need to find a way to act in a manner that allows you to blend in with people.
This is absolutely the right answer, but maybe Facebook can do some things to help. It's possible to gamify the process of learning to spot biases by presenting people with situations and getting them to think critically to solve the biases or fallacies. Especially on Facebook, people love filling out questionnaires that tell them something about themselves.
I'm not saying that one smart game is going to fix the entire problem, but adding the development of logic skills into the types of activities that people enjoy doing on Facebook is definitely possible.
It doesn't work that way. It assumes that what people are doing is wrong - it's not. The mental overhead to assess everything clinically is sufficient that faster thinkers can just outrun you. People/brains are making trade offs to deal with the world around them.
Other humans are taking advantage of those gaps to mobilize votes or advertisements.
There's no way out and I've been wondering when people would realize the depth and intricacy of the mess we're in. Hopefully people pay more attention to the way cognitive cheat code are being abused.
There's really an amazing amount of misinformation and manipulation going on. I think there's a place for people developing critical thinking skills, but there's also a limit to what people can sort through. There's a big need for more software to fact-check and compare sources across the Internet. Having Facebook run it is one part of the solution, but people need to have transparent software that they control to do the same kind of work so it isn't entirely centralized. Google, Amazon, and Facebook assistants are great, but we need personal assistants that work for us.
The fix is for people to be aware of these cognitive biases in themselves, to actively seek out information that differs from their preconceptions, and to challenge inaccurate information (with evidence!) when presented with it.
Exactly right. This should be strongly emphasized in our educational curriculum.
As any alien arriving in your world would immediately ask, why doesn't having a true representation of the world help you survive and gain status? And if it doesn't, why does it matter?
It's almost as if solving the problem of truth is antithetical to the notion of having a for-profit entity be the main channel of [legit|mis-]information in our world.
It's almost as if the way we understand and use money has hardened into a giant screwball of lies, base instincts, and perverse incentives.
I think I would argue that this is beyond an AI-complete problem, it is an unanswerable problem. For instance, consider the following spectrum of ideas:
- The earth is flat
- 9/11 conspiracy theories
- Barack Obama was born in Kenya
- Anti-vax
- Climate change denial
- Anti-GMO
- Denial of underlying quantum randomness in the universe
- The Clinton Foundation has been or is involved in some shady dealings
- Muslims want to implement Sharia law in Europe
- The minimum wage decreases employment
- Donald Trump has sexually assaulted a number of women
- Monetary policy impacts the real economy
- The sun will, on balance, probably rise tomorrow
I've sorted this in a (rough, personal) order of increasing plausibility / conformance with facts. Where is the line between misinformation and alternative information? As a human, it's pretty unclear to me.
Man, it must be me, but you ranked "Monetary policy" WAY more likely than I would (rates near zero for so long doing so little must at least put SOME doubt in your model), and "denial of quantum randomness" way less credible than I see it as. Like, man, reasonable people can disagree about pilot waves and many worlds.
Are there some confusing, emotionally laden, difficult questions of fact in there? Yes. Are some things ultimately unknowable (like what people want, which involves scrying into their minds)? Yes.
These, as it turns out, are not the things I can't help but be side-tracked on. I'm diverted by you claiming monetary policy has an effect with nearly the certainty of the sun rising.
Haha, sorry I knew this list would cause this sort of trouble. I'm a big believer in pilot wave theory myself, it's just very much not the 'mainstream' view. And I didn't mean to imply that the last two were close together, either. I do think monetary policy probably has some effect (whether we understand or can model that effect is another matter), but I didn't mean at all to imply that it was anywhere near the certainty of the sun rising, it just happened to be the closest :).
I think perhaps we disagree more about the probability of these two:
- The Clinton Foundation has been or is involved in some shady dealings
- Muslims want to implement Sharia law in Europe
Than the other two. I think both of those are, in some sense, almost certainly true. There are Muslims who want to implement Sharia law in Europe. The Clinton Foundation has been involved in transactions that at least have the appearance of moral uncertainty. The headlines that I listed are just overly strong statements/generalizations of those (IMO undeniable) facts.
I keep getting sidetracked by other issues, but you're statements are about certainty and uncertainty and where do you draw the line for heavy-handed intervention, so I might as well get into them:
There, Clinton denies the foundation has been engaged in anything shady. Boom, definitely not undeniable #hackernewsdrama, hahaha.
If you want a disagreement, though, I prefer many-worlds over pilot wave theories, largely because I'm not entirely sure _what_ the propagation speed of the pilot waves would be. 'Greater than the speed of light' either (1) doesn't really narrow it down at all or (2) narrows it down to having no options at all. But they should have _a_ speed, because I'm not sure you can even have standing waves if the disturbances propagate instantly.
Rates near zero is part of the reason the stock market has been on a tear the last few years. It also was a driver of the housing bubble.
It didn't cause the hyperinflation that a lot of people thought would come, but the Great Depression is Bernanke's area of expertise, and he had sound theoretical reasons to believe QE wouldn't spark serious inflation.
Rates were also comparatively low during the 2000s to juice the economy after the tech bust and 9/11. They got even lower after the crash and QE, but they were the lowest they had been in the 00s since the early 60s.
"This problem isn't that hard. You filter this stuff and rank it by its SOURCE, not by the content of individual articles."
I don't think reifying ad hominem into code is the solution to the problem.
Of course, if you don't mind a rare few false positives here and there on articles, I've got your filter right here:
def source_is_trustworthy(source):
return False
Remember as you sit here thinking through your exceptions what we're talking about; if your "trustworthy" source was confidently telling you about how Clinton was going to win, it's not one of the exceptions. I won't say I was confident Trump would win, but I was certainly less suprised than most; the fact that I generated this somewhat more predictive model by tossing out pretty much every "mainstream" news source is not a good sign for them. And to be honest, Trump news isn't the only thing that I find this useful for. Beyond the bare facts, most media outlets really aren't good for much anymore, and do you ever have to de-spin their news just to get those bare facts in the first place. (I can't, however, pack up into a recipe how to do this yourself. I think we're in a period of transition in the news industry, much greater than just "the internet makes the old dinosaurs stumble" makes it sound, and I'm at the moment not really all that confident in anything.)
As someone who vehemently opposes Trump, I feel that allegations of anti-Trump bias in the mainstream media were entirely correct and somewhat to be expected. Although Trump is certainly a name that sells papers, Trump's repeated threats to open up news media to broader libel laws were not well received. The news organizations' endorsements were pretty one-sided[0]. I don't want to take you too literally, but it seems to me pretty obvious that a model ignoring the MSM entirely would not be preferable to one that took into account the MSM opinions and then applied a correctional factor. I'm sure we agree that truth is a function of our means for determining truth, but I suspect we disagree strongly on the reliability of Internet news sources.
You must have missed the leaked emails in which Clinton exerts massive influence on the media, planning every last details in exchange for all kinds of rewards [0]
I did miss that. Was that article intended to support that view? The incidents listed don't seem to have been terribly well planned or executed, or to have garnered any positive results. That seems difficult to reconcile with the idea of a powerful media conspiracy.
"source_is_trustworthy" applies to all sources, not just "MSM". As I said, I don't have anything right now that I can point to and say "If they say it, I trust it."
I used to try to apply a correctional factor, but I've found it not even worthwhile anymore. I honestly don't know entirely why. I don't know if the news has gotten that much more biased to the point where it almost swamps the signal entirely, or if the combined financial pressures away from expensive real reporting and towards click-bait headlines has removed the signal, or what, but beyond very bare facts they just aren't worth much anymore. It is also possible this has always been the case and I wasn't aware enough to realize it; there's some classic history stories that back that possibility up.
I do know primary sources are getting easier and easier to consult directly. Which could itself also be a reason my opinion of them has plummeted so much in the past 10 years; it was a lot harder to see through media spin and simplification (deliberately for their audience, and accidentally when they in fact don't understand what's going on themselves, see especially science journalism for that) when they were the only source of information.
It won't be reached in practice, but if you make it the policy, you have an argument to take down any bullshit. With 50:50 it's going to be an endless quarrel over who has had enough so far, who is underrepresented and why everything is dominated by two polar opposites with no voice given to people who don't want to belong to either camp.
The case of news sources failing to predict events seems like a red herring in the larger debate here - news media is generally about reporting events that have actually happened. Predicting presidential elections or other uncertain events is a side hustle they've been dragged into because it sells. And if you throw out their predictive failures, I think it's pretty easy to compose a trustworthiness function based on overall factual accuracy. And sure, there may be lingering biases that need to be corrected for, but even so, the MSM biases generally don't extend into the realm of pure fabrication, whereas clickbait outrage farms absolutely do - and they are a large chunk of the problem.
PageRank is also not unbiased, it has been played before many times and pure PageRank would be pretty easy to play - that's why Google keeps tweaking it years after it was invented. And still SEO industry exists. Now, SEOs just compete on being on the first page, not claiming something as grandiose as "truth" - add that to the mix, and all the biases there are, and you get pretty much hopeless task. And of course you get under a constant stream of criticism, and maybe also government regulation - it's one thing to just produce search results (even that is controversial, remember "right to forget"?), it's another thing to claim being able to "detect falsehood" - that's very sure a way to get sued and for the government to get involved.
To determine correctness, the adjudicator requires knowledge outside the system. In the scientific method, this is done by generating falsifiable hypotheses and performing the experiment. Failing that, you have to fall back on heuristics, like "what I already believe", Occam's razor (simplest theory consistent with observations wins), or reliance on authority/consensus.
Consistency is simply majority wins, which completely stifles new ideas or minority opinions.
> Consistency is simply majority wins, which completely stifles new ideas or minority opinions.
No isn't.
You don't need to have a "non objective view of the truth" - i..e you don't need the ability to ask the system "Is this statement true?"
You just need the ability to ask the system "Will I think this is true, given the other things I have claimed to be true?"
Maintaining internal complexity of your own claim graph gets really tricky unless you keep it as accurate as possible.
For example, consensus view was that trump would not win the election. Trump did win the election. Thus, those two claims:
* (before) Trump will not win
* (later) Trump won
These two are inconsistent. It's possible to rectify that inconsistency by marking the earlier statment cancelled. So now we have three statements
* (before) Trump will not win
* (later) Trump won
* (now) I was wrong to say, "Trump will not win."
The gap between "now" and "Before" can be used to compute "probability of a statement being retracted." The smaller those gaps, the more likely it is what whatever you say comes with an asterisk next to it, saying "the expected time to retraction of this statement is: x days"
Ah, you're only speaking of temporal consistency (or "constancy" - does the source's statement change over time).
I was speaking of consistency with the general body of knowledge in the system, (correspondence?).
In the specific case of evaluating major news sources for trust, I think the vast majority of their statements are never retracted/changed, and therefore they would rate highly on an overall "constancy" scale as well. Their many statements about Trump's likelihood of winning were very constant as well - for months they said he'd lose, then they changed, and forever after they will say he won.
Sounds very temporally consistent, with just a single change.
Not sure if the adjudicator could narrow it just to "trust of the source's future predictions about politics", but basically the whole country just got a negative feedback on the trust weighting for major media in that domain.
Are you referring to wrong when they described what they thought was going on, which contains a large amount of opinion, or wrong on the facts as they exist?
Facts can be presented in a biases way (generally through omission of other contextual facts), but if what is said is factual, at least that bit of information is still correct. If combining multiple sources to generate a grouping of facts, that presentation bias way well disappear. If system weeds out people that present opinion as fact, all the better.
Frankly papers that lie through crafted contexts are much more dangerous because they give a false sense of authority due to their correctness at the micro level but incorrectness at the macro level.
Imagine I develope a newspaper that only reports crimes committed by a particular race and I excruciatingly fact check every incident and highlight when they are uneducated (but don't mention education otherwise). What will result is one of the biggest macro lies that will fuel racism through its broader message while everything will be perfect to a fact checker. This would rank perfectly in such a system while broader newspapers would get hammered for messing up picture captions and confusing Airbus A319s with 737s.
Using the source to figure out whether something is accurate/correct has its own issues unfortunately. These include:
1. Editors/writers/management changes, since quite a few media publications have gone from being fairly reliable to biased as all hell based purely on who's taken over there. Or in some (rarer cases), the opposite has happened.
So you'd have to make sure older less biased pieces weren't being punished for the actions of the publication in the present.
2. A lot of sources are correct about some things but not about others. It's very possible to have a publication that's terrible at writing about politics in an accurate and unbiased way, but fantastic at writing about technology or sports. So ideally your system would have to detect the subject of the article as well as the publication.
However, I guess you could at least add known 'satire' and fake information sites to a 'non credible' list. Like the ones listed here:
this assumes their interest is in being fair, but I suspect their goal will gravitate mostly toward 'not being sued/investigated' which may very well create a selection bias.
Non-starter. The NYTimes and Washington Post, for example, are respected but have a very strong yet subtle bias.
On the one hand, they seem more interested in fact-checking. On the other, they propagate concepts like: on-balance the US is a force for good and should lead the world. This is highly contentious and non-obvious for many (most?) people outside the US.
Pretty much all mainstream media outlets did, at some point, publish something that can be qualified as "bad propaganda" by at least some of the observers. Now, either you filter out pretty much all the internet except maybe for bare statistics, numbers, math, etc., or you say "well, these guys are not as bad as those guys, because obviously being completely wrong about X much worse than being slightly less than right about Y", at which point you are just implementing your own biases.
Yes. We're talking about a platform that at it's best is birthday wishes, vacation and cat pictures. It's running on a paid content model, with advertisers selling through viral videos and click-bait. It's good not to expect too much out of it.
We're actually not; that's simply false. No, Facebook is not "at it's best" [sic] when it sticks to birthday wishes and cat pictures.
It's actually at its best when it is truly social, in all senses of the word. When it facilitates real communication on real issues across a broad spectrum of people. When it spreads information, not misinformation.
Not trying to speak for the original poster, but what I took away from the argument is that when FB is "truly social", involving real issues, the potential rewards are sufficiently high that misinformation becomes valuable, thus dangerous. When it sticks to birthdays and cat pictures, there's no incentive to go hostile. Think of it as being somewhat similar to the arguments about the Flash Crash regarding HFT and liquidity: that HFT is great for liquidity until you need it, at which point it goes away[1].
I agree with others that detecting misinformation in the general case likely requires a theory of mind, and so something approaching "real" AI. I also agree that FB has a serious problem on their hands. And I'm so very glad I don't use it.
[1] I advance no comment here about the correctness of that; just a comparison to illustrate what seems like an overlooked point.
This is the same as reddit. As social platforms get larger they starts diverging from their original userbase (programmers/geeks for reddit, college students for facebook) to apply to a wider and wider audience. Eventually the wider audience starts to look a whole lot like the demographics of a whole country (or maybe the world), and it becomes a platform with as many arguments as society in general.
The solution to not becoming a battleground which is inherantly destabilizing to the platform is to not grow, or only grow in a single direction. State up front what your targeted user base is and keep to them. NeoGAF as a "gamers only" forum will survive a very long time, for instance.
Facebook has sidestepped it by becoming a personal community for each user, but whenever they try to connect that personal community to the wider world through "shared" news, "shared" trends etc that are not personalized it causes a big ruckus.
If it is truly social then it will spread misinformation like wildfire. That's what people do -- gossip game magnified. That said, maybe Facebook is best when it is just spreading obviously unimportant content (in the sense of national and global affairs), like birthday wishes and cat pictures. No harm if the cat is implicated in wild conspiracy theories.
It's not the cat pic sharing that is paid for by entities with motive and interest in spreading misleading messages.
When Facebook is at it's best is taking money from nefarious actors and fucking up democracy
In Poland, for good half year before our presidential election, then a year after I was getting obviously paid for posts from dozens upon dozens of right wing extremists.
I am left-centrist (although my dad jokes at me that im a bolshevik). I don't have many friends who skew righ-wing either. Yet, there was my wall, chock full of sensationalist bs, for over A YEAR
Wonder how much money Facebook earned in that time just from sad little Poland
I have no doubt that a lot of social information is being transferred in a new way, much the same as the introduction of the telephone allowed a lot of social information to be transferred in a new way. My point is that true socializing and socialization are human, not digital experiences.
Not trying to be pedantic here. It's just important to sort out the terms used if any analysis is going to be useful. What's happening on facebook is not social. It's bits of data around being social. Different thing entirely. Perhaps we need a new term for whatever it is, but whatever we call it, it's not the same as face-to-face real socializing with real people.
Are you picking on me? Making a joke that we share? Challenging me to come up with a retort? Asking me to join in to a shared humorous narrative? Giving me a friendly jibe about an error or weakness you perceive?
We assume positive intent online because otherwise we get flame wars, but there's a helluva lot of body language and inflection that can take the comment you made and make it mean a dozen things. All of that is lost. And all of it is important.
In addition to the nuance, would I know severine if I saw them on the street? Miss them if they died? Remember the nice way they did X everytime we met?
We are exchanging written opinions and statements, yes. These exchanges also occur in social interactions. But we are not socializing. There's no bond here that's become stronger, no subtle nuance or interplay that we're managing at a subconscious level. For all either of us knows, the other one is a bot. Or an alien. There's no humanity here.
I disagree. For me, and I think for many, online interactions are a just as much a form of socialization as going out in person. For very small groups, I generally prefer the in person version, but for larger groups, I think I get more out of the online setting.
Let's take our relationship, for example. I've been cohabiting some of the same online spaces as you for at least the last decade. For me, you are a valued part of that ecosystem, more so than most of my physical neighbors who I chat with in person a couple times a year.
No, I wouldn't recognize you on the street, but I struggle with face blindness, so it's a weak criterion. Would I "miss you" if you died? No, but I'd certainly reflect sadly on your death if I learned about it. "Remember the nice way you did X"? No, but I have trouble coming up with such an X most people I interact with locally.
There's no humanity here.
I'm surprised you would say this. Is this a recent change in your view of online interactions or have you always felt this way? Quantitively I find about the same amount of humanity here as I do elsewhere, and qualitatively I prefer many things about what I find here to what I see when I go out in person. Are you OK?
(ps. I notice that many of the links in the sidebar of your site are broken.)
Facebook completely controls the presentation priority, no? Bolt a Watson-like truth scoring system on to evaluate content and use the resulting confidence to boost or bury content in other's feeds.
It would absolutely have to be automatically generated, but it doesn't seem impossible (just much harder NLP that Watson playing Jeopardy).
If the scoring system does anything other than maximize engagement with Facebook, Facebook ends up making less money and performing worse on KPIs than they otherwise would. It'd be very difficult for any decision maker at Facebook to build, deploy, and maintain that feature. It'd be the sort of thing that'd only be there if otherwise the company would be under existential threat.
You mean government regulation a la China because some memes really are dangerous? If you increase the velocity of transmission, then even in a free system there are some dangerously infectious falsehoods that it might be in the public good to slow. Think War of the World on Facebook.
Ah, but herein lies the difference. Memes seem to universally promote freedom of communication, communication about freedom, and a way to react to the oppression of the status quo. In China, this is counter to the powers that be, but in America, this is fully aligned with the anti-regulation party that is sweeping through all branches of government. Memes play into their hands, and so Facebook would never be under existential threat.
That, or a public reaction to their product (ie, clickbait fatigue - Facebook gets really good at local optimization for the most engaging news posts but creates a news feed that convinces people that Facebook as a whole isn't worthwhile).
It basically needs to be serious enough to overwhelm the internal incentives of maximizing key performance indicators.
I think there's the global maximum vs local maxima argument in favor of doing something too. Why do people ultimately spend time on Facebook?
I would say they don't do so because of micro-optimized KPI targeting: they do so because they feel time on Facebook is valuable and makes them happy. If Facebook can't deliver increases in that, then optimizations only take them so far.
Yup. There's definitely some Goodhart's Law going on here as well. Micro-optimized KPI targeting is an inexact measure of feeling like ones time on Facebook is valuable.
Watson scoring something for truth is only as good as the data drawn from. Who decides that data? Jeopardy questions center around long-established, non-controversial information; how does a Watson-like system evaluate unprecedented breaking news? It's not a question of the language used, it's a much, much higher-order issue that current Watson sidesteps.
Agreed, much more difficult, but ultimately the same way we do so? We believe certain ideas to be facts (hopefully based on scientific sources), then we parse incoming information in light of those.
I can't imagine an unprecedented breaking news story that has no basis in any factual information. I'm not talking about an upvoter here, but primarily a downvoter (incongruence with accepted facts being easier to prove than vice versus).
Sure, and that's the higher-order problem. If we could make machines that solve any problem the way we do, we'd be a lot closer to AGI, but for now, it's still science fiction for a computer to have beliefs, and understand information in the context of those beliefs.
It seems to me the hardest part would be separating truthiness from virality. How do they seed legit sources in the current media climate of racing to the bottom for eyeballs.
Well, to be more specific about what I mean is, Google's algorithm is successful if it finds the pages that people find valuable. It's sort of baked into the value proposition that the end user will be able to gauge the quality of results, so there is a sort of feedback loop. The problem with misinformation is there is no in-band signal for Google about its truthiness. I suppose there heuristics you can use related to fact-checking quality of certain publications, etc, but it's much trickier than general perceived quality.
Nevertheless, I agree that if anyone can do it algorithmically, it would probably be Google.
No, it doesn't. Sometimes, popularity blocks the process of scientific inquiry. As Kuhn pointed out. But in any case, there is no truth, as Popper pointed out.
This is basically impossible given the current and near future state of the art.
The Allen AI group has been working on solving year 4 multiple choice science tests for about 3 years. They now score a bit over 60% correct, and that is a much easier task.
What I'm suggesting we start with is not exactly the same. The analogous question in that domain is "How accurate have they been in eliminating wrong answers?"
Everything has to be right for an answer option to be correct. Only one thing has to be wrong for it to be incorrect.
Like most false stories, there is a hint of truth to this but that doesn't make it true. Politfact spends 3 pages discussing it, and concludes:
Courts did object in three cases to decisions Kander made as secretary of state. But the specifics of those cases are much more nuanced than Blunt lets on. And the claim that Kander tried to manipulate the election is, at best, unproven.
A SOTA computer system could probably parse that last paragraph I posted, but couldn't get anywhere near getting close to doing the research needed to get to that conclusion. We are years off that (I work on research in this area)
I'm not surprised parsing something nuanced is beyond state of the art. But what about "illegal immigrants are committing an epidemic of rape against our children" or "we have a flood of illegal immigrants"?
A lot of claims that were made this cycle are rooted in measured quantities where the measurements we have disagree with the claims.
Define "epidemic of rape" or "flood of illegal immigrants".
If there is at least one rape or at least one illegal immigrant then it becomes subjective or contextual.
Is 3 rapes a flood? What if it is in a week in one town? What if one was by a former illegal immigrant? What if it is over the course of a year, but all 3 occurred in one day?
Is 20 illegal immigrants a flood? What if they all arrive in a town with a population of 100?
Who is "we"? Is a story about millions of illegal immigrants in Greece relevant? What if it about thousands of illegal immigrants working in Athens? What if Athens is in Georgia?
A real example:
"Broken Families: Raids Hit Athens' Immigrant Community Hard" vs "Athens crackdown shows no hospitality for illegal migrants"
One is a real headline from Georgia, the other from Greece. Can you tell which is which?
They can just form a group of truth checker, a ministry of sort, who's job is to inform people of the truth. They can call it the Ministry of Truth, where busy citizens who have no time to research can get their truths from.
Fortunately, you don't really have to solve that problem. It would be sufficient to assess claims of actual fact about the actual world. I think the first step in writing the classifier would be to write something like
Your statement is that of an atheist. You don't seem to realize that many of the followers of these religions believe their assertions to be factual in every way. When I got off the subway today, at the Port Authority, in New York City, there were 2 women, holding up signs about Jesus. One of the signs literally said "Historical fact: Jesus rose from the dead".
Is it established that that is in fact what Twitter does? I see something non-chronological on Twitter and it hasn't been clear for years what criteria are used.
This can't be fixed at Facebook scales by building a platform that so highly incentivizes low-quality content of all kinds, then trying to stop the content so late in the cycle. The entire incentive structure has to be rethought. Unfortunately, that's probably a problem that Facebook is literally incapable of solving without going out of business, because Facebook the corporate entity is that incentive structure. To fix Facebook requires Facebook to become not-Facebook.