Whyso? I defer to your experience, but I would expect that incorrectness is simply inconsistency measured against a more complete/complex basis?

To determine correctness, the adjudicator requires knowledge outside the system. In the scientific method, this is done by generating falsifiable hypotheses and performing the experiment. Failing that, you have to fall back on heuristics, like "what I already believe", Occam's razor (simplest theory consistent with observations wins), or reliance on authority/consensus.

Consistency is simply majority wins, which completely stifles new ideas or minority opinions.

> Consistency is simply majority wins, which completely stifles new ideas or minority opinions.

No isn't.

You don't need to have a "non objective view of the truth" - i..e you don't need the ability to ask the system "Is this statement true?"

You just need the ability to ask the system "Will I think this is true, given the other things I have claimed to be true?"

Maintaining internal complexity of your own claim graph gets really tricky unless you keep it as accurate as possible.

For example, consensus view was that trump would not win the election. Trump did win the election. Thus, those two claims:

* (before) Trump will not win * (later) Trump won

These two are inconsistent. It's possible to rectify that inconsistency by marking the earlier statment cancelled. So now we have three statements

* (before) Trump will not win * (later) Trump won * (now) I was wrong to say, "Trump will not win."

The gap between "now" and "Before" can be used to compute "probability of a statement being retracted." The smaller those gaps, the more likely it is what whatever you say comes with an asterisk next to it, saying "the expected time to retraction of this statement is: x days"

Ah, you're only speaking of temporal consistency (or "constancy" - does the source's statement change over time).

I was speaking of consistency with the general body of knowledge in the system, (correspondence?).

In the specific case of evaluating major news sources for trust, I think the vast majority of their statements are never retracted/changed, and therefore they would rate highly on an overall "constancy" scale as well. Their many statements about Trump's likelihood of winning were very constant as well - for months they said he'd lose, then they changed, and forever after they will say he won.

Sounds very temporally consistent, with just a single change.

Not sure if the adjudicator could narrow it just to "trust of the source's future predictions about politics", but basically the whole country just got a negative feedback on the trust weighting for major media in that domain.

I am not saying build a machine to detect correctness, I'm saying build one to detect incorrectness.