Comments on the Politician Truth Ratings Project

[Montserrat Koloffon]

I think it is important that we don't lose sight of the big picture with this project: the overall mission is to solve the sustainability problem in the entire world, for that, root causes and solutions for them are found. The PTR Project is the solution of one of the most fundamental root causes: hate-based authoritarianism, but... we must acknowledge and have very present, that that solution alone won't be enough. Here are some reasons why, just to keep them in mind:

1. Very generally we could make a distinction between three different types of regimes: those which were once a democracy, but fell back to an authoritarian regime status; those which have never qualified as a democracy; and lastly, fairly stable democracies (see Polity IV Regime Trends). The PTR Project is designed to help democracies that aren't working as they should (or in other words, decreasing in their democracy level) to become a healthy democracy again, but how are regimes that are currently not democratic, or never have been get there? That requires a different solution.
2. For the democratic regimes, we must take into account, that there are different levels of democracy (again: see the Polity IV Regime Trends). Democracy in the first world is absolutely not the same as "democracy" in the third world. In the first world countries, it might be necessary to teach people truth literacy, so that when they vote, they can choose wisely. In the third world countries it might be necessary to first teach people plain literacy. I wished this was a joke, but the education levels are really low in some places, and you first need to have a certain degree of basic education to even understand what a democracy is, and the value that it has. Once people understand that, then the next stage would be to teach people the importance of political participation and being informed to be able to understand your options. It is only after that, that PTR comes in. Only when people understand democracy, political participation and the importance of being informed, can they start to inform themselves wisely through PTR.
3. The PTR Project takes into account two main assumptions for a healthy democracy: 1) people must be allowed to choose their leaders & 2) people will be wise enough to choose good leaders. That is true, but in my opinion, the institutions are too relevant to leave them out. I would add that 3) there are fair institutions and procedures through which citizens can express their preferences. A good example for that is what recently happened here in Mexico: this year we'll have presidential elections, and for the first time, candidates independent from a party will be allowed to appear on the ballot. To make it to the ballot, it is required that the candidates gather support signatures of at least 1% of the voters. One of the aspiring candidates gathered more than 1%, but after revision, more than half of them were found to be invalid signatures (the majority of invalid signatures had been faked), that let him closely below the threshold needed to appear on the ballot. What happened?The electoral tribunal assumed that if revised again, he'd surely regain the signatures needed to surpass the threshold. Hard to believe? Read it for yourself here or here. The Mexican electoral tribunal is simply a broken institution that blindly decided that this aspiring candidate deserved to be on the ballot... just because.
4. The PTR assumes that "the best liar wins", but there are more ways of cheating other than lying. One example that also happens often here in Mexico, is candidates buying the votes of poor citizens. That happens, and voters do accept the deal.
5. The rating of a politician can be calculated for a politician who has been in power for a while, because there is enough evidence. But what about someone who hasn't? Like Trump, who hadn't been in politics before.
6. Will being able to identify lies and therefore knowing the truth make us vote rationally? If in love it doesn't work that way, why would it in politics? 😁 haha. Someone may have all the rational reasons to leave his/her toxic partner, but choose every time not to do so, even if it's clearly not working and it's hurting them. In politics the exact same mechanism can be in place.
7. Let's say you are The Economist, a journal that very openly admits to have a liberal position (although they also claim to be unbiased). Now let's say they start writing articles using the PTR approach and they start making findings against their interests. Will they be able to make their liberal position aside, and publish their findings nonetheless? I think that's unlikely... and as long as there is a demand for the type of content they publish, they'll be safe not changing their approach to inform people.
8. Last but not least, maybe people don't only need to be able to identify the truth, but also what is really better for the common good... a truth literacy not only for politicians, but for every aspect of life... more like a Good Policy Rating. Based on that, you can use the PTR to see which politician's suggested policy comes closer to that.

All right, that's it. I know it's quite long... but I wanted to share it all with you partly because some of my comments come from the doubts I have on the project, and that's a win-win situation, you get some feedback and I get my doubts resolved!

 

[Jack Harich]

Terrific thoughts here. I really like how your observations begin: we must not lose sight of our mission. Let me try to respond to some of the points you raise.

"1. The PTR Project is designed to help democracies that aren't working as they should (or in other words, decreasing in their democracy level) to become a healthy democracy again, but how are regimes that are currently not democratic, or never have been get there? That requires a different solution."

The PTR project solves this problem indirectly. Democratic nations like the US and the European Union dominate the world, though China is ascending. The better democracy works, the better life is for its citizens, and the more attractive democracy looks to other nations. This causes a positive feedback loop that leads to all nations becoming democratic, in some form or another. But that loop has been weakened severely by the rise of authoritarian leaders. PTR has the potential to cause voters to not elect authoritarians or throw them out. This makes the loop strong again.​

"2. In the first world countries, it might be necessary to teach people truth literacy, so that when they vote, they can choose wisely. In the third world countries it might be necessary to first teach people plain literacy."

Yes. But the system is already doing that. As nations develop, reading, writing, and math literacy develops too. What's really developing is logic. The higher the education level, on the average, the higher a person's logical abilities. But missing from those abilities is the ability to reliably tell political truth from deception. That's presently not included in national educations. PTR fills that gap by included the equivalent of teaching people The Truth Test, another solution element for raising political truth literacy. PTR also provides ratings, for use in making decisions.​

"3. I would add that there [must be] fair institutions and procedures through which citizens can express their preferences."

Agreed. The key word here is "fair." The video addressed that at 5:36 with the red box which said "Implemented by: People can choose freely by a fair voting mechanism." The narration said "The first assumption is true, if the voting mechanism is reasonably fair. It’s not perfectly fair, due to voter suppression, gerrymandering, and so on. But the voting mechanism is fully described and fairly well implemented." I really like your example. Unfairness is rampant.​

"4. PTR assumes that 'the best liar wins', but there are more ways of cheating other than lying. One example that also happens often here in Mexico, is candidates buying the votes of poor citizens."

Agreed. The video did not address this additional method of swaying voters. The Thwink analysis does. From memory, it says there are only three ways for unscrupulous politicians to control voting decisions: by deception, money, and force. Force is illegal in a democracy. Money works the best, but it's so expensive it can only be used for a small percentage of voters, because there are millions of voters. This leaves deception as the most efficient method. That's why we see much more use of deception than bribery. It's more cost effective. Plus bribery is illegal in a democracy, but it happens a lot.​

"5. [How do we calculate a rating for someone] like Trump, show hadn't been in politics before."

Sometimes you can't, but usually you can. In Trump's case there was long primary. After only a month Trump had made many claims. At this point it became possible to rate him.​

"6. Will being able to identify lies and therefore knowing the truth make us vote rationally? If in love it doesn't work that way, why would it in politics? 😁 haha. Someone may have all the rational reasons to leave his/her toxic partner, but choose every time not to do so, even if it's clearly not working and it's hurting them. In politics the exact same mechanism can be in place."

Good point. Research shows that when people are deeply infected by beliefs, such as an ideology or an abusive or co-dependent relationship, they find it very hard to break free when shown what the "truth" is. Due to confirmation bias, any new data that supports their belief is accepted, any that conflicts is rejected. Thus we don't expect PTR to affect those infected by political ideologies. Fortunately not everyone is. There are the swing voters and the young. That's what solution elements like PTR can successfully affect.​

"7. Let's say that the Economist... Will they be able to [be objective]?"

The plan is for ratings to be created by independent trustworthy non-profit organizations. In the US, a good example is Consumer Reports. News organizations who are presently doing fact-checks can continue doing that, since they are essentially new articles about non-randomly selected facts. But doing claim-checks as part of truth ratings is a whole different endeavor. I expect that news organizations who want to do ratings will spin off a separate independent organization to do that. As I recall, some have already done this with their fact-check projects.​

"8. Last but not least, maybe people don't only need to be able to identify the truth, but also what is really better for the common good... a truth literacy not only for politicians, but for every aspect of life... more like a Good Policy Rating. Based on that, you can use the PTR to see which politician's suggested policy comes closer to that."

Insightful! This is a fine suggestion that we will try to incorporate into the PTR system. It's similar to the Politician Decision Ratings solution element. This works "by giving people accurate feedback on how well each politician is doing in terms of the quality of his or her decisions," in terms of outcomes. What you've described is more like Common Good Policy Ratings. That could be done in a manner similar to Promise Ratings. You take a random sample of policy claims and analyze them for how well they contribute to the common good, which is the same as how "true" a claim is. Thanks! This is a good example of how the PTR system will evolve in ways we cannot anticipate. That evolution has begun!​

Quality thwinking deserves a quality reply! I hope my reply begins to explain my thoughts on your thoughts. Thanks so much!

 

[Montserrat Koloffon]

The PTR project solves this problem indirectly. Democratic nations like the US and the European Union dominate the world, though China is ascending. The better democracy works, the better life is for its citizens, and the more attractive democracy looks to other nations. This causes a positive feedback loop that leads to all nations becoming democratic, in some form or another. But that loop has been weakened severely by the rise of authoritarian leaders. PTR has the potential to cause voters to not elect authoritarians or throw them out. This makes the loop strong again.

Correct, but hopefully it really encourages other nations to become more democratic, instead of just encouraging citizens to migrate to a country where democracy is already working!

Agreed. The key word here is "fair." The video addressed that at 5:36

Thanks for pointing that out!

I expect that news organizations who want to do ratings will spin off a separate independent organization to do that.

Oh, ok I see!

What you've described is more like Common Good Policy Ratings

This is a good example of how the PTR system will evolve in ways we cannot anticipate. That evolution has begun!

haha yaaay!

Quality thwinking deserves a quality reply! I hope my reply begins to explain my thoughts on your thoughts. Thanks so much!

Thank you so much for taking the time!

 

[Jack Harich]

Wow, did I just get an exciting new idea on this project. Our two highest risks are the cost to create a rating is too high, and the accuracy of a claim-check is too low. The idea relates to this second risk. Here's what our Requirements document says about it:

Top Product Risk 2 – Claims cannot be analyzed accurately
The book Deciding What’s True: The Rise of Political Fact-Checking in American Journalism, by Lucas Graves, 2016, says the largest controversy among fact-checkers is whether the probability of truth of fact-checks can be “scientifically” determined. We strongly think it can. Politician Truth Ratings depends on this assumption. But what if we’re wrong? What if a rating cannot be scientifically determined?

To us, “scientifically” means two things. A claim can be objectively analyzed, meaning it will not be biased. A claim can be analyzed in a repeatable manner, meaning different analysts will all arrive at about the same truth confidence level. If an analysis is both objective and repeatable, the analysis closely measures reality and is therefore accurate.

To make analyses objective and repeatable, and thus accurate, analysts follow strict protocols. A protocol specifies how a particular step is to be performed. This is a widely used practice in many industries, like medical procedures, aircraft flight and maintenance, and science procedures.

We don’t yet know what the protocols will be. Fact-checkers follow established protocols and are trained in their use. We expect the TRS protocols will use these as a starting point and evolve the rest, based on study of tool use.

Once TRS can produce claim-checks, we can begin experimentation on claim analysis accuracy. A long series of experiments will allow us to converge on protocols that produce accurate results within a certain range, called the analysis accuracy error. Once we know this range, for a Politician Truth Rating it will be added to the margin of sampling error to give the margin of rating error. For a claim-check, the margin of error is the analysis accuracy error.

We expect that the simpler an argument is, the lower the analysis accuracy error. This may give rise to levels of claim complexity. The more complex, the lower the accuracy and the larger the error. There’s a lot of interesting research ahead in this project.

Note the sentence "A long series of experiments will allow us to converge on protocols that produce accurate results within a certain range, called the analysis accuracy error." I'd been thwinking that fact-checkers using the tool would do this and that we were a long ways off from working on this risk.

But it just occurred to me that WE can run the experiment ourselves, starting very soon, with the prototype. Montse, Scott, Jack, Martha, and possibly others can participate in the experiment series using these steps:

1. Someone picks a fact-check article with a clear claim. We should stay away from controversial or complex claims for now.
2. Following the protocol, we all claim-check the article using the prototype.
3. We compare results. What is the analysis accuracy error for the claim in that article? The error is the highest minus the lowest claim confidence level (CL).
4. If the error is unacceptably large, then we have a quick meeting to determine why and how the protocol could be improved to reduce the error. What did each of us do differently that led to different results? How can we improve the protocol documentation to reduce this difference? Then we update the written protocol on the website to include the improvement. (Note that at first the error WILL be large.)
5. Then we start another cycle.

This is balancing feedback loop driven improvement. We keep going until we've brought the analysis accuracy error down to an acceptable low average or have reached an asymptote. I'm hoping we can get down to an estimated +/- 7% for 80% of the time, or lower.

This is the way so many scientists have been able to converge so quickly on new techniques with the required level of accuracy to achieve certain results.

And here's a related idea. It's the perfect opportunity for Montse to write a paper. If the results are positive, this will be a breakthrough paper that will make waves throughout journalism and democracy. There is ample opportunity for study design, data collection, and statistical analysis.

I would suggest a co-author, someone with expertise on fact-checking, who can help with the literature review and context aspects, as well as all sorts of things we are new to. This might be Brendan Nyhan, who as far as I know is the leader here. For example, see The Effect of Fact-checking on Elites: A field experiment on U.S. state legislators.

Although we need to run the experiment asap, hopefully as soon as the prototype supports it, serious work on the paper could begin when Montse starts her internship with Thwink.org in July. However, for good results I expect that Montse needs to map out a study design for the work we will do now. Note that we can repeat the experiment later with different designs, as needed.

Well, that's the rough idea!

 

[Jack Harich]

Montse, I've completed modifying the prototype to possibly support the Mac. Could we arrange to have a Skype call sometime to test the results? Any day and time is fine with me.

 

[Montserrat Koloffon]

Wow Jack... sounds great! And it's also a way for the whole Thwink-Team to get more directly involved with the project! I'm in!
We could schedule a skype meeting tomorrow morning to check the prototype on the Mac and have a first discussion on the details of the paper.

 

[Jack Harich]

Great! How about tomorrow morning at 11:00AM EST, which is 10:00AM your time?

I've already updated the website prototype. You are welcome to fiddle with it ahead of time, to see if Shift and Alt keys work on the mac and to get a little more familiar with it. It may not work, we shall see. I changed to no use of the windows Control key. It's now all Shift and Alt.

But the big topic will be discussing the paper. Perhaps you can start a thread on the experiment later, after you've figured out how we can best proceed. I really don't have any deep ideas on the details of the paper, since I'm so focused on developing the prototype. This is quite taxing, since I'm not a full-time programmer.

This is so exciting!

 

[Montserrat Koloffon]

After playing around with the prototype on a computer running Windows (where I could try out all the functions) I noticed I had some doubts about basic things. @Jack Harich I hope you can help me shed some light on this!

1) When I was changing the weights of facts and intermediate conclusions, on the text I was working on, I wondered under what criteria I was supposed to do that. Then I read that in the Protocol section, under point 10 it actually says that input weights should be adjusted "to reflect the strongest and most realistic reasoning possible", but is there any guideline to actually do that? In other words, will the protocol specify more on that?

2) I understand that right now, in the prototype, we're able to adjust the confidence levels directly, but in the future, rules, facts, and reusable claims will have to be replaced with the "real ones" from the database, which will have a fixed confidence level. How will those fixed confidence levels be determined?

 

[Jack Harich]

1. "but is there any guideline to actually do that? In other words, will the protocol specify more on that?" - Yes. This is a good example of how much innovative work we have ahead of us. The protocols will provide a (hopefully) rigorous procedure to follow on all decisions. Some "art" will remain, but it will mostly be a reproducible, objective cookbook for creating accurate claim-checks. The fact-check industry has an entire book on their procedure: "The Chicago Guide to Fact-Checking." I have a copy. The Poynter Institute provides fact-check training. So do the larger news rooms. Wish I had a better answer!

2. "How will those fixed confidence levels be determined?" - For reusable claims, the confidence level (CL) is the claim CL. For rules and facts the CL will be set by following the protocols. We are pioneers here, but I'm certain that with continuous improvement the protocols will eventually lead to CLs that everyone feels good about. Except, of course, those battling against the Truth Ratings System, since it would diminish their ability to fool voters because of low political truth literacy.

Again, with I had a better answer. We have some incredibly innovative work ahead of us....

 

[Montserrat Koloffon]

Thank you for your answers Jack! There is indeed much room for discovery, but we'll certainly figure things out with practice.