What I've named the Taber & Lodge trainwreck for now:

1. Prior attitude effect
2. Disconfirmation bias
3. Confirmation bias
4. Attitude polarization
5. Attitude strength effect
6. Sophistication effect
   • whereby a clever arguer who knows about biases and fallacies is more likely to end up stuck in their beliefs

Subvert this trainwreck.

#2 & #3 look easy-ish to subvert: simply try to spend more time denigrating supportive evidence than seems natural & make a habit of seeking out contrary sources to learn more about a topic – it's often an efficient way to learn anyway.

For #4: the habit of seeing equivalently-weighted options as an unimportant choice (because they're worth about the same to you). Also known as the habit of flipping a coin to settle difficult decisions.

For #5: taking a dim view of strongly-professed opinions as if a moron is speaking (because this is often the case) – to counterbalance our instinctive feeling that a strong opinion-haver must know what they're talking about.

Never teach people the following knowledge unless they take seriously that knowing about biases can harm you and are the sort of people to take steps such as above. Some of the following knowledge may be unnecessary anyway, since You don't need to know about biases to debias.

• calibration
• overconfidence effects
• …

School doesn't spend much time on diff-eqns, but Carr says they have lots of use in data science & statistics.

It's a tragedy that these words sound so similar. We have the words "truth" and "lying", totally different vowels and consonants. But we have "rational" and "rationalization". It's as if lying was called "truthization", so that we have "truth" and "truthization" and must constantly remind ourselves that the one is nothing like the other.

Rationalization is impossible. No matter how much you want to, no matter how smart you are, you cannot make rational an idea that wasn't originated in a rational way. You cannot make a lie true. You can only swap out the lie entirely for the truth, just as you can only start over the search for solutions and do the search rationally this time.

A thought experiment, from www.greaterwrong.com/posts/9f5EXt8KNNxTAihtZ/a-rational-argument

You are, by occupation, a campaign manager, and you’ve just been hired by Mortimer Q. Snodgrass, the Green candidate for Mayor of Hadleyburg. As a campaign manager reading a book on rationality, one question lies foremost on your mind: “How can I construct an impeccable rational argument that Mortimer Q. Snodgrass is the best candidate for Mayor of Hadleyburg?”

Sorry. It can’t be done.

“What?” you cry. “But what if I use only valid support to construct my structure of reason? What if every fact I cite is true to the best of my knowledge, and relevant evidence under Bayes’s Rule?”

Sorry. It still can’t be done. You defeated yourself the instant you specified your argument’s conclusion in advance.

This year, the Hadleyburg Trumpet sent out a 16-item questionnaire to all mayoral candidates, with questions like “Can you paint with all the colors of the wind?” and “Did you inhale?” Alas, the Trumpet’s offices are destroyed by a meteorite before publication. It’s a pity, since your own candidate, Mortimer Q. Snodgrass, compares well to his opponents on 15 out of 16 questions. The only sticking point was Question 11, “Are you now, or have you ever been, a supervillain?”

So you are tempted to publish the questionnaire as part of your own campaign literature . . . with the 11th question omitted, of course.

Which crosses the line between rationality and rationalization. It is no longer possible for the voters to condition on the facts alone; they must condition on the additional fact of their presentation, and infer the existence of hidden evidence.

Indeed, you crossed the line at the point where you considered whether the questionnaire was favorable or unfavorable to your candidate, before deciding whether to publish it. “What!” you cry. “A campaign should publish facts unfavorable to their candidate?” But put yourself in the shoes of a voter, still trying to select a candidate—why would you censor useful information? You wouldn’t, if you were genuinely curious. If you were flowing forward from the evidence to an unknown choice of candidate, rather than flowing backward from a fixed candidate to determine the arguments.

[…]

If you really want to present an honest, rational argument for your candidate, in a political campaign, there is only one way to do it:

• Before anyone hires you, gather up all the evidence you can about the different candidates.
• Make a checklist which you, yourself, will use to decide which candidate seems best.
• Process the checklist.
• Go to the winning candidate.
• Offer to become their campaign manager.
• When they ask for campaign literature, print out your checklist.

Only in this way can you offer a rational chain of argument, one whose bottom line was written flowing forward from the lines above it. Whatever actually decides your bottom line is the only thing you can honestly write on the lines above.

Differences between rationalization and rational:

The purpose of my personal wiki is Cruxiness,

No two rationalists can agree to disagree.

EY proposes a norm that allows asking "is that your true rejection?" (be careful though because publicly psychoanalyzing someone can degenerate the conversation fast)

Ideal disagreers ask themselves what is their true rejection – they seek out their cruxes (perhaps with a technique such as Internal Double Crux). Or disagreers can play Double Crux with each other. They have learned to ask themselves how and where they got a belief in the first place and skip arguments that sound good today, and can feel the difference between their true rejection and an argument they came up with just now.

Disagreement can often be traced back to one of the following or other reasons:

• Uncommon, but well-supported, scientific knowledge or math;
• Long inferential distances;
• Hard-to-verbalize intuitions, perhaps stemming from specific visualizations;
• Zeitgeists inherited from a profession (that may have good reason for it);
• Patterns perceptually recognized from experience;
• Sheer habits of thought;
• Emotional commitments to believing in a particular outcome;
• Fear that a past mistake could be disproved;
• Deep self-deception for the sake of pride or other personal benefits.

People may find it embarrassing to talk about some of these in the moment, which is partly why you can't expect to resolve every disagreement in the moment – they also need alone time to sort themselves out. But you can at least ask about some of them, e.g. like this: "is that simple straightforward-sounding reason your true rejection, or does it come from intuition-X or professional-zeigeist-Y?"

www.greaterwrong.com/posts/TGux5Fhcd7GmTfNGC/is-that-your-true-rejection