Belief-in-belief

Carl Sagan parable: dragon in garage. Someone claims they have one. Fascinating—let's set out for it at once!

Sagan's moral: fast footwork

Eliezer's moral: there's a correct model of reality somewhere in their head

Not all beliefs are direct anticipations.

But they have consequences which are anticipations, if they're rational. But many ppl keep a pool of propositional beliefs from which they don't demand rent, that can contradict each other. The virtue of empiricism is a countermeasure. Anyway, there's no answer to "but what do they really believe?" because minds are tangled.

---

What links here

• Human heuristics, biases and fallacies
• Teaching examples used by Yudkowsky
• Belief in Belief
• "It's raining outside but I don't believe it is"