Belief-in-belief

Carl Sagan parable: dragon in garage. Someone claims they have one. Fascinating—let's set out for it at once!

Sagan's moral: fast footwork

Eliezer's moral: there's a correct model of reality somewhere in their head

Not all beliefs are direct anticipations. (But they have consequences which are anticipations, if they're rational – but many ppl keep a pool of propositional beliefs from which they don't demand rent, which can contradict each other. The virtue of empiricism is a countermeasure. Anyway, there's no answer to "but what do they really believe?" because minds are tangled.)

"Belief" is not limited to verbal sentences, it includes all anticipation-controllers.

What links here

• Human heuristics, biases and fallacies
• Teaching examples used by Yudkowsky
• *Belief in Belief
• "It's raining outside but I don't believe it is"
• TODO *Don't Believe You'll Self-Deceive