www.greaterwrong.com/posts/6FmqiAgS8h4EJm86s/how-to-convince-me-that-2-2-3

Hilary Putnam's Twin Earth thought experiment on conceivability vs logical possibility seems to touch on this ( www.greaterwrong.com/posts/ne6Ra62FB9ACHGSuh/heat-vs-motion):

Once we have discovered that water (in the actual world) is H₂O, nothing counts as a possible world in which water isn’t H₂O. In particular, if a “logically possible” statement is one that holds in some “logically possible world”, it isn’t logically possible that water isn’t H₂O.

On the other hand, we can perfectly well imagine having experiences that would convince us (and that would make it rational to believe that) water isn’t H₂O. In that sense, it is conceivable that water isn’t H₂O. It is conceivable but it isn’t logically possible! Conceivability is no proof of logical possibility.

www.greaterwrong.com/posts/XC7Kry5q6CD9TyG4K/no-evolutions-for-corporations-or-nanodevices

One of the misunderstandings of evolution: there's a meme going around that all kinds of things can evolve by natural selection, not just DNA/RNA but also self-copying nanodevices and even things like human corporations. After all, the market is a battleground where survival of the fittest applies, right? Presto, evolution! With time, the corps that survive will be stronger and stronger.

Except no. Many things have to go right for a force like evolution to meaningfully apply. A big thing missing from corps and nanodevices is copying fidelity.

Look. A corp may be outcompeted by another corp who learns the lesson their enemy failed to learn – but that one's successor in turn will easily have never learned this specific lesson, and so be vulnerable to the same mistake. The same way you carry many of your parents' life-lessons but not so many of your great-grandparents' life-lessons, especially if you've never met them, and so you repeat your great-grandparents' mistakes. The same way that post-WW2 was a great time for human rights, but a few generations later fascism is on the rise. Current generations do not have the life-lessons to see through the farce: the life-lessons failed to copy themselves for more than 2-3 generations.

All the old lessons of history, you are doomed to repeat them again and again.

—Leto II, God Emperor of Dune

Contrast with DNA evolution. Consider a genetic mutation that confers a 3% increase in fitness on average. DNA copies perfectly, so after about 768 generations(!), every member of the species has inherited it – and all copies are exactly identical (if just one molecule sits differently, we regard it a different gene).

DNA copies with something like 10^-8 errors per copy, several orders better copying fidelity than we can do with any machine for now – I think we're at 10^-2 or something? Our nanomachines would have something like a million times the mutation rate (i.e. rate of copying errors), which isn't viable for life.

www.greaterwrong.com/posts/q7Me34xvSG3Wm97As/but-there-s-still-a-chance-right

One of the chief morals of the mathematical analogy between thermodynamics and cognition is that the constraints of probability are inescapable; probability may be a subjective state of mind, but the laws of probability are harder than steel.

[…] Here's a little experiment: Smash an egg on the floor. The rule that says that the egg won't spontaneously reform and leap back into your hand is merely probabilistic. A suggestion, if you will.

[…] It may help to think of it this way—if you still have some lingering intuition that uncertain beliefs are not authoritative:

In reality, there may be a very small chance that the egg spontaneously reforms. But you cannot expect it to reform. You must expect it to smash. Your mandatory belief is that the egg's probability of spontaneous reformation is ~0. Probabilities are not certainties, but the laws of probability are theorems.

If you doubt this, try dropping the egg on the floor a few decillion times.

–www.greaterwrong.com/posts/zFuCxbY9E2E8HTbfZ/perpetual-motion-beliefs

A popular idea of "rationality" is Spock from Star Trek. Yet Spock is much more like a sub-par human statistician than a rational being, if you follow Cognitive science's definition of rationality. He's merely the autist of the TV-show, the straw Vulcan, and if anyone is the rationalist, it's Captain Kirk. Why? He makes the decision that saves everyone's butts, over and over again.

The Hollywood-Rationalist Spock says something like: "our probability of survival if we go into that nebula is 2.234%". Then Kirk says to go into it anyway, and nine episodes out of ten, the Enterprise is fine! Clearly he knew something Spock didn't.

Kirk doesn't survive despite irrationality; on the contrary, because he survives so reliably, you can see that his choices are pretty rational.

Rationality is about winning. It's making the decision that saves the Enterprise; the decision with the greatest chance of success, whatever you mean by "success", and no matter if you don't have an exact idea of the numbers involved.

It's not about ignoring your emotions and ignoring what your "gut" tells you in favour of a deliberate probability or utility calculation (though it can be!). If your gut has been reliable in the past in similar situations, why would you think it's not rational to listen to it?

Spock is not how a rationalist sounds… but he's often what an aspiring rationalist sounds like. The final form is Kirk.

Which leads us to another trope, emotionlessness. Yudkowsky:

For my part, I label an emotion as "not rational" if it rests on mistaken beliefs, or rather, on mistake-producing epistemic conduct: "If the iron approaches your face, and you believe it is hot, and it is cool, the Way opposes your fear. If the iron approaches your face, and you believe it is cool, and it is hot, the Way opposes your calm."