www.greaterwrong.com/posts/veN86cBhoe7mBxXLk/categorizing-has-consequences

www.greaterwrong.com/posts/6FmqiAgS8h4EJm86s/how-to-convince-me-that-2-2-3

Hilary Putnam's Twin Earth thought experiment on conceivability vs logical possibility seems to touch on this ( www.greaterwrong.com/posts/ne6Ra62FB9ACHGSuh/heat-vs-motion):

Once we have discovered that water (in the actual world) is H₂O, nothing counts as a possible world in which water isn’t H₂O. In particular, if a “logically possible” statement is one that holds in some “logically possible world”, it isn’t logically possible that water isn’t H₂O.

On the other hand, we can perfectly well imagine having experiences that would convince us (and that would make it rational to believe that) water isn’t H₂O. In that sense, it is conceivable that water isn’t H₂O. It is conceivable but it isn’t logically possible! Conceivability is no proof of logical possibility.

www.greaterwrong.com/posts/XC7Kry5q6CD9TyG4K/no-evolutions-for-corporations-or-nanodevices

One of the misunderstandings of evolution: there's a meme going around that all kinds of things can evolve by natural selection, not just DNA/RNA but also self-copying nanodevices and even things like human corporations. After all, the market is a battleground where survival of the fittest applies, right? Presto, evolution! With time, the corps that survive will be stronger and stronger.

Except no. Many things have to go right for a force like evolution to meaningfully apply. A big thing missing from corps and nanodevices is copying fidelity.

Look. A corp may be outcompeted by another corp who learns the lesson their enemy failed to learn – but that one's successor in turn will easily have never learned this specific lesson, and so be vulnerable to the same mistake. The same way you carry many of your parents' life-lessons but not so many of your great-grandparents' life-lessons, especially if you've never met them, and so you repeat your great-grandparents' mistakes. The same way that post-WW2 was a great time for human rights, but a few generations later fascism is on the rise. Current generations do not have the life-lessons to see through the farce: the life-lessons failed to copy themselves for more than 2-3 generations.

All the old lessons of history, you are doomed to repeat them again and again.

—Leto II, God Emperor of Dune

Contrast with DNA evolution. Consider a genetic mutation that confers a 3% increase in fitness on average. DNA copies perfectly, so after about 768 generations(!), every member of the species has inherited it – and all copies are exactly identical (if just one molecule sits differently, we regard it a different gene).

DNA copies with something like 10^-8 errors per copy, several orders better copying fidelity than we can do with any machine for now – I think we're at 10^-2 or something? Our nanomachines would have something like a million times the mutation rate (i.e. rate of copying errors), which isn't viable for life.

www.greaterwrong.com/posts/q7Me34xvSG3Wm97As/but-there-s-still-a-chance-right

One of the chief morals of the mathematical analogy between thermodynamics and cognition is that the constraints of probability are inescapable; probability may be a subjective state of mind, but the laws of probability are harder than steel.

[…] Here's a little experiment: Smash an egg on the floor. The rule that says that the egg won't spontaneously reform and leap back into your hand is merely probabilistic. A suggestion, if you will.

[…] It may help to think of it this way—if you still have some lingering intuition that uncertain beliefs are not authoritative:

In reality, there may be a very small chance that the egg spontaneously reforms. But you cannot expect it to reform. You must expect it to smash. Your mandatory belief is that the egg's probability of spontaneous reformation is ~0. Probabilities are not certainties, but the laws of probability are theorems.

If you doubt this, try dropping the egg on the floor a few decillion times.

–www.greaterwrong.com/posts/zFuCxbY9E2E8HTbfZ/perpetual-motion-beliefs