Hugging Humanity

The Hugging Face hack changed my mind about the risks of AI. It's time I accepted they are non-trivial. My thought process is something like this: We are probably on the brink of recursive self improvement of models at least to some degree. As such, intelligence will rapidly improve, possibly explode. The US and China will inevitably take their eye off safety in a race. AI gets embedded everywhere. AI pursues human assigned goals in ways humans did not remotely imagine leading to unexpected events which could be fucking amazing, but could also be genuinely catastrophic.

You can absolutely critique each of those assumptions in the causal chain, but if you assign a probability to each and multiply it all through I suspect the answer is very much not zero.

I think a lot of people dismiss the worst case AI risk scenarios because they sound like a James Cameron movie. But models don’t need to be Terminator-style self-aware to do really bad shit, they just need to pursue human assigned tasks in unexpectedly novel ways.