Hi Lugh,
thanks for this nice link (and article).
Researchers in the domain express increasing worryiness on this topic :
"... Reinforcement learning (RL) agents that plan over a long time horizon far more effectively than humans present particular risks. (...)
https://www.science.org/doi/10.1126/science.adl0625
(i don't have access to the full article)
My opinion is that we should hope and worry (at least a little bit).
This sounds like marketing hype. Giving AI reasoning is a problem researchers have been failing to solve since Marvin Minsky in the 1960s, and there is still no fundamental breakthrough on the horizon. Even DeepMind's latest effort is tame; it just suggests getting AI to check itself more accurately against external sources.
there is still no fundamental breakthrough on the horizon.
I mean, we're currently in the midst of one, so that might be obscuring the horizon somewhat. Modern AIs are able to reason in ways that no AIs could previously, don't let the perfect be the enemy of the good.
I think the recent LLMs are a breakthrough. Back in the 1970s people used expert systems. And you needed to put in lots and lots of if-then-rules by hand. Do the tedious job of creating knowledgebases and semantic connections... And the results were pretty limited. Nowadays you just feed in all the texts from the internet, do some magic and it'll autocomplete "This is how you change the oil in your car in 7 easy steps" and it'll autocomplete. They can write emails or roleplay a helpful assistant. Translating text is something that works quite well. All of that was unthinkable 10 years ago and utter scifi. The transformer architecture from 2017 has been some good progress. And I don't see why they shouldn't improve in capabilities in the time to come.