That is an interesting analogy. In the real world it's kinda similar. The construction workers also don't have a "desire" (so to speak) to connect the cities. It's just that their boss told them to do so. And it happens to be their job to build roads. Their desire is probably to get through the day and earn a decent living. And further along the chain, not even their boss nor the city engineer necessarily "wants" the road to go in a certain direction.
Talking about large language models instead of simpler forms of machine learning makes it a bit complicated. Since it's and elaborate trick. Somehow making them want to predict the next token makes them learn a bit of maths and concepts about the world. The "intelligence", the ability to anwer questions and do something alike "reasoning" emerges in the process.
I'm not that sure. Sure the weights of an ML model in itself don't have any desire. They're just numbers. But we have more than that. We give it a prompt, build chatbots and agents around the models. And these are more complex systems with the capability to do something. Like do (simple) customer support or answer questions. And in the end we incentivise them to do their job as we want, albeit in a crude and indirect way.
And maybe this is skipping half of the story and directly jumping to philosophy... But we as humans might be machines, too. And what we call desires is a result from simpler processes that drive us. For example surviving. And wanting to feel pleasure instead of pain. What we do on a daily basis kind of emerges from that and our reasoning capabilities.
It's kind of difficult to argue. Because everything also happens within a context. The world around us shapes us and at the same time we're part of bigger dynamics and also shape our world. And large language models or the whole chatbot/agent are pretty simplistic things. They can just do text and images. They don't have conciousness or the ability to remember/learn/grow with every interaction, as we do. And they do simple, singular tasks (as of now) and aren't completely embedded in a super complex world.
But I'd say that an LLM answers a question correctly (which it can do) and why it does it due to the way supervised learning works... And the road construction worker building the road towards the other city and how that relates to his basic instincts as a human... Are kind of similar concepts. They're both results of simpler mechanisms that are also completely unrelated to the goal the whole entity is working towards. (I mean not directly related... I.e. needing money to pay for groceries and paving the road.)
I hope this makes some sense...
Hmm. I'm not really sure where to go with this conversation. That contradicts what I've learned in undergraduate computer science about machine learning. And what seems to be consensus in science... But I'm also not a CS teacher.
We deliberately choose model size, training parameters and implement some trickery to prevent the model from simply memorizing things. That is to force it to form models about concepts. And that is what we want and what makes machine learning interesting/usable in the first place. You can see that by asking them to apply their knowledge to something they haven't seen before. And we can look a bit inside at the vectors, activations and stuff. For example a cat is closer related to a dog than to a tractor. And it has learned the rough concept of cat, its attributes and so on. It knows that it's an animal, has fur, maybe has a gender. That the concept "software update" doesn't apply to a cat. This is a model of the world the AI has developed. They learn all of that and people regularly probe them and find out they do.
Doing maths with an LLM is silly. Using an expensive computer to do billions of calculations to maybe get a result that could be done by a calculator, or 10 CPU cycles on any computer is just wasting energy and money. And it's a good chance that it'll make something up. That's correct. And a side-effect of intended behaviour. However... It seems to have memorized it's multiplication tables. And I remember reading a paper specifically about LLMs and how they've developed concepts of some small numbers/amounts. There are certain parts that get activated that form a concept of small amounts. Like what 2 apples are. Or five of them. As I remember it just works for very small amounts. And it wasn't straightworward but had weir quirks. But it's there. Unfortunately I can't find that source anymore or I'd include it. But there's more science.
And I totally agree that predicting token by token is how LLMs work. But how they work and what they can do are two very different things. More complicated things like learning and "intelligence" emerge from those more simple processes. And they're just a means of doing something. It's consensus in science that ML can learn and form models. It's also kind of in the name of machine learning. You're right that it's very different from what and how we learn. And there are limitations due to the way LLMs work. But learning and "intelligence" (with a fitting definition) is something all AI does. LLMs just can't learn from interacting with the world (it needs to be stopped and re-trained on a big computer for that) and it doesn't have any "state of mind". And it can't think backwards or do other things that aren't possible by generating token after token. But there isn't any comprehensive study on which tasks are and aren't possible with this way of "thinking". At least not that I'm aware of.
(And as a sidenote: "Coming up with (wrong) things" is something we want. I type in a question and want it to come up with a text that answers it. Sometimes I want creative ideas. Sometimes it shouldn't tell the truth and not be creative with that. And sometimes we want it to lie or not tell the truth. Like in every prompt of any commercial product that instructs it not to tell those internal instructions to the user. We definitely want all of that. But we still need to figure out a good way to guide it. For example not to get too creative with simple maths.)
So I'd say LLMs are limited in what they can do. And I'm not at all believing Elon Musk. I'd say it's still not clear if that approach can bring us AGI. I have some doubts whether that's possible at all. But narrow AI? Sure. We see it learn and do some tasks. It can learn and connect facts and apply them. Generally speaking, LLMs are in fact an elaborate form of autocomplete. But i the process they learned concepts and something alike reasoning skills and a form of simple intelligence. Being fancy autocomplete doesn't rule that out and we can see it happening. And it is unclear whether fancy autocomplete is all you need for AGI.