Technology

Ok, but if you aren't assuming it's valid, there doesn't need to be evidence of invalidity. If you're demanding evidence of invalidity, you're claiming it's valid in the first place, which you said you aren't doing. In short: there is no need to disprove something which was not proved in the first place. It was claimed without any evidence besides the LLM's output, so it can be dismissed without any evidence. (For the record, I do think Google engages in monopolistic practices; I just disagree that the LLM's claim that this is true, is a valid argument).

To me, all the mental gymnastics about AI outputs being just meaningless nonsense or mere copying of others is a cop-out answer.

How much do you know about how LLMs work? Their outputs aren't nonsense or copying others directly; what they do is emulate the pattern of how we speak. This also results in them emulating the arguments that we make, and the opinions that we hold, etc., because we those are a part of what we say. But they aren't reasoning. They don't know they're making an argument, and they frequently "make mistakes" in doing so. They will easily say something like... I don't know, A=B, B=C, and D=E, so A=E, without realizing they've missed the critical step of C=D. It's not a cop-out to say they're unreliable; it's reality.