TechTakes

1533 readers

47 users here now

Big brain tech dude got yet another clueless take over at HackerNews etc? Here's the place to vent. Orange site, VC foolishness, all welcome.

This is not debate club. Unless it’s amusing debate.

For actually-good tech, you want our NotAwfulTech community

founded 2 years ago

MODERATORS

[email protected]

LLM training bots are a plague (pod.geraspora.de)

submitted 1 week ago by [email protected] to c/[email protected]

14 comments fedilink hide all child comments

Evidence for the DDoS attack that bigtech LLM scrapers actually are.

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 0 points 1 week ago (1 children)

The only simple possibles ways are:

robot.txt
rate limiting by ip
blocking by user agent

From the article, they try to bypass all of them:

They also don't give a single flying fuck about robots.txt ...

If you try to rate-limit them, they'll just switch to other IPs all the time. If you try to block them by User Agent string, they'll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet.

It then become a game of whac a mole with big tech 😓

The more infuriating for me is that it's done by the big names, and not some random startup.

[–] [email protected] 0 points 1 week ago

I do believe there's blocklists for their IPs out there, that should mitigate things a little