Technology

The real problem is robots.txt is an honour system in the first place - It's never been a defence against bad (or even simply poor faith) actors.

Am I missing something in this article? I'm not defending either company, but it doesn't seem like they actually have any evidence to confirm either is doing this.

The world's top two AI startups are ignoring requests by media publishers to stop scraping their web content for free model training data, Business Insider has learned.

It claims this, but then they say this about the source of this info:

TollBit, a startup aiming to broker paid licensing deals between publishers and AI companies, found several AI companies are acting in this way and informed certain large publishers in a Friday letter, which was reported earlier by Reuters. The letter did not include the names of any of the AI companies accused of skirting the rule.

So their source doesn't actually say which companies are doing this, but then they jump straight into this:

AI companies, including OpenAI and Anthropic, are simply choosing to "bypass" robots.txt in order to retrieve or scrape all of the content from a given website or page.

So they're just concluding that based on nothing and reporting it as fact?

Novice web site owner/coder here: wondering if I can block them somehow via IP address in addition to robots.txt. Server firewall rule? Remember, I said I was a novice....

TollBit, a startup aiming to broker paid licensing deals between publishers and AI companies.

If we can't scrape data freely, it instantly kills the open source scene. These regulation only benefit companies like OpenAi and Google, who will happily pay an exorbitant price to have exclusive rights on data they don't already own and get a monopoly in return, as well as the companies who own this data like Reddit, Getty, Adobe, etc.

Getting a dime was never in the cards for individuals except maybe the outliers like GRR who can throw their weight around.

Almost all regulation being proposed only benefit big AI companies and are meant to kill any competition. They are flooding the media with bad sentiment articles to manipulate people so they can tell congress their constituents want this.

So, the same thing media websites do when they ignore my "do not track" request?

Yes, they’re evil. Was there a question?

Oh boy, if they're ignoring robots txt, then I better ...add a useless link at the bottom of every comment I make. That'll really show them!

If you put it online, you gave it away. If someone reads it and then uses that information to answer questions for someone else without giving credit to the author, that’s called a conversation. As long as no copyrights are being abused, there is no problem and this is just corporations upset with what they think is piracy, pandering to people who are still on the fence about AI.