379
Reddit is now blocking major search engines and AI bots — except the ones that pay
(www.theverge.com)
just begin with site:reddit.com test for ddg and it still works
this post was submitted on 25 Jul 2024
379 points (98.7% liked)
Technology
36030 readers
30 users here now
This is the official technology community of Lemmy.ml for all news related to creation and use of technology, and to facilitate civil, meaningful discussion around it.
Ask in DM before posting product reviews or ads. All such posts otherwise are subject to removal.
Rules:
1: All Lemmy rules apply
2: Do not post low effort posts
3: NEVER post naziped*gore stuff
4: Always post article URLs or their archived version URLs as sources, NOT screenshots. Help the blind users.
5: personal rants of Big Tech CEOs like Elon Musk are unwelcome (does not include posts about their companies affecting wide range of people)
6: no advertisement posts unless verified as legitimate and non-exploitative/non-consumerist
7: crypto related posts, unless essential, are disallowed
founded 5 years ago
MODERATORS
Based on my testing if you filter results by the last week or last day you get nothing. Past month works.
Are they new posts or old ones? They are blocking new ones, not old ones.
new posts do not work
this post in /r/selfhosted is from 8hr ago: SWEKIT v0.1 - an open source library to build software engineering agents (DEVIN) in a agentic framework agnostic manner!
reddit/redlib: https://redlib.kylrth.com/r/selfhosted/comments/1eb86lf/swekit_v01_an_open_source_library_to_build/
doesn’t appear in DDG results: https://duckduckgo.com/?q=site%3Areddit.com+SWEKIT+v0.1+-+an+open+source+library+to+build+software+engineering+agents+%28DEVIN%29+in+a+agentic+framework+agnostic+manner%21&t=ffit
load more comments
(2 replies)
The cycle continues:
- Hey you guys can have everything for free
- WTF this is expensive to provide, I think I’m gonna start taking advantage of you guys which someone will pay me to do
- WTF where’s everyone going
- WTF I’m still losing money and always have been
- Screw you guys, screw everybody, I didn’t want y’all anyway
- (fades into irrelevance, gets bought by someone and stripped for parts)
Idk it’s not as pithy as Cory Doctorow’s version I guess
Anyway we’re at step 5 at this point
honestly I'm not convinced step 6 is inevitable. I think enough people are okay with whatever reddit does.
load more comments
(6 replies)
Yeah, Reddit is Digging its own grave.
It's getting Fark'd
load more comments
(2 replies)
Would lemmy instances do this?
I know they can't afford to now, but hypothetically? A lot of people here don't seem to like data scraping for AI.
You don't need to scrape. If you want to get all the content on Lemmy, just set up an instance and subscribe to all the top communities, and the instances will just send you all the content.
So there isn't really a way to monetise or block it. I guess you could only federate to a whitelist, but the biggest instances will federate by default with any new instances until they are given a reason to defederate.
Some Lemmy instances disallow indexing in robots.txt, however indexers can choose to ignore that and actually blocking them takes a lot more effort.
Some places on a "budget" like Ao3 just rate limit hard.
I don't like that solution at all though.
Your Lemmy posts are already being scraped for AI
The level of effort it would take to prevent would be infeasible to ask of even a non volunteer admin let alone a volunteer let alone literally all of them
Your Lemmy posts are already being scraped for AI
Good, hopefully it’ll make AI that is slightly less toxic than the rest of the internet.
It always baffles me that people don’t want their content represented in an AI - every word you write that gets indexed is a vote for how future AI will behave.
load more comments
(2 replies)
That's what I figured, but I am envisioning a future where lemmy is huge and the network of admins is quite sizable.
I guess that doesn't change much?
- Run Lemmy instance
- Gain userbase
- Intercept data users are reading and posting from your instance and others
- Feed to AI
- Profit?
Lemmy is way less privacy oriented than reddit and that's by design.
load more comments
(1 replies)
The only way they get my clicks now are when I Google something and they come up.
They really keep making sure that I don't end up there.
libredirect helps with that on desktop
(browser extension that turns links to sites like reddit, youtube, etc into links to redlib, invidious)