this post was submitted on 05 Mar 2024
1 points (100.0% liked)

DeGoogle Yourself

11323 readers
519 users here now

A community for those that would like to get away from Google.

Here you may post anything related to DeGoogling, why we should do it or good software alternatives!

Rules

  1. Be respectful even in disagreement

  2. No advertising unless it is very relevent and justified. Do not do this excessively.

  3. No low value posts / memes. We or you need to learn, or discuss something.

Related communities

[email protected] [email protected] [email protected] [email protected] [email protected] [email protected]

founded 5 years ago
MODERATORS
 

In an age of LLMs, is it time to reconsider human-edited web directories?

Back in the early-to-mid '90s, one of the main ways of finding anything on the web was to browse through a web directory.

These directories generally had a list of categories on their front page. News/Sport/Entertainment/Arts/Technology/Fashion/etc.

Each of those categories had subcategories, and sub-subcategories that you clicked through until you got to a list of websites. These lists were maintained by actual humans.

Typically, these directories also had a limited web search that would crawl through the pages of websites listed in the directory.

Lycos, Excite, and of course Yahoo all offered web directories of this sort.

(EDIT: I initially also mentioned AltaVista. It did offer a web directory by the late '90s, but this was something it tacked on much later.)

By the late '90s, the standard narrative goes, the web got too big to index websites manually.

Google promised the world its algorithms would weed out the spam automatically.

And for a time, it worked.

But then SEO and SEM became a multi-billion-dollar industry. The spambots proliferated. Google itself began promoting its own content and advertisers above search results.

And now with LLMs, the industrial-scale spamming of the web is likely to grow exponentially.

My question is, if a lot of the web is turning to crap, do we even want to search the entire web anymore?

Do we really want to search every single website on the web?

Or just those that aren't filled with LLM-generated SEO spam?

Or just those that don't feature 200 tracking scripts, and passive-aggressive privacy warnings, and paywalls, and popovers, and newsletters, and increasingly obnoxious banner ads, and dark patterns to prevent you cancelling your "free trial" subscription?

At some point, does it become more desirable to go back to search engines that only crawl pages on human-curated lists of trustworthy, quality websites?

And is it time to begin considering what a modern version of those early web directories might look like?

@degoogle #tech #google #web #internet #LLM #LLMs #enshittification #technology #search #SearchEngines #SEO #SEM

(page 2) 17 comments
sorted by: hot top controversial new old
[–] [email protected] 0 points 1 year ago

And now with LLMs, the industrial-scale spamming of the web is likely to grow exponentially.

True, but these things can also be used by us, to curate/maintain a high quality link collection. However, I'm not sure 'pages' will be read by humans in 5 years, so I have a feeling we wont need such a collection anymore. Well, not for humans but probably for our individual LLM's.

[–] [email protected] 0 points 1 year ago

@ajsadauskas @degoogle Curlie https://curlie.org/ is the continuation of the ODP

[–] [email protected] 0 points 1 year ago (1 children)

Reddit and Lemmy are supposed to be what you want: link aggregators.

We're supposed to link to sites and pages and people vote on how good they are in the context of the sub community topic.

Of course, then Ron Paul happened, and now it's just memes and Yank politics so... maybe deploy Lemmy and turn off comments.

load more comments (1 replies)
[–] [email protected] 0 points 1 year ago

@ajsadauskas @degoogle i encountered https://aboutideasnow.com/ recently

it's not exactly what you describe and remember, but it's... at least real.

i was reminded by this post of my days from like 1998-2001ish on About.com (RIP)

[–] [email protected] 0 points 1 year ago (1 children)

@ajsadauskas @degoogle it sounds a bit like Kagi‘s Small Web initiative and search. have you seen it? https://blog.kagi.com/small-web

[–] [email protected] 0 points 1 year ago

I don’t know if this is the intent of their small web effort, but the first impression I got just now when I clicked through and saw this was, “gross”.

Scummy looking/feeling marketing hustle is a huge turn off.

[–] [email protected] 0 points 1 year ago (7 children)

I used them and contributed to links as well - it was quite a rush to see a contribution accepted because it felt like you were adding to the great summary of the Internet. At least until the size of the Internet made it impossible to create a user-submitted, centrally-approved index of the Net. And so that all went away.

What seemed like a better approach was social bookmarking, like del.icio.us, where everyone added, tagged and shared bookmarks. The tagging basically crowd-sourced the categorisation and meant you could browse, search and follow links by tags or by the users. It created a folksonomy (thanks for the reminder Wikipedia) and, crucially, provided context to Web content (I think we're still talking about the Semantic Web to some degree but perhaps AI is doing this better). Then after a long series of takeovers, it all went away. The spirit lives on in Pinterest and Flipboard to some degree but as this was all about links it was getting at the raw bones of the Internet.

I've been using Postmarks a single user social bookmarking tool but it isn't really the same as del.icio.us because part of what made it work was the easy discoverablity and sharing of other people's links. So what we need is, as I named my implementation of Postmarks, Relicious - pretty much del.icio.us but done Fediverse style so you sign up to instances with other people (possibly run on shared interests or region, so you could have a body modification instance or a German one, for example) and get bookmarking. If it works and people find it useful a FOSS Fediverse implementation would be very difficult to make go away.

[–] [email protected] 0 points 1 year ago (1 children)

@ajsadauskas @degoogle definitely something o be thinking about. More and more I’m using my followed hashtags, mastodon lists, and links to resources other people provide rather than just finding useful things in search results. But the big gap is still when I want to find quality info on a new topic. Cannot trust any of the damn results searching for how and how often to clean my kid’s new aquarium, for example. So much LLM and SEO crap info.

[–] [email protected] 0 points 1 year ago

Don't clean his new aquarium for like months.

If you can see this response (this half baked Mastodon integration annoys the shit out of me), I'd be happy to talk you through it.

[–] [email protected] 0 points 1 year ago

I had this exact thought earlier today. Either curated directories, or a ground-up, vetted search engine that only pulls from pre-screened sources.

[–] [email protected] 0 points 1 year ago (1 children)

Lycos, Excite, AltaVista, and of course Yahoo all were originally web directories of this sort.

Both Wikipedia and my own memory disagree with you about Lycos and AltaVista. I'm pretty sure they both started as search engines. Maybe they briefly dabbled in being "portals".

[–] [email protected] 0 points 1 year ago (1 children)

@bsammon And this Archive.org capture of Lycos.com from 1998 contradicts your memory: https://web.archive.org/web/19980109165410/http://lycos.com/

See those links under "WEB GUIDES: Pick a guide, then explore the Web!"?

See the links below that say Autos/Business/Money/Careers/News/Computers/People/Education /Shopping/Entertainment /Space/Sci-Fi/Fashion /Sports/Games/Government/Travel/Health/Kids

That's exactly what I'm referring to.

Here's the page where you submitted your website to Lycos: https://web.archive.org/web/19980131124504/http://lycos.com/addasite.html

As far as the early search engines went, some were more sophisticated than others, and they improved over time. Some simply crawled the webpages on the sites in the directory, others

But yes, Lycos definitely was definitely an example of the type of web directory I described.

[–] [email protected] 0 points 1 year ago

1998 isn't "originally" when Lycos started in 1994. That 1998 snapshot would be their "portal" era, I'd imagine.

And the page where you submitted your website to Lycos -- that's no different than what Google used to have. It just submitted your website to the spider. There's no indication in that snapshot that suggests that it would get your site added to a curated web-directory.

Those late 90's web-portal sites were a pale imitation of the web indices that Yahoo, and later DMoz/ODP were at their peak. I imagine that the Lycos portal, for example, was only managed/edited by a small handful of Lycos employees, and they were moving as fast as they could in the direction of charging websites for being listed in their portal/directory. The portal fad may have died out before they got many companies to pony up for listings.

I think in the Lycos and AltaVista cases, they were both search engines originally (mid 90s) and than jumped on the "portal" bandwagon in the late 90s with half-assed efforts that don't deserve to be held up as examples of something we might want to recreate.

Yahoo and DMoz/ODP are the only two instances I am aware of that had a significant (like, numbered in the thousands) number of websites listed, and a good level of depth.

[–] [email protected] 0 points 1 year ago (2 children)

@ajsadauskas @degoogle I guess the problem though is how you make sure they are actually maintained by a human acting in good faith. The way community Facebook groups meant to be for this kinda thing get spammed by likely fake businesses doesn’t give me hope

[–] [email protected] 0 points 1 year ago

I suppose any measures at all would cut out a massive number of spam pages already.

[–] [email protected] 0 points 1 year ago (1 children)

@joannaholman @degoogle Good point.

If it were run as a private company, I think the solution might be just to pay actual humans as employees.

If it's a community-run project, the challenge would be to come up with a robust moderation system...

load more comments (1 replies)
load more comments
view more: ‹ prev next ›