A few years ago anything you wanted to search for would generate hundreds of pages of results. From sources all over. If you structured your query well you could dramatically cut down the irrelevant results and then spend a bit of time skimming through the offerings until you found your target.
Searching felt like a hunt, it could take you to far flung corners of the web, and hitting the jackpot on some obscure forum was part of the fun.
Now searching anything feels incredibly frustrating. You know what you are looking for is out there somewhere but it feels impossible getting to it. Basically we have only two main routes now, Google and Bing (and I swear Bing is just copying Google’s homework). I’ve tried a handful of others and they all fall very short.
Something has changed dramatically under the hood.
It seems like the results are very shallow and no amount of prodding will get either search engine to produce anything besides the laziest output. Any subject has a limited pool of sites they lean on and then pretend that that’s all of the Internet.
It generalizes very quickly and if you try to counter by putting things in quotations it gives up.
Even in the rare moments I’ve buckled and used their stupid AI, it takes even more time browbeating the program to just deliver the damned specifics.
What is going on??
How do we get our old search back?
I swear every day of trying to navigate the web feels worse than the day before.
How did we get here?
There’s got to be something better, right? Or something we can do to pave a new structure that benefits the denizens of the web better than the lazy slop our tech overlords scrape off the table for us. What can we do?
SEO did this. It started before LLM and chatGPT. Blogger make money be generating trafic. So they wrote shits in a format that please search engine.
That’s why if you search “search into file Linux”, instead of just a grep command line, you end up on 2000 lines article with introduction, paragraph, explanation, blablabla. Because search engine boost this kind of content. Title, introduction paragraph that repeat the title, subsections, etc…
I’ve never seen any search engine officially confirm this, but search across all search engines became terrible just a few months after the release of ChatGPT.
Which is why my assumption is that LLMs are being used to generate heaps of credible-looking, but ultimately worthless webpages. And search engines have to try to filter those out or their results would be even worse.But then there’s no obvious distinction between a human-crafted blog and some spam webpage, which means the human-crafted blog is likely to be caught in the crossfire and get filtered as well.
Well, and on top of that, search engines presumably got hit by AI scrapers looking for training data, so that’s why I would assume, they now limit the search results to one page…
Yeah. Noticed this too. Pages that look legit but are filled with useless word salad that doesn’t go anywhere.
Gives you an impression that those pages didn’t exist before your search. How it got to top results is beyond me.
I use kagi and metager. They’ve solved it for the most part for me.
I do think that part of the problem is that so much up-to-date information is now trapped in discord, also that people are generally using forums and the like less to solve their technical problems because they’re privately querying robots. This on top of enshittification… We did not know how good we had it 20 years ago
That and everything is a YouTube video now. Or at least that’s what the search company that makes money off their video service wants you to think.
Because “Fuck you, pay me”
As others have mentioned, it was engineered to be worse for the capitalists to make profits from ads and the statistics to look better to shareholders. This in turn has made people to create a shit ton of garbage articles and blogs that say a ton while saying nothing, so even if you used a new search engine with the funding of Google but the algorithm of back then it would still be shit probably, because the volume of shit on the internet has now reached a level where finding anything in it is impossible.
The explosion of AI has come right at this moment where search engines have been lobotomized and gives people an option to be able to find things again, which is also another factor contributing to the profits of the AI companies, but even that is limited to what an LLM can average out from the totality of information that it has been fed and has the obvious danger of making stuff up and treating its biased training data as truth.
Generally, I think we’re fucked. I really don’t know if we’re ever going to get back the internet experience that once existed and what it’s going to take to get there, but it would probably involve a massive societal change to come before it.
I’d add that traditional search can’t reasonably hope to compete with AI in this environment. If you’re running a traditional search engine right now, there’s a real argument that the best business decision is to accelerate the enshitification to capture as much value as you can from your remaining audience while you still have it.
This is the usual cycle with these sorts of technologies - once a replacement arrives, the natural choice is to shift from growth to exploitation.
This is going to get worse before it gets better.
At this point I think the only way forward is for enough like-minded people to sort of stake their claim on some semi-walled off portion of the internet and greatly limit access from swaths of IP addresses.
Sort of like web rings 2.0. Everyone, or enough people, will have to chip in on costs and provide either money or hardware.
There will have to be some sort of regulatory body with voting members.
To be ‘in’ the site has to abide by certain advertising restrictions and privacy agreements.
A decent use (finally) for llms would be to audit the private index and remove slop. I think we’re just at the beginning of what will be a resource hogging AI arms race.
It’s sort of what we’ve been trying to avoid since the beginning but it seems inevitable at this point.
People with bad info put more effort into SEO than people with good info. Google changing focus from getting people good results to getting people to stay on their site. Powers that be filling the web with misinformation. There’s always been conspiracies and misinformation on the internet, but now there’s major money in pushing it.
It’s simple. Cause Search engines transformed to ad engines.
SEO. Everybody is blaming corporations but nobody today can develop a search engine that can’t be spammed. If they could, they would.
If only we (the collective users) spent more time creating a framework and vision of how the Internet should develop and aggressively pushed back on attempts to deviate from it…
When the people refuse to convene a governing body with teeth, the vacuum is filled with opportunists.
In a sense we are suffering from our own success. A place where anybody can build anything they want digitally. Left a ripe field for unrestrained commercial interests to infest the whole thing. Now it is bloated with human and AI slop to such a degree it smothers everything else and every major tech push is tuned towards data scraping every morsel.
True, but that reeks of victim blaming. The spammers are to blame here, not the mathematical inevitability of antispammers being behind on the arms race. Also, fucking AI.
No. It’s a challenge to all the brainiacs out there hacking high security servers with a freaking TV remote. I hope that they’ll read “Nobody today can develop a search engine that can’t be spammed,” and say to themselves, “I accept that challenge.”
Also, AI has been spamming forever. One of the tricks they did in the 2000s was to spam forums with links to websites so that Google would think those sites were more important than they actually were. In 2009, I hosted a forum that went nowhere, left it unattended for a month and came back to find bots had spammed it into oblivion with links. AI isn’t a new problem.
I think they intentionally made search engines worse to push people into using LLM’s
Which are amazing now, but not to worry, LLMs will be complete dog shit too once they pull the rug.
They already are dogshit.
If your LLM prompt isn’t about something common or mainstream is has an incorrect response most of the time. Unfortunately due to how they work, they will present the incorrect answers with the same confidence as correct answers.
LLMs seem like their only use is writing reports or resumes that no one needs to read. Everything else is better done by hand.
The goal of a search engine is to make the company money, not to aid you in finding what you are looking for.
We’re in between a rock and a hard place. The only way to shake the grip of enshitified companies is to either turn to government/tax backed public utility type incentives, which opens up a whole other can of worms, or lean on crowdsourcing, which is very hit or miss.
There was more money in showing you things than there was in helping you find things.
If you ever watched any of the saturday morning, over the air, wrestling “news” shows in the 80s and 90s, then that’s all everything is now. There used to be a pretty good article about it.
If you’re not paying for it, then you’re not the user, you’re the product.
Open source: am I a joke to you?
There are leaked emails from Google (from many years ago) where it was proposed to purposely degrade search quality to increase ad clicks. While some at Google responded with dismay as this, it is not a great leap to assume that this faction eventually won and they did in fact deliberately do that. Advertising is Google’s main revenue source, they are clearly motivated to do so.
Because data is never free. Google, duckduckgo, kagi, they all cost money one way or another whether that be a subscription service or by forcing websites to be shit so your only option is using their LLMs. Yes the reason why the internet has gone to shit is because of LLMs
I’ve went through hundreds of search results on duckduckgo, even when I make it intentionally broad and genetic that wikipedia would cover it, it still doesn’t come as the first search result. Search engines have gone to shit because LLMs make the most money in terms of data surveillance. Meaning (take for example google) they have a profile of you, they have queries to train, they have data that can be sold to advertisers. The only reason LLMs are being pushed so much isn’t because its intelligent, it’s because it makes the most money.
There is a “search engine” that hasn’t gone to shit, and that’s called marginalia search, It’s not exactly a search engine where you can type a url and it will bring you to a page, Instead it compliments your general search engine as a research aid and you can find some golden sites.
I’ve personally used it for data sheets on IC electronics and their implementations. Unfortunately I’ve found it insufferably difficult on duckduckgo (like multiple pages deep for a datasheet, something that should pop up on the first page), and I don’t trust the output of LLMs (large lying models).
So yea, hopefully this helps.
I’ll have to check that out. Thanks
Ironically I think I’ve gotten better results from AI-optimized search engines than from DuckDuckGo
