Reddit has apparently blocked numerous search engines to prevent them from displaying the platform’s content. Google seems to be the only major exception. The reason for this is probably that the two companies have concluded a license deal worth millions.
Reddit began blocking numerous search engines and their web crawlers in early July 2024. That comes from one report of the online magazine 404 Media out. Accordingly, Google is currently one of the few search engines that displays current Reddit content.
Reddit blocks search engines because they don’t pay?
The background: Google concluded a deal with Reddit at the beginning of 2024. For around 60 million US dollars, the company secured exclusive access to the content of the Internet forum in order to be able to train its in-house AI systems.
But now Google seems to be the only search engine that shows current Reddit results in its search. For example, if you enter “site:reddit.com” into Bing or DuckDuckGo and select “last week” as the date in the search settings, you will not receive any search results.
A Reddit spokesperson denied this to the tech magazine The Verge Officially there was a connection with the Google deal, but confirmed it indirectly:
This has nothing to do with our recent partnership with Google. We have been in discussions with several search engines. We have not been able to enter into agreements with all of them because some are unable or unwilling to make enforceable commitments regarding the use of Reddit content, including its use for AI.
Microsoft confirms: Reddit blocks search engines
In concrete terms, this means: Reddit apparently not only demands money so that other companies can train their AI systems with the content of the Internet forum, but also apparently wants to be paid for basic access – for example via search engines.
In order to prevent Bing, DuckDuckGo and Co. from accessing Reddit, the social news aggregator updated its robots.txt file at the beginning of July 2024 to prevent search engines from crawling. Ben Lee, Reddit’s chief legal officer, said The Verge in a previous report “It sends a signal to those who do not have an agreement with us that they should not access Reddit data.”
Microsoft spokeswoman Caitlin Roulston said: “Microsoft respects the robots.txt standard and we respect the instructions of websites that do not want content on their pages to be used with our generative AI models. She also confirmed that Bing stopped crawling Reddit after the platform updated its robots.txt file.
Reddit’s decision to block some of the most popular search engines seems drastic, but it doesn’t really come as a surprise. The platform had already started protecting its data more and more strictly last year in order to generate additional sources of income and attract investors.
Is the time of the open Internet over?
Reddit reportedly even initially threatened to deny Google access to its content if the company didn’t stop using the data to train its AI system for free. As more and more AI content floods the Internet, human content like Reddit is likely to become more important in the future.
Many users add the name “Reddit” to their search queries in order to receive targeted answers that come from people. But in the future this only seems to be possible on Google. Reddit itself has thereby achieved an extremely powerful position.
However, corresponding licensing deals like those with Google also mean that the Internet becomes more closed when platforms like Reddit close themselves to other search engines. Since forums are generally becoming more important, this trend could continue and also extend to the social media area.
A few years ago such a development would not have been conceivable. Search engines have primarily brought website operators traffic and thus indirectly advertising revenue. However, due to AI and spam content, search engines and the internet have been getting worse for quite some time now.
The crucial question is therefore not only how the search engines will deal with Reddit’s advance, but also whether it will find imitators – not to mention how to deal with AI spam, which even Google cannot get under control.
Also interesting:
- AI spam will ruin the internet – this is how scammers are tricking Google
- Study reveals: This is how you can distinguish deepfakes from real recordings
- Google has lost control of its search
- Google still recommends glue as a pizza ingredient
The article Reddit blocks search engines: Is the time of the open internet over? by Fabian Peters first appeared on BASIC thinking. Follow us too Facebook, Twitter and Instagram.
As a Tech Industry expert, I believe that Reddit blocking search engines is a concerning development that goes against the principles of an open and accessible internet. While Reddit may have valid reasons for wanting to limit exposure to certain content or protect user privacy, blocking search engines ultimately restricts the flow of information and limits the ability for users to discover new and diverse perspectives.
In an age where the internet plays a crucial role in connecting people and facilitating the exchange of ideas, it is important to prioritize openness and transparency. By blocking search engines, Reddit is essentially creating barriers that prevent users from fully engaging with the platform and accessing valuable content.
While it is understandable that platforms may want to control their own content and user experience, it is essential to find a balance between protecting privacy and promoting accessibility. The decision to block search engines raises questions about the future of the open internet and highlights the need for a more inclusive and collaborative approach to online information sharing. It is important for tech companies to consider the broader implications of their actions and strive to maintain the principles of a free and open internet.
Credits