It’s quite sad that we are now at a point where we are forced to make our own search engines from scratch. Search engines are hard! Google’s original search algorithm (about 2 decades ago) was quite amazing. You were able to give vague search terms and yet still find the answer you wanted. The secret sauce was ranking based on relevance to the search query. I’m not aware of any guides/projects on search engines. I wish there was a good way I could search for this. (The irony!) But a great starting resource is this series on networks from wikipedia. (https://en.wikipedia.org/wiki/Network_theory)

Some random tips:

• The main goal of any search engine should be to minimize the number of times a user returns to the ranking page to click on a new link. Big tech should be doing this anyways but they have other goals.
• The main metadata database needs to topologically connect you to any part of the internet. (https://en.wikipedia.org/wiki/Graph_theory) Think of it as a hub/portal giving you general directions, but doesn’t tell you exactly where you should be heading. The ideal solution is to download everything from the internet and query each result for relevance to a search query individually, but this is intractable. Instead you have to group the internet into graphs and sub graphs - STEM, Social, Forums, E-commerce etc. Hyperlinks offer an objective way to calculate connections between websites. For example Lemmy.world <-> Wikipedia.org. The weight of these connections gives you a way to guide a traversal algorithm during search. Semantic analysis of some form allows you to find more efficient ways to draw connections making your search more efficient.
• The most powerful way to find connection/relevance to a search term is with transformers and their attention mechanism. For example if the search query is “Open source search engine”, the attention heatmap would be on groups of websites subjects like Forums, Q&A, Programming, Network Science, etc. There would also be a negative heatmap for topics like Cooking, Sports, Entertainment, etc. From there you want to load up recursively metadata for websites. For example for Lemmy it would be the title of all posts (and maybe their top comments). If it fits, load as much of this as you can into a transformer and calculate the heatmap relative to the search query. Again you are not using the transformer to generate answers. This is a bad idea. Instead you are using it to rank search results in terms of relevance/attention, what the transformer is fundamentally designed for.

As a side note, you are able to tune your model to your own search preferences with little data. You are also able to exchange computation time for search quality! This is amazing. If computation is a concern, traditional traversal algorithms and basic relevance/ranking algorithms work too but at the cost of more engineering.

I hope this sorta helps, if you have any other question feel free to ask! The future of search will likely be self-hosted as conflicts of interest within current search engine providers degrades the quality to the point where they are unusable.

Glad you found it awesome! :)

Oh? I didn’t know that. They seemed like a good organization to me too. Open source hardware is quite lacking compared to the software side so I hope they succeed.

Sorry didn’t mean to cause any trouble. I collect and share internet resources with others. If you want to verify this for yourself, my post history has questions similar to this one. I removed the image to make this post more ‘generic’. I am genuinely trying to share resources. My apologies if it came across as advertising.

Sorry didn’t mean to cause any trouble. I collect and share internet resources with others. If you want to verify this for yourself, my post history has questions similar to this one. I removed the image to make this post more ‘generic’. I am genuinely trying to share resources. My apologies if it came across as advertising.

No it’s not. This was one example of a powerful resource. Another is Libgen, and the List of Awesome

https://github.com/topics/awesome https://en.wikipedia.org/wiki/Library_Genesis

No it’s not. This was one example of a powerful resource. Another is Libgen, and the List of Awesome

https://github.com/topics/awesome https://en.wikipedia.org/wiki/Library_Genesis

Thanks for the further detail!

Please feel free to ask any questions! Math is a wonderful field full of beauty but unfortunately almost all education systems fail to show this and instead makes it seem like raw robotic calculations instead of creativity.

Math is best learned visually and with context to more abstract terms. 3Blue1Brown is the best resource in my opinion for this!

Here’s a mindblowing fact for you along with a video from 3Blue1Brown. Imagine you are sliding a 1,000,000 kg box and slamming it into a 1 kg box on an ice surface with no friction. The 1 kg box hits a wall and bounces back to hit the 1,000,000 kg box again.

The number of bounces that appear is the digits of Pi. Crazy right? Why would pi appear here? If you want to learn more here’s a video from the best math teacher in the world.

https://www.youtube.com/watch?v=HEfHFsfGXjs

Wow thank you!

Thank you mods for your work! I’m loving these communities!

Glad you are enjoying it! Please feel free to share any other channels like Computerphile, Numberphile, Mutual Information, & Veritasium. It’s difficult to find gems nowadays on YT.

Thanks for the comment! It is cool and also pretty aesthetically pleasing!

Thanks for sharing! (No worries, changed the title from “fact” to “thing” to be a bit more broad)

Thank you all for your answers!

I wanted to add one resource I found that has helped me find even more relevant search results:

A Lemmy Search Engine https://www.search-lemmy.com/

Thank you so much!

Half Life is awesome! If I had to choose from all the games I’ve played, the half life series has to be best in my opinion. There are so many small details and world building elements in each half life game that it really does make it feel like you are in the game. And then there is Half Life: Alyx…

It was so immersive, I smashed my head against the wall trying to run away from something lol

Not a channel, but some resources. The addons are all open source. Also unfortunately many subreddits that had quality channel suggestions have been either banned due to being “unmoderated” or are lost to time. Please post any finds here or help archive them if you can. The Datahorders community is a great place to start!

• DeArrow - Better Titles and Thumbnails on YouTube
• Sponsorblock
• Return YouTube Dislikes
• uBlock Origin

List of resources to find more gems. Note many posts and subreddits have been lost/banned/removed. Here are the ones I found.

• https://old.reddit.com/r/mealtimevideos/
• https://old.reddit.com/r/WoahTube/
• https://old.reddit.com/r/TheWayWeWereOnVideo/
• https://old.reddit.com/r/DeepIntoYouTube/
• https://old.reddit.com/r/Documentaries/
• https://old.reddit.com/r/ObscureMedia/

deleted by creator

Awesome! Thanks for this post by the way! It has been my favorite post to go through since I don’t have a google account and can’t use their recommendation system. Plus, the recommendation system definitely doesn’t give us as high quality channels as some from this thread.

There are still hundreds if not thousands of hidden gems on YouTube. I’ll come back to this post and share some as I rediscover them.

Here is another one before I go. Astartes. Sad they stopped but glad they found employment at least for their skills.