Search Tech Blog


I would like to introduce myself, my nickname is vanGato, I was born in Germany and have many years of experience in the IT- and new-media industry. In this blog I write about how search engines work, facts, ideas, code experiments, and the possibility to develop a simple search engine from scratch that can handle a few million entries at an acceptable speed.

Web Crawler System Design


Distributed Web Crawler System Design to crawl Billions of web pages Learn web crawler system design and software architecture to Design a distributed web crawler that will crawl all the pages on the internet. Let’s learn how to build a google spider bot or google distributed web crawler. Crawler System Design​ Spider Systemdesigntips​ Bot Systemdesign​ Search Engine Computerscience​ Learn...



Dogfooding a Quick Guide to Internal Testing Eating your own dog food, also called dogfooding, occurs when an organization uses its own product. 1970 Alpo’s Dog Food Advertising Alpo, with the help of Lorne Greene, convinced consumers to buy their products because they themselves used them. 1980 Memo from the Apple CEO „We believe the typewriter is obsolete. Let’s prove it inside before we try...

PHP dynamic Favicon Script


What is a dynamic Counter Favicon you will ask? No, you already know! We all have already seen the small icon in the Browser Tab called Favicon. But if you want to read more about this tiny little thing, take a look at the Favicon-Cheat-Sheet on GitHub Page from Audrey Roy Greenfeld. But today i want to direct your attention to an other topic. Dynamic content in such an favicon. This can be used...

User-Agents of the Top 10 Web-Crawler


There are thousends of bots and web crawlers working the internet but below is my list of the 10 popular search engines user-agents. If you browse the logfiles of your website, you will always see the access to a file called “robots.txt”. These are usually calls from search engines. Their web crawlers with there user-agents that read the robots.txt file (hopefully you have one). They...

Linklist you need if you want to build a search engine


Linklist Here are some background informations about how a search engine exactly work. We light ub what is difficult to crack if we try to build our own web crawler search engine from scratch: Giga Blast This page is a bit outdated (2004). But here you can read from the developer Matt Wells personally: All steps the search engine GigaBlast went through during the development process: After that...

Stop-Word List


What is a stop-word list and what advantage does it have to remove them? Stop words are extremely common words A Stopword is a word without essential information content, such as “and”, “the”, or “www”, etc. In English, the terms “stopword” or “stopwords” are used for this purpose. They are used very often, but do not really provide any...

Writing Your Own Search Engine is Hard


Why is it so hard? Anna Patterson, Software Engineer. As well as contributed to search engines and artificial intelligence at Google, and co-founded Cuil. Makes following quotation about developing search engines: “There must be 4,000 programmers typing away in their basements trying to build the next “world’s most scalable” search engine. It has been done only a few times. It has never...

Search Tech Blog

Latest posts

Latest comments


Tag Cloud