Search Tech Blog

How-To develop a real search engine from scratch

User-Agents of the Top 10 Web-Crawler


There are thousends of bots and web crawlers working the internet but below is my list of the 10 popular search engines user-agents. If you browse the logfiles of your website, you will always see the access to a file called “robots.txt”. These are usually calls from search engines. Their web crawlers with there user-agents that read the robots.txt file (hopefully you have one). They...

Build a search engine in just 3 days


Data is a fairly new resource, which gets more important every day. BigData is a common and important keyword for every company that develops and is trying to success over its competitors. The company that can use its data best may win this competition. For this you need software – to collect, to store, to provide and to analyse data. A search engine is a key tool to get the needed data...

Linklist you need if you want to build a search engine


Linklist Here are some background informations about how a search engine exactly work. We light ub what is difficult to crack if we try to build our own web crawler search engine from scratch: Giga Blast This page is a bit outdated (2004). But here you can read from the developer Matt Wells personally: All steps the search engine GigaBlast went through during the development process: After that...

Stop-Word List


What is a stop-word list and what advantage does it have to remove them? Stop words are extremely common words A Stopword is a word without essential information content, such as “and”, “the”, or “www”, etc. In English, the terms “stopword” or “stopwords” are used for this purpose. They are used very often, but do not really provide any...

Writing Your Own Search Engine is Hard


Why is it so hard? Anna Patterson, Software Engineer. As well as contributed to search engines and artificial intelligence at Google, and co-founded Cuil. Makes following quotation about developing search engines: “There must be 4,000 programmers typing away in their basements trying to build the next “world’s most scalable” search engine. It has been done only a few times. It has never...

Search Tech Blog

Latest posts

Latest comments


Tag Cloud