Search Tech Blog

AuthorI. Gaffling

I would like to introduce myself, my name is Igor Gaffling, I was born in 1968 and have more than 30 years of experience in the IT- and new-media industry. In this blog I write about how search engines work, facts, ideas, code experiments and the possibility to develop a simple search engine from scratch that can handle a few million entries at an acceptable speed.

User-Agents of the Top 10 Web-Crawler

U

There are thousends of bots and web crawlers working the internet but below is my list of the 10 popular search engines user-agents. If you browse the logfiles of your website, you will always see the access to a file called “robots.txt”. These are usually calls from search engines. Their web crawlers with there user-agents that read the robots.txt file (hopefully you have one). They...

Linklist you need if you want to build a search engine

L

Linklist Here are some background informations about how a search engine exactly work. We light ub what is difficult to crack if we try to build our own web crawler search engine from scratch: Giga Blast This page is a bit outdated (2004). But here you can read from the developer Matt Wells personally: All steps the search engine GigaBlast went through during the development process: After that...

Stop-Word List

S

What is a stop-word list and what advantage does it have to remove them? Stop words are extremely common words A Stopword is a word without essential information content, such as “and”, “the”, or “www”, etc. In English, the terms “stopword” or “stopwords” are used for this purpose. They are used very often, but do not really provide any...

Writing Your Own Search Engine is Hard

W

Why is it so hard? Anna Patterson, Software Engineer. As well as contributed to search engines and artificial intelligence at Google, and co-founded Cuil. Makes following quotation about developing search engines: “There must be 4,000 programmers typing away in their basements trying to build the next “world’s most scalable” search engine. It has been done only a few times. It has never...

Build your own Search Engine

B

The Collection In this blog we want to collect all information that are needed to develop a real search engine from scratch! So we need your help, if you want to contribute you are welcome. You can get co-author, comment the posts, write an article, send some PHP code that solve a special problem. You will get all the honor and a backlink if you like and all the source code to have your own...

Search Tech Blog

Latest posts

Latest comments

Categories

Tag Cloud