Findxbot – a real search engine bot

We received many questions in our last AMA on reddit and from webmasters about Findx, about how we build the Findx search engine index, and how our crawler, Findxbot, works. We hope this post can answer some of those questions!

Read more

robots.txt – subtle challenges

robots.txt controls which content web crawlers may reach, but the lack of proper specifications has lead to some interesting corner-cases.

Read more