About LinkoriaBot

How to identify it

When we crawl the pages of a site — its robots.txt, its sitemaps and the pages themselves — the requests carry this user agent:

LinkoriaBot/1.0 (+https://linkoria.io/crawler)

The URL in the user agent points to this page, so anyone reading your logs can find it. Not every request we make carries it — see “What it does” below.

What it does

• Reads robots.txt before anything else, and honours it.
• Reads sitemaps to find which pages to check.
• Requests those pages and extracts the links on them.
• Follows each outbound link through its redirects to find where it ends up. These are single requests to the link's destination, and they are sent with an ordinary browser user agent rather than as LinkoriaBot, because many affiliate networks and shops refuse anything that announces itself as a bot.

Why it is on your site

Someone using Linkoria asked us to check the links on a site. Either that site is yours, or a link on another site points to yours — in which case we requested a single URL to see whether it responds, and nothing more. A request like that carries a browser user agent, so it does not show up as LinkoriaBot in your logs.

How it behaves

• Requests are rate limited and spread out over time.
• It obeys robots.txt, including crawl-delay.
• It does not submit forms, log in, or execute JavaScript.
• It does not collect personal data from your pages.

How to block it

Add this to your robots.txt and we will stop crawling your pages on the next visit:

User-agent: LinkoriaBot
Disallow: /

That covers crawling. It does not cover the single request we make to a URL on your domain when a link on someone else's site points to it — those look like ordinary browser traffic, and no robots.txt rule can single them out. To have your domain excluded from those as well, or to have crawling stopped immediately, email us.

Contact

For anything about the crawler, including takedown or exclusion requests: abuse@linkoria.io