To keep our servers stable, and to prevent widespread abuse of the resources used by sites hosted on our shared servers, the default configuration prevents most malicious and abusive automated traffic from accessing your site, but we recognize that this kind of ‘one-size-fits-all’ approach doesn’t actually work for everyone, so we left you with a way to customize exactly which bot categories are allowed to access your website.
Bots are largely detected by the user-agent they identify themselves as when making requests to your website. This is how we have sorted them:
Security Scanners and Hostile Recon bots (Rule ID 77999910) — The most dangerous category of bots. We do not recommend unlocking this rule unless you’ve spoken to support first and can confirm you have a real need to allow this traffic in!
Forged/Swarm Browsers (Rule ID 77999920) — A lot of swarm activity forges these user-agents although it also coincides with some legitimate browser use in China (MicroMessenger, Kinza, etc) so if you have genuine traffic coming from China, this rule can be disabled.
Scrapers and Scrape-as-a-service Bots (Rule ID 77999930) — Only disable this rule if you are contracting one of these services to access your site.
AI Model Training Bots (Rule ID 77999940) — Only disable this rule if you specifically wish to allow AI to use your site content for training purposes.
AI Answer-Engine Indexers (Rule ID 77999950) — If you want to allow AI to index your site for reference so they can quickly answer with content from your site, disable this rule. This rule does not prevent agents from accessing your site to answer an immediate question.
SEO/Backlink Crawlers (Rule ID 77999970 — Only disable this rule if you specifically need an SEO or backlink-checking third party to access your site.
Commercial Data Mining/Brand Monitoring Bots (Rule ID 77999980) — Only disable this rule if you need services from this category to access your site, which is usually pretty rare.
Non-Google/Bing Search Engine Crawlers (Rule ID 77999990) — It’s usually pretty safe to disable this rule but it’s only going to be useful for websites that are primarily getting traffic from the less common search engines.
We also monitor one additional category (Rule ID 77999960) for active AI agents making requests on direct behalf of a user, but at this time we do not block these agents.
To allow specific categories through to your site, find and edit the .htaccess file within your site and add this to the top, one line for each category you want to allow:
<IfModule LiteSpeed>
SecRuleRemoveById 77999940
SecRuleRemoveById 77999950
</IfModule>
The ‘IfModule’ lines are generally optional on our servers but may save you headaches if you migrate your site to another host, as these configurations are only going to work on HostUpon shared hosting servers. In the above example, we enabled access for AI training bots and AI crawlers and indexers.
If you have a question about specific user agents or what category they go into, open a ticket to Support and we’ll be happy to help.