Digital Diva: Wanna kick some bots? They're not all bad, but they can be annoying
Chances are there's something crawling all over your website and it may — or may not — be harmless.
The bug equivalent on the Internet is called a bot. And bots are constantly at work. Some are good. Some are nuisances. Some are just downright bad.
The good bots are the types that Google and other search engines send out to crawl websites and index the Web. Without those bots, Google's search-engine results wouldn't be useful and we'd all stop using it for everything.
Another Web crawler that's garnering attention from the security industry is called Semalt, but it's for all the wrong reasons. There's a good chance that if you check your website's Google Analytics, you'll find referral traffic from Semalt.com.
Creepy crawlies?
Semalt, which purports to be based in Ukraine, came on the scene in January and has been crawling websites pretty relentlessly, causing inflated traffic numbers. If your website traffic has been up over the last few months, you might want to make sure it's all real visitors and not Semalt or some other bot racking up referral traffic.
Matt Ward, CEO of Gardner-based inConcert Web Solutions, said Semalt “is a huge issue right now,” and that he's seeing the spam referral traffic from Semalt on his company's website, as well as a lot of his clients' sites.
If you go to Semalt.com, you'll find a simple landing page that prompts you to sign up for search engine optimization (SEO) services. There's little additional information and no links for more information. If you Google Semalt, you can find an “About Us” page that provides a few details, including the fact that the company is based in Ukraine.
The lack of information and the international headquarters left me very suspicious, as did a number of blog posts decrying Semalt as a referral spammer that must be up to no good.
I decided to go right to the source and emailed Semalt. Much to my surprise, I got a quick response and within a few days was on a Skype call with a woman by the name of Julia Vashneva, who said she was press secretary for Semalt.
Vashneva said Semalt is harmlessly crawling websites, just like Google and others do, to gather data for the company's SEO services. The lack of detailed information on the company's website is simply an oversight, and one that Semalt plans to correct soon, she said.
While I was on the call with her, Vashneva couldn't tell me what Semalt was doing with all the data it's gathering from its voracious site crawling.
“I'm not an engineer. I can't tell you for sure what they're using this data for. It's just for improvement of our service,” she said. Later, she followed up with a Skype message to say the company is launching a new SEO service that will give Semalt customers access to information on what competitors are doing on their websites.
Quite frankly, Vashneva had all the right answers for me on the phone and even provided pricing information for Semalt's services. I can't say that they're doing anything wrong by crawling your site, but there are steps you can take if you simply want them to leave you alone.
Semalt does have a webpage that allows you to request to be removed from its crawl list. But if you'd rather not interact with them at all, you can do one of two things.
The first is to filter out Semalt activity from your Google Analytics, so your Web traffic numbers won't be inflated by the referral traffic. The second is to use what's called your htaccess file to block any crawling from Semalt. With both of these options, you'll probably want to contact your Web support staff or vendor and ask it to take those steps.
Others may have devious intentions
Semalt isn't the only Web crawler out there, and it's certainly not the last. And some Web crawlers really are up to no good.
For example, there are bots that crawl the Web and hack into websites and use those hacked sites to create what's called a “bot net,” according to Igal Zeifman of the Web security company Incapsula.
And don't think that just because your website is small and you don't do ecommerce, you aren't a target for bad bots.
“Contrary to popular belief, if you are small you are not immune,” Zeifman said. “This is exactly opposite the case.” By cobbling together a bot net that harnesses the server power of a number of smaller sites, a hacker can then go after a big site that may have valuable information, such as credit card data.
Even if a bot isn't aimed at hacking into your site, it may be after marketing intelligence that it plans to sell to your competition. Such efforts are kind of like how retailers might send an employee to a competitor's store to check pricing on the store shelves, but on a global scale.
“Bots that are after marketing intelligence are not necessarily malicious by nature, but they are unwanted,” he said.
Christina Davis can be reached at cdavis@wbjournal.com.
WBJ Web Partners
Most Recent
Most Recent
0 Comments