Maintain your old Siemens Hipath system

60 Day Sandbox for Google & AskJeeves; MSN Indexes Quickest, Yahoo Next

Search engine listing delays have come to be called the Google Sandbox effect are actually true in practice at each of four top tier search engines in one form or another. MSN, it seems has the shortest indexing delay at 30 days. This article is the second in a series following the spiders through a brand new web site beginning on May 11, 2005 when the site was first made live on that day under a newly purchased domain name.

First Case Study Article

Previously we looked at the first 35 days and detailed the crawling behavior of Googlebot, Teoma, MSNbot and Slurp as they traversed the pages of this new site. We discovered the each robot spider displays distinctly different behavior in crawling frequency and similarly differing indexing patterns.

For reference, there are about 15 to 20 new pages added to the site daily, which are each linked from the home page for a day. Site structure is non-traditional with no categories and a linking structure tied to author pages listing their articles as well as a "related articles" index varied by linking to relevant pages containing similar content.

So let's review where we are with each spider crawling and look at pages crawled and compare pages indexed by engine.

The AskJeeves spider, Teoma has crawled most of the pages on the site, yet indexes no pages 60 days later at this writing. This is clearly a site aging delay that's modeled on Google's Sandbox behavior. Although the Teoma spider from Ask.com has crawled more pages on this site than any other engine over a 60 day period and appears to be tired of crawling as they've not returned since July 13 - their first break in 60 days.

In the first two days, Googlebot gobbled up 250 pages and didn't return until 60 days later, but has not indexed even a single page in 60 days since they made that initial crawl. But Googlebot is showing a renewed interest in crawling the site since this crawling case study article was published on several high traffic sites. Now Googlebot is looking at a few pages each day. So far no more than about 20 pages at a decidedly lackluster pace, a true "Crawl" that will keep it occupied for years if continued that slowly.

MSNbot crawled timidly for the first 45 days, looking over 30 to 50 pages daily, but not until they found a robots.txt file, which we'd neglected to post to the site for a week and then bobbled the ball as we changed site structure, then failed to implement robots.txt in new subdomains until day 25 - and THEN MSNbot didn't return until day 30. If little else were discovered about initial crawls and indexing, we have seen that MSNbot relies heavily on that robots.txt file and proper implementation of that file will speed crawling.

MSNbot is now crawling with enthusiasm at anywhere between 200 to 800 pages daily. As a matter of fact, we had to use a "crawl-delay" command in the robots.txt file after MSNbot began hitting 6 pages per second last week. The MSN index now shows 4905 pages 60 days into this experiment. Cached pages change weekly. MSNbot has apparently found that it likes how we changed the page structure to include a new feature which links to questions from several other article pages.

Slurp gets strangely inactive then alternately hyperactive for periods of time. The Yahoo crawler will look at 40 pages one day and then 4000 the next, then simply look at the home page for a few days and then jump back in for 3000 pages the next day and back to only reviewing robots.txt for two days. Consistency is not a curse suffered by Slurp. Yahoo now shows 6 pages in their index, one an errors page and another is a "index/of" page as we have not posted a home page to several subdomains. But Slurp has crawled easily 15,000 pages to date.

Lessons learned in the first 60 days on a new site follow:

1) Google crawls 250 pages on first discovery of links to site. Then they don't return until they find more links and crawl slowly. Google has failed to index new domain for 60 days.

2) Yahoo looks for errors pages and once they find bad links will crawl them ceaselessly until you tell them to stop it. Then won't crawl at all for weeks until crawling heavily one day and lightly the next in random fashion.

3) MSNbot requires robots.txt files and once they decide they like your site, may crawl too fast, requiring "crawl-delay" instructions in that robots.txt file. Implement immediately.

4) Bad bots can strain resources and hit too many pages too quickly until you tell them to stay out. We banned 3 bots outright after they slammed our servers for a day or two. Noted "aipbot" crawled first then "BecomeBot" came along and then "Pbot" from Picsearch.com crawled heavily looking for image files we don't have. Bad bots, stay out. Best to implement robots.txt exclusions for all but top engines if their crawlers strain your server resources. We considered excluding the Chinese search engine named Baidu.com when they began crawling heavily early on. We don't expect much traffic from China, but why exclude one billion people? Especially since Google is rumored to be considering a possible purchase of Baidu.com as entry to Chinese market.

The bottom line is that we've discovered all engines seem to delay indexing of new domain names for at least thirty days. Google so far has delayed indexing THIS new domain for 60 days since first crawling it. AskJeeves has crawled thousands of pages, while indexing none of them. MSN indexes faster than all engines but requires robots.txt file. Yahoo's Slurp crawls on again off again for 60 days, but indexes only six of total 15,000 or more pages crawled to date.

We seem to have settled that there is a clear indexing delay, but whether this site specifically is "Sandboxed" and whether delays apply universally is less clear. Many webmasters claim that they have been indexed fully within 30 days of first posting a new domain. We'd love to see others track spiders through new sites following launch to document their results publicly so that indexing and crawling behavior are proven.

© Copyright July 18, 2005 Mike Banks Valentine

Mike Banks Valentine is a search engine optimization specialist who operates WebSite101 eCommerce Tutorial and will continue reports of case study chronicling search indexing of Publish101 Article Resource

Click to Contact Mike Valentine

In The News:

North Korea 'agrees to close missile test site'
Wed, 19 Sep 2018 03:55:00 +0100
North Korea dictator Kim Jong Un has vowed to permanently dismantle a missile test site and a launch pad after talks with South Korea.

Merkel removes spy chief to quell racism row
Tue, 18 Sep 2018 21:16:00 +0100
Germany's leader Angela Merkel has removed her spy chief to end a row over immigration which threatened to destroy her fragile coalition.

Brett Kavanaugh accuser 'faces death threats'
Mon, 17 Sep 2018 18:04:00 +0100
A woman who accused US Supreme Court nominee Brett Kavanaugh of sexual assault says she has faced death threats and has called on the FBI to investigate her allegations.

Russia blames Israel after Syria shoots down plane
Tue, 18 Sep 2018 06:27:00 +0100
Russia has blamed Israel and threatened retaliation after one of its military planes was shot down by Syrian missiles in a "friendly-fire" incident.

China retaliates in Trump trade war escalation
Mon, 17 Sep 2018 23:41:00 +0100
China has announced a series of retaliatory tariffs on $60bn (£45.6bn) of US imports after Donald Trump raised the stakes in his trade war with Beijing.



tikatoshop.it

Erfahrungen mit Pallhuber Wein
Agen Bola SBOBET Terpercaya

Travel in comfort and at your leisure with CT Airlink Limousine & Car Service for top quality private transportation and exceptional customer service. We operate Sedans, SUVs & Vans for CT Car Services to covering all Connecticut airports including Car Service from CT to Newark Airport , Mohegan Casino Uncasville CT, Foxwoods Casino Mashantucket CT, Manhattan Cruise Terminal NYC, Brooklyn Cruise Terminal NYC and Bayonne Cruise Terminal NJ. CT Airlink hire licensed and friendly chauffeurs who have in-depth knowledge of the Areas.

How To Weather Google Algorithm Changes

If you spend much time reading the various webmaster forums,... Read More

Six Reasons Why Your Alexa Rating Is Still Important

1. Additional Exposure For Your Site.At the end of 2003,... Read More

SEO Expert Guide - Ongoing Monitoring of Results (part 9/10)

In the Guide, you have, so far, learnt how to... Read More

Martial Arts Webmasters: Time to Optimize Your Site!

A few months ago I was looking through the search... Read More

How to Verify and Monitor Your Search Engine Listing on Google?

Being listed in search engines and ranked high on searches... Read More

SEO Expert Guide - Black Hat SEO - Activities to avoid (part 8/10)

In parts 1 - 7, you learnt how to develop... Read More

Expert Help From Google Answers

Web users turn to search engines for answers to their... Read More

Banned By Google And Back Again

The date: 29th July 2005. The time: early morning. I... Read More

How to Avoid the Google Duplicate Content Filter?

More and more webmasters are building websites with publicly available... Read More

How To Syndicate Your Content

Note: In order to preserve the proper format for the... Read More

KEI Concerns and CID Alternative

Like many folks, I have been using KEI for some... Read More

7 Search Engine Optimization Strategies

Search engine optimization refers to the technique of making your... Read More

Submit Your WebSite: Add Your URL to Web Directories & Search Engines

Why get into Search engines and directories?Increase backlinks of your... Read More

How to Protect Your Search Engine Placement by Keeping Up-to-date on Industry Changes

There's no denying it: Search engines are a dominating force... Read More

How Google Indexes Content From Your Web Directory

In a fluke, I was able to notice something about... Read More

META Tags Explained and How To Use Them For Ranking

The META tags are used to provide extra information about... Read More

Top Ten Listing? Hmmmmmmmm

This is my question...I spend a fortune on SEO and... Read More

Being dumped by Google? Learn how to avoid becoming a victim next time around!

After Google latest update nicknamed "Florida", many webmasters discovered that... Read More

How Do I Improve My Web Site Conversion Rate? Part 3

Question 1How do keywords effect your conversion rate in terms... Read More

The Google Strategy

Webmasters across the Internet were totally floored by what happened... Read More

High Search Engine Rankings - A Long Term Strategy

The last 1.5 years have shown major changes in search... Read More

Internet 2005: The Ten Billions Pages Search Engines

To see each and every page during 10 seconds would... Read More

History of World / Regional Search Engines and Directories

Computers have become a way of life for people around... Read More

SEO Expert Guide - Conclusions (part 10/10)

As you have seen throughout the guide, search engine optimization... Read More

Website Promotion: 10 Search Engine Optimization Blunders to Avoid

If you want to develop a successful search engine optimization... Read More