How Quickly Can A Page Get De-Indexed and Re-Indexed in Google?

I thought I’d try a quick experiment out to see how quickly Google would de-index my most popular article, and re-index it. But first, some assumptions and caveats so you know what we’re working with here:

  • My blog is build on WordPress and has an RSS feed
  • The page I am about to remove has a PR of 1 and ranks in 3-4 position
  • I currently have 20 active plugin, one of which is the Yoast WordPress SEO plugin installed
  • I have an XML sitemap file that automatically updates when a URL is published/removed when it goes back into draft

For this test, I used my Google Calendar and iPhone Cal syncing issue page. It’s my most visited page in search, as you will see below, with 20+ comments, not very many shares. This page received 2,914 visits as a landing page and the average user spend 1 minute on that page in January 2012.

People's search behaviours are different, but they look for the same thing.

My Approach:

  1. Put the article back into Draft mode
  2. Have it removed from my XML sitemap file and off the HTML sitemap page (automatically done through plugins)
  3. See my traffic drop and search for my top referring keyword which was “google calendar not syncing with iphone” to see if my page had disappeared off the grid, if not wait until it did
  4. Once the page is gone, wait and republish
  5. Diligently watch the search results for my page
  6. Write this blog to tell you what happened :)
TIP: I used the Annotations feature in Google Analytics to track all changes on my website. It is great when it comes time to review your traffic and look at what activities you’d done on your site (i.e. public an article, campaign, technical issue fixes, etc.)

The Result:

Google Analytics Traffic when page was de-indexed

Traffic for when I put the page to rest, and when I awakened it from the dead.

  • I put my article into Draft on February 20, 2012 – I took about 24 hours before my page was gone from Google’s index. By February 22, my analytics tool reported 1 visit to my site.
  • 2.5 weeks later on March 9, I republished the article and see how quickly Google would pick it up. Keep in mind that the WordPress SEO tool automatically adds the URL to my XML sitemap, and also writes it to you HTML sitemap. They both likely helped the search engines discover it quicker.
  • On March 15, I started to gain traffic for those keywords, except a large number of them I noticed filtered into (not provided)… dang! So I cross checked my landing page report and what do you know, it was the page! It took Google about 5-6 days to discover and bring my page to the top of the search results for  ”google calendar not syncing with iphone” again.

You know what the funny thing is? This long tail search term is a “breakout” keyword in Google Insights for Search – see here. Not only that, the Google Keyword Tool (GKT) reports that there are 260 monthly searches for that phrase (exact match) globally and 210 searched in the US. Looks like a lot of people are having a tough time syncing their calendars across devices. Apple, Google… hope you are reading this…

What does this tell us?

This tell us that,

  1. If you’re allowing Google bots to easily access your content, it will crawl it and index your page quicker. If you don’t have essentials like a sitemap, I encourage you to look into creating one and telling Google where it is.
  2. Google can remove your site in a matter of hours. Be sure you are not allowing just anyone into your backend CMS. It can turn ugly if you’re a large organization that relies on a page to be alive 24/7.
  3. It takes a bit of time for bots to crawl, index and rank your page, but it will happen. Assuming you haven’t made any major changes to your site, your page should show up in the search results in the matter of days/weeks if you send a ping, update your sitemap files, and pray (ignore that last tip).
  4. Simply, publish valuable and helpful content on a blog that has an RSS. These pages typically get discovered and indexed quicker.

Have you done a similar experiment or have any thoughts about my approach? Please share your comments below!

email

Jackson Lo

Jackson is the Analytics / SEO Manger at adjump media and MENU.CA. His expertise spans from digital marketing (search engine optimization, local search marketing) to digital intelligence (digital analytics, data analysis). Very passionate about what he does, he also has big interests in photography and travelling.

More Posts

Follow Me:
TwitterLinkedInGoogle Plus

Posted on March 23, 2012 by . This entry was posted in SEO. Bookmark the permalink.
  • http://www.eyeoncloud.com Viquar Arfat

    Thanks Jackson, This is what I have been looking for. I have recently restructured my website but didn’t know how long will it take for google to de-index and re-index

  • http://www.barrowvale.ie/ Oliver

    Hi Jackson, that’s an excellent experiment, I like to see real numbers rather than people hypothesing about what may or may not be happening.  I am carrying out some experiments on  what effects the speed at which my blog is indexed. I have deliberately not put a link to my blog on my website yet and the only thing that I have done so far is set up notification of pingomatic.com and technorati.com.  You can see the results of what I have done so far by going to my domain above and then add /blog

    Feel free to contact me if you are carrying out any other experiments or if you are interested in any collaboration etc.

    Regards
    Oliver Dempsey

  • http://jacksonlo.com Jackson Lo

    Hi Oliver,

    Thank you for your comment. I think the speed at which your pages/posts can get indexed depend largely against one thing; accessibility. Because crawlers discover new pages through links on a page, if that link is not present, there’s a change that Google bots are not finding it. Thus, if you don’t have a link /blog on your homepage, likely it will take longer. 

    Blogs that are built on WordPress actually reap the benefit of their auto-ping services, so that might help with the quickness of seeing posts in search engines than a regular page. 

    Social media shares will likely give the post a lift too, did not test that out in this case.

    I also did not test the actual speed, up to the minute, but that would be an interesting study. Of course, we would need to consider what CMS (whether it be WordPress or any other), and compare those. Let me know if you come up with any interesting findings with this :)

  • http://jacksonlo.com Jackson Lo

    You’re welcome Viquar! :)

  • UK web hosting

    Really good and informative discussion. 

  • http://www.barrowvale.ie/ Oliver

    Hi Jackson,
    thanks for your reply. The main thing I found so far is that it is now taking 24 hours to be indexed and I have it pinging pingomatic.com and technorati.com. 

    I have another site where my blogs are consistently indexed within 1minute and one theory I have for this is that because information is posted by users every 30 minutes on average and maybe Google crawls it more regularly.  Having said that I checked a minute ago and on the other part of the site there is a customer posting that was done 30 minutes ago and it hasn’t shown up yet on Google’s index and the last one before that was one and half hours ago and it is showing up. 

    Do you know what the protocol is for pinging the update services when other information is posted on your site like users requesting information on a public platform like this or a discussion forum?  For example if you had a discussion forum with 50 threads posted a day could you ping the update services with each of those?

  • http://www.fluxedigitalmarketing.com/ Teyona

     This is what a beginner like me in SEO is always asking. Google is the biggest factor in SEO so it is important to know when it de-indexed or re-indexed. Thanks for the share.

  • http://jacksonlo.com Jackson Lo

    Glad you found this resourceful!

  • http://jacksonlo.com Jackson Lo

    Typically, if you are consistently publishing content on your site, crawlers will come more frequently to your site to look for new pages to crawl. It’s been reported that sites on CNN, for example, get crawled every few minutes. So if you are a large entity site, expect the crawl rate to be much greater than a site that updates their content once a month. 

    Also, if you are publishing content in a deeper directory, if could take longer for crawlers to find that page, thus it will delay your indexing rate. My recommendation would be to link to deeper pages from the homepage to get it discovered quicker. 

    I imagine Forums get crawled quite frequently, depending on how accessible that content is. It largely depends on how the forum is configured, whether they blog certain pages to members only, then those don’t get crawled or indexed. For a site like Warrior Forums, their pages/discussions are indexed and I often see those in the results from time to time. 

    Other thing to check and ask are:

    - Do I have a custom crawl rate configured in Google Webmaster Tools?
    - Do I have any pages blocked in my robots.txt?
    - Have I set properly in my sitemap file? 
    - Am I sharing that content on Twitter, Google+ for faster discovery?
    - Does my CMS automatically ping the search engines when there is new content published (WordPress does this)

    Those are just some ideas for you, but the check is endless. Have fun :)

  • http://jacksonlo.com Jackson Lo

    No problem Viguar. I’m glad you found this article helpful.

  • Aloeskinbeauty

    As someone who recently dealt with my site being de-indexed I know the frustration this has on websites. I still haven’t learned why my site was removed for some of the keywords I was trying to rank well for I’ll still continue to earn it back.

    Thanks for the tips! I’m off to try them :)

  • Jason Kelley

    You maybe able to help me with a possible issue. I have a site with a lot of static pages which I have added to sitemap manually. The only problem is when I add a post or page to the site it updates the “modified date” of all the static pages too. This means google thinks 100s of pages have been updated when they haven’t. Can’t be good imo. Especially if I am adding new content every day.

    My question is. If I remove the static pages from the sitemap but not the site will Google react and de-index those pages even though they are still on site and functioning.

  • http://jacksonlo.com Jackson Lo

    Typically, no. If the page is still live and it has been indexed, it will stay in Google’s index. 

  • Jason Kelley

    Thanks.

    Also, I think that your de-index re-index test would need to be longer to get a better idea. I would imagine that google would still be trying to crawl an old indexed page even after 2.5 weeks (especially if it has a share or link) so obviously it would get picked up quite quickly when re-posted as it was looking for it.  I wonder if it would be picked up as quickly if it had been left for 3-6 months.

  • http://jacksonlo.com Jackson Lo

    Google did indeed de-index that page after I moved it from published to draft. Because it was removed from the XML file and it was no longer accessible, it disappeared off Google’s radar. 

    It may or may not make a difference, but IMO I don’t think it would matter. Googlebots will crawl your site everyday, but it’s a matter of how deep your pages are (accessibility). If this post was 3 levels down, it likely will take longer for it to be discovered again. I did not tweet or share this the article after it was republished, but would likely get noticed quicker if I did. 

    You should run a test and see if you get similar results :)

  • Jason Kelley

    The only thing similar I have done I guess is that on one  site I removed a page and I was getting crawl errors to the URL long after a couple of weeks. So I would guess if a page  had any incoming links or shares before being removed crawling attempted would most likely go on for some while.

  • http://jacksonlo.com Jackson Lo

    Great point. If there are links pointing to the destination URL, that could help speed up the indexation of those pages (from the external sources) are crawled and the bots land on your site.

  • http://www.heartratewatchcompany.com/polar-cs500-cycling-computer-p/polar-cs500-cycling-computer.htm polar cs500

    how to remove index of several pages of a website from google?

  • http://mymobitricks.com/ Sagor

    Bro,Itz really a helpfull tips for me.. Tnx Lo.

  • http://www.techvowel.com/ Rohit Agarwal

    Nice tips..

  • Donna Gaeta

    Those tips are wonderful if you understand what a sitemap is.  Although most hosting sites require you have one.  Especially if you build on a site builder.  Other wise donate a complete page and list each ftp file connecting to your main domain.  The search engines love that.  But they also love back links.  (posting your url all over the internet) Google+ will rank your local ad campaign the best. And get your website crawled very quickly. Donna Gaeta

  • Simi Reizen

    Thanks, wondering about the same, just finished making updates to my site (Simi Reizen), and now waiting for it to pick up again.

  • comments_botw

    weird experiment:)