Sites hosted on SiteGround last week were not crawled by Googlebot, Google’s crawler. The problem appeared to have been a DNS issue between the provider’s partners (AWS) and Google according to the hosting company. The DNS issue was resolved after a few days, but with DNS it takes a long time to update and sites now start crawling again.

Like I reported at Search Engine Land last week, Matt Tutt was one of the first to notice this problem – where it started on Monday, November 8. SiteGround said it fixed the issue on Friday, November 12.

Here are the tweets from SiteGround:

We have reported the issue to Google and are working to resolve and identify the cause of the issue. We will keep you posted once we have more information or the issue is resolved.

– SiteGround (@SiteGround) November 10, 2021

Status Update: We are happy to inform you that we have implemented a fix for the Google bot crawling issue encountered by some sites. The websites are already crawled successfully. Please wait a few hours for the DNS changes to take effect. Thank you for your patience!

– SiteGround (@SiteGround) November 12, 2021

But was it completely fixed?

Why did I miss something? They just said they traced it back to that specific problem, no solution was found. I just tested one of my sites and got the answer below. pic.twitter.com/Mh42Eci1QC

– Matt Tutt (@ MattTutt1) November 12, 2021

There were more complaints after the 12th and then SiteGround said it applied additional fixes on the 13th:

We applied a patch last night and the sites are gradually being re-explored. Regarding CPU request, please send your domain to us by DM so that we can investigate and advise you further: https://t.co/zMYnoTzgn7

– SiteGround (@SiteGround) November 13, 2021

It seems that since the 13th SiteGround customers have been quieter about the issue, which may mean the issue is now resolved?

The lesson Google Help for Webmasters The thread where Google’s Caio Barros helped deliver updates to those affected also appears to have calmed down. It’s an interesting thread to browse, if you want to see it.

Google’s John Mueller also provided information on how Google is dealing with these DNS issues, in short – once the issue is resolved, things should get back to normal over time.

First of all, it happens from time to time, it can happen to us, it has happened to other great services ( https://t.co/NjQeVR0Hrr ). Short-lived outages sometimes go unnoticed because DNS can be cached.

– John 🧀 (@JohnMu) November 12, 2021

Googlebot usually performs DNS lookups on the same servers. So our new list of IP addresses can help you verify and authorize the list. You can test using the mobile-friendly test tool and URL inspection in Search Console.https://t.co/32pLcCkiXz

– John 🧀 (@JohnMu) November 12, 2021

Because DNS server changes take a long time to propagate, unless you know they will persist for more than a day, it’s usually not worth the effort to move to another infrastructure. It is often easier to maintain DNS with hosting, so that any change of IP address can be updated automatically.

– John 🧀 (@JohnMu) November 12, 2021

There are usually no lasting effects from temporary outages like these. Technical issues come and go, we need to do our best to make sure that users can find their way to your wonderful sites through the search results.

– John 🧀 (@JohnMu) November 12, 2021

Due to the way Search Console aggregates URL information through indexing, the error count is unlikely to drop to zero immediately (although the errors are now gone) – that’s fine and just a problem. report.

– John 🧀 (@JohnMu) November 12, 2021

If you’re still around and wondering how to completely avoid this in the future: it’s difficult. Traditional hosting providers do an incredible job of keeping things running, and it’s extremely rare that I hear of outages.

– John 🧀 (@JohnMu) November 12, 2021

Send them chocolate (not cheese):

What would I do if a site I loved was affected? I would probably be angry and send chocolate to the people involved. They’ve done a great job, and I’m sure they’ll coordinate with the infrastructure folks to make it work even better. ??

– John 🧀 (@JohnMu) November 12, 2021

SiteGround is not a small hosting company, Wikipedia said the company hosts around 2 million domain names.

To be clear, I don’t think this was a big AWS issue, if it was, a lot more sites would have been affected. Seemed something specific to SiteGround and how they used AWS?

Discussion forum at Twitter and Google Help for Webmasters.