Google Ends Sitemap Ping; the Endpoint Retires in 6 Months

With this announcement, every sitemap ping request – HTTP ping requests – now shows as 404 in the console.

The sitemap protocol uses the REST method to submit the sitemap file to search engines. REST is an architecture for distributed systems that transfers data over the HTTP protocol. With the REST method, search engines could receive the sitemap file by sending an HTTP request to the sitemap endpoint. (which is now deprecated)

http://www.google.com/ping?sitemap=http://www.example.com/sitemap.xml

Google says: submitting a sitemap through the REST method was an unauthenticated REST method — it needed no verification and no signing in to the system, and the sitemap file could be sent to the search engine with the REST method alone. Based on our research, including research on the Bing search engine, we consider that approach pointless and unhelpful, and in many cases we also identify it as spam.

Note: unauthenticated REST method pings to the endpoint can be spam for search engines.

Google says: You can declare your sitemaps through Search Console and robots.txt as always, but keep the recommendations we give below in mind.

Pay attention to lastmod in the sitemap.

The concept of content updates and the use of a last modified date on a page had been mentioned before, but this time Google states clearly that a change to a page or to the content has to be substantial, and that annotating a page or making a few minor edits here and there does not mean editing or changing anything at all. Google stresses that you should use the lastmod element in your sitemap file only when you are sure the page really has changed.

Then there is the more sensitive point, which is also a technical matter: keeping lastmod current. If your CMS changed a trivial string in a sidebar or in the footer, that does not mean you should refresh the lastmod value of that page. However, if you changed the main text, some of the links or the page’s schema, you can update the lastmod value.

Google can quite easily come to distrust your update dates and ignore them, even without your knowing it.

Google says: Lastmod is genuinely useful in many cases, and we use it as a signal for scheduling the crawl of links that have already been discovered. We recommend two things so that you get the best out of lastmod and everything runs well:

1. Check the date you put in lastmod so that it matches the sitemap standards. If there is a problem with a date in the sitemap, the console will warn you.

2. The other decisive point is that the lastmod date has to match reality. If your page changed a few years ago but the lastmod element tells us it changed yesterday, then when the page really does change and the sitemap is refreshed, we will no longer believe you.

Google says: You can use the lastmod element for every page in the sitemap or only for the pages you are confident about. That means the element is not mandatory for all addresses; what matters is that it is accurate and up to date.

For example, some site software may not be able to determine the last change date of a category page easily, because it only counts the other pages on the site. In other words, the update date of some pages, such as category pages, is derived in aggregate from the data of the other pages in that same category.

Sometimes the site software cannot easily determine a category page’s last change date, because the information about it comes from the other pages in the same category. A news site, for instance, may have several articles in one category, and in that case the software aggregates the last change dates of the articles in that category to date the category page. So the software cannot determine that page’s last change date easily, and when that happens, using the lastmod element for that page is not necessary.

Google recommends that although each page’s last change date is given with the lastmod element, if you cannot determine the last change date for some pages, you should remove the element.

Now we come to the http address defined in xmlns, which has made some people ask why it is not https. Google indirectly says that address string has to stay exactly in this form.

In an XML sitemap file, the xmlns attribute has to be used to identify and define the namespace in use. In other words, xmlns lets you specify the namespace used in the XML sitemap file. xmlns sits in the root tag of the XML sitemap file as an optional attribute and is usually set to the value “http://www.sitemaps.org/schemas/sitemap/0.9” so that it identifies the sitemap namespace.

To explain further: to define the XML sitemap file you declare the root tag as urlset and give the xmlns attribute the value “http://www.sitemaps.org/schemas/sitemap/0.9”. That attribute tells search engines that the XML file contains a sitemap and that it must be read and processed in this format.

We come to another sensitive discussion, this one about the use of changefreq.

Google does not use the changefreq and priority elements. The changefreq element seems to overlap conceptually with lastmod, and the context of the priority element is so subjective that it does not accurately reflect a page’s real priority compared with the other pages of a site.

1. Google no longer supports sitemap Ping or the REST method; it considers them pointless and unhelpful, and it identifies them as spam in many cases.

2. Every sitemap ping request will return 404, so if this method is used in your site’s code, in your site software or in your plugins, make sure from now on that you are putting it aside.

3. Google recommends using the lastmod element in the sitemap but also flags points that matter. A lastmod date means a substantial change to the page, and a few ordinary edits do not mean the page’s update date has changed, so lastmod should not be altered for something like that.

4. If the page’s real update date belongs to a time that conflicts with the lastmod date, Google comes to distrust your update date and ignores it. That is to say, when an update really is made on the page and you refresh the lastmod date too, Google may still disregard it and carry out its own assessment.

5. You can use the lastmod element for the pages you are genuinely sure have changed, and leave it out for the other page addresses.

6. Google does not use the changefreq and priority elements, because they overlap conceptually with lastmod or because they do not accurately reflect a page’s real priority compared with the other pages of a site.

Note: given the importance of the news published here, extra explanation has been added so that some of the definitions are easier to follow.