Wednesday, Aug 3, 2016
We are no longer recommending the AJAX crawling proposal we made back in 2009.
Here is an example. Take a look at one of the simplest Baasic blog starter kits. It is built using AngularJS and pulls a list of blog posts from our API, rendering it in within a simple layout. Everything seems to be in order, isn’t it?
However, this is what happens when we try to find a word from an individual post in the Google search results:
The static content of the page itself is indexed, but the dynamically generated stuff is missing - here is a proof for that:
So, while Google is generally able to render and understand your web pages like modern browsers, it is not currently covering all possible scenarios. This is obviously not acceptable for production environments, so let’s go a step back and revisit the existing standard.
Ajax crawling scheme
Google and other advanced search engines support the hashbang (!#) URL format, which was an original method of creating permalinks for JS applications. Whenever a crawler sees the URL containing the hashbang -
it will transform the URL into
The newer HTML5 pushState doesn’t work the same way, as it modifies the browser’s URL and history. If you are using pushState, you should use the following tag in the header of your pages:
<meta name="fragment" content="!">
This tells the Googlebot to revisit the site using the ?escaped_fragment= in the URL.
Don’t be confused if you have read that Google doesn’t support the Ajax crawling scheme anymore. They are not recommending it, but it is still supported, and it will stay that way for the foreseeable future.
We have opted to run a dedicated prerendering service on our own servers, using Prerender.io.
The general idea is to have the Prerender.io middleware installed on the servers rendering your applications. Middleware is just a fancy name for a package or a set of URL rewriting rules that check each request to see if it’s coming from a crawler. If it is a request from a crawler, the middleware will send a request to the prerendering service for the static HTML of that page.
If you are serving your applications from ASP.NET, the middleware can be a simple HTTP module. However, we are usually opting for the approach that uses URL rewriting rules. It can be used with Apache, Nginx, or any other server - the actual packages and instructions can be downloaded from here.