The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

Crawlers and Ajax


Does anyone know whether search engine crawlers can see Ajax generated pages?

For example if I have a link that when clicked runs a javascript ajax callback, will search engines like Google be able to process this?

Wednesday, July 12, 2006
They don't. There have been rumors about crawlers based on headless browsers being developed and tested, but apparently they aren't ready for the prime time yet.

To get search engines index your content, it has to be reachable without JavaScript and there must be pointers (well, links, really) to it that don't need JavaScript either.

You might want to rethink your approach in any case, since being reliant on JavaScript usually means not supporting deep linking. And not supporting deep linking means making people less likely to link to you. Not to mention potentially compromising browser compatibility and accessibility unless implemented carefully.

Try looking up graceful degradation and progressive enhancement on Google. These are development approaches that allow you to increase compatibility in less than optimal circumstances without downscaling the best available user experience.
Aapo Laitinen Send private email
Wednesday, July 12, 2006
Take a look at Designing Rich Internet Applications For Search Engine Accessibility -
son of parnas
Wednesday, July 12, 2006
I heard Google is doing tests with their new crawler based on Firefox instead of Lynx... So, I guess it's coming.

Thursday, July 13, 2006
Depends how the app is coded.
I expect that if crawlability was a priority, you could look into ways of making your AJAX app still crawlable in some contexts. But yeah this is obviously harder work (and may not always be possible.)

I guess one approach you could take is code up bare-bones HTML views for your app, and then serve them up to user agents who can't deal with AJAX, or appear to be spiders...
Matt Send private email
Thursday, July 13, 2006
Yo -- it's not Ajax!! Please stop calling it Ajax and funding the hype.  It's called JAVASCRIPT and the standardized version is called ECMAScript.
Friday, July 14, 2006
Wrong. Javascript/ECMAScript are languages, XML is a data description, and Asychronous communication is a design pattern or something.

AJAX is the use of all three in a web app to create a thicker client.

If you're not building a thick client using those technologies (a static dropdown menu), then it's probably just Javascript. But if you're capturing the click to load content off the server asychronously to avoid a pageload and improve user experience, voila! AJAX it is.
PHDude Send private email
Friday, July 14, 2006
Why would google base their crawler on lynx or firefox? Surely their crawler would be written from scratch.
Zach M.
Friday, July 21, 2006

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
Powered by FogBugz