Yesterday was the first day we saw that the Well.ca servers started to cave under increased load. What could be happening? “We’ve been so diligent about handling demand,” we thought.
After a full day of off-and-on debugging, and a couple server-restarts we noticed that we were being hit by thousands and thousands of page requests from individual, but spread-out IP addresses. Our MySQL servers started to cave under the pressure. The pages were all 404′s (i.e. “Page not found” pages). Did some Canadian anti-lipstick hacker mafia choose to DoS-hack our site?
Nope, it turns it out it was a quirk with Internet Explorer 6.

If you go to any random page on our server that doesn’t exist, we display a nice “I am sorry the page is not found” message but we also give navigation so you can shop around, etc. We also load some Javascript in the header. Here’s the beginning of what used to be the source code for a 404 page on our site:

Notice any problems? No, Firefox thinks it’s okay.
Internet Explorer 6 (IE6) on the other hand…
If you look at line 11 of the code above, we specify a base href property. What this does is effectively say: “Hey, browser, if I mention a link to any other page, assume the beginning is http://well.ca”. So if we provide a link to “image.gif” the browser should understand that to be “http://well.ca/image.gif”. This is really important for a website where you don’t really know what URL you’re in, so relative URLs don’t make sense, and absolute URLs are too much of a pain because you don’t know if your site might be located in the directory of another site like “http:/site.com/yoursecondsite/”. Change the directory? All you have to do is change your base href location.
Firefox thinks this is all greatness. It’s fine.
IE6, on the other hand, requires a single slash (“/”) in front of every URL you use in the header, and it ignores the base href for things in the header. It treats these URLs as relative!
So some images or scripts don’t load, whatever, right?
Well, not exactly. See, once our 404 page refers to the javascript in line 16, and since we didn’t put a slash in front of the URL, it goes and looks for the javascript in a location relative to the current 404 page. A location that doesn’t exist. Returning… yes, you guessed it, another 404 page that has the same problem. As long as a user keeps their browser open, their browser will constantly and continually request non-existing files ad infinitum. In the process, pounding and pounding our server.
Why did this problem creep up all of a sudden yesterday? We have no idea.
But the fix is simply this:

Notice the slashes?
So what’s the lesson:
- If you’re going to have heavy 404 pages, think about possible recursion problems
- Otherwise, think about static 404 pages
- But if you’re going to have rich 404 pages (which is recommended for eCommerce), don’t return rich 404 pages on requests for non-page objects such as Javascript or images (this can be a hard rule to implement with modern URL formats that no longer indicate the filetype requested)
- Be careful with the <base href=”…”> tag in IE6 and links in your header. Use absolute URLs for at least the stuff in the header.
- Consider avoiding the base href tag completely and design sites that assume they are the root.
- Periodically, spend time looking at your 404 error log.
Fact: Awstats is reporting that Well.ca served up 100 GB of 404 pages so far this month — accounting for 97% of our traffic. Yep.
Did this affect sales? Oddly enough, no. Yesterday was our busiest (yet crashiest) day this month for sales, and possibly our busiest day in sales ever. Of course, having a fast, responsive site is important to us — we’ve fixed the problem and we’ll be looking at it even more.