The Gentle Deceit of HTTP Response Headers

by Jos Claerbout
Web Engineer, WebTV Networks, Inc.
October 2, 1998

Let's say you ask me a question:
"Do you have any bananas?"

To which I respond:
"No. I have no bananas today."

How likely are you to stick around to hear me talk about string beans and onions, cabbages and scallions? Not very long, especially if you've got kids in the car.

Such is life on the web. Every second you're surfing online, your browser is sending out HTTP requests and interpreting HTTP responses in order to get you to the pages you want to visit. Some of the error codes associated with these HTTP responses, such as "404-File Not Found", are very similar to a web server saying "No, we have no bananas". And on a browser like Netscape Navigator or MSIE, you'll be taken to a page that says pretty much that.

A WebTV user, however, will never see an actual "404-File Not Found" page (that link above was fake. Sorry for the drama). Just like our fruit-curious visitor in the example above, the WebTV browser will not display any page that comes after a 404 header.  Instead, the WebTV browser will throw up an alert saying "The publisher could not find the requested page". Similarly, "403 - Forbidden" headers trigger an alert saying "The publisher refuses to give you access to that page".

The advantage of these alerts is clear - it alerts users to the condition without dragging them to an entirely new site just to tell them the page they're looking for couldn't be found.. However, not displaying the 404 page can cause some problems too.

Since the WebTV browser will not display any page that accompanies a 400 or 500 series HTTP header, any content or redirects put on those pages would be lost. Time was, there wasn't any content on 404 pages. But as webmasters have become more concerned with the overall look and experience of their sites, even 404's have become customized. Most major sites now convey informative pages along with their 404's (Note - on a WebTV, you won't be able to see the pages listed below):

These pages do say that the file was not found, but they also add links to other parts of the site, and sometimes an email address to report the problem. So, how do you get these pages to be seen on a WebTV? Instead of saying "No, we have no bananas", try it how the song does: "Yes! We have no bananas today". We'll send the same error page as before, but now with a slightly different header on top.

If you configure your server properly, you can have your customized "404-File Not Found" messages actually return under a "302-Moved Temporarily" header. This can done by specifying a full URL, rather than just a relative pathname, in the file that points to your customized 404.

Apache Servers
For Apache servers, this file is called ".htaccess", and is located in whatever directory you want its directives to affect. If you are using FrontPage, you will want to make these changes in the "srm.conf" file instead, because FrontPage will get persnickety if you modify ".htaccess".

For example, on one of my sites, the ".htaccess" file has only one line:

ErrorDocument 404 http://www.toessel.com/notfound.html

Even though the file "notfound.html" is on the same server as the site, explicitly spelling out the full URL forces a "302" header instead of a "404". This allows my customized "File Not Found" page to be shown on a WebTV.

IIS Servers
The process is very similar for IIS servers. For  IIS version 4, right click on your "default web site" in the MMC, then select "properties". On the resultant pop-up window, click on the "Custom Errors" tab. This interface will allow you to customize the pages your site generates for various error conditions (404-File Not Found, 500-Server Error, 403-Forbidden, etc). A pull-down menu will determine if you are to use a URL, a local file, or the "default" setting. Specifying  "file" will generate a "404" heading, but specifying the same file under the "URL" setting will generate a "302".

If You're Not Your Own Webmaster ...
If you do not have administrative access to the server on which your site resides, you may be limited in your ability to implement the suggestions above. Your best bet is just to contact the technical support department of your hosting company and ask them directly.

Lessons
Had the refrain of that famous song instead been "No, We Have No Bananas", it would not only have been a very short song, but not particularly memorable either. The same applies to web designers. If you want your content to reach the widest possible audience, never begin any sentence with "No", but rather "Sure! Just give me a sec ...".