AOLserver Tuning and Scaling Guide

$Header: /cvsroot/aolserver/,v 1.2 2002/09/26 19:51:32 kriston Exp $

AOLserver is immensely scalable because it has to be.  It is the web server for the most heavily-used production environment in the world -- AOL.

Only load modules you need

Modules like "nsperm" register extra requests that are checked on each connection, for example.  Other modules like nscgi and nscp do not need to be loaded if you're not actually using CGI or the control port, respectively.

Turn off DNS lookups wherever possible

The nslog module has these turned off by default, but it helps to see if it your configuration has it turned off.  Also, the nsperm module uses DNS lookups for access control -- turn this off if you don't use host access control rules with nsperm.  The nscgi module also can use DNS but, again, it's off by default (gethostbyaddr setting).  If you must use DNS, AOLserver has a DNS cache that can be tweaked -- see "dnscache" and "dnscachetimeout" in the config reference.

Use the standard ADP parser

The default parser is fully functional but leaves out thing such as registered ADP tags and the "SCRIPT" tag.  The default parser is much more efficient than the fancy parser and should be used on heavily-loaded sites.  In the ns/server/servername/adp/parsers section, only 'ns_param adp ".adp"' should be listed.

Experiment with the ADP cache

In the ns/server/servername/adp section, the parameters are "cache", "cachesize", and "threadcache".  AOLserver defaults to a 5-megabyte cache.  This cache is used to store parsed ADP pages -- this means that ADP scripts are only parsed once and every subsequent time they are run directly out of memory.  This means, though, that you should have a fair amount of core memory on your system to accomodate the cache if you decide to make it very large.

Experiment with the FastPath cache

This is a separate cache used to store static HTML pages.  The section is ns/server/servername/fastpath, and the optionjs are "cache", "cachemaxentry", and "cachemaxsize".  Like the ADP cache, this defaults to 5 megabytes.  On some systems enabling the "mmap" parameter can make it work even faster.

Disable CheckModifiedSince

Is your site updated rarely or often?  On some slower systems, setting "checkmodifiedsince" to false in the "ns/server/servername" section can speed up things considerably as AOLserver reads all scripts and data directly from the in-memory cache without checking to see if the file has changed each time a cache entry is hit.

Tune the Threads

Thread settings are sometimes helpful, sometimes not.  In ns/server/servername, the options are "connsperthread", "flushcontent", "maxconnections", "maxdropped", "maxthreads", "minthreads" and "threadtimeout.  In nearly all cases the defaults are optimal, with the exception of maxthreads which should be carefully adjusted based on your load.  Along with "maxthreads", the "maxconnections" setting can be tricky, since making it too large can cause your system to thrash.

Monitor the Memory

Memory considerations are paramount on heavily-used servers.  Use the "ps -leaf" on most systems to look at the "nsd" processes.  Nearly all the memory used by nsd should be in RSS (resident set size).  If the nsd process' RSS is less than 2/3 its SZ, then it's a good chance that your operating system is thrashing, meaning it's spending more time managing memory than allowing the system to work to its capacity.  Some operating systems actually limit the total RSS used by any one process.  That
1-gigabyte machine might only allow AOLserver to use 500 megabytes of core at any one time -- this is especially the case with SGI servers.

Beware the Database

Databases are a bottleneck.  Do you use lots of simple queries that return a hoard of data?  Investigate using stored procedures or finely-tuned queries so that you get only the data you actually want and make the database do the work it was designed to do.  Don't make your system spend its time putting together those ns_getrow structures.

Collect metrics and use them

AOLserver has a built-in statistics-gathering system that collects data on the caches, tcl interps, threads, and other interesting data.  We use these at AOL to gather an immense amount of data on how the systems are working and where they need to be improved (more cache, less cache, more memory, more threads, etc).   The file "nstelemetry.adp" can be dropped into any running server to get a snapshot of how it's doing.  The "nstelemetry.adp" file can be found in the tests/ directory of the AOLserver Source Distribution.