You first need to have the subdomain setup with your host. Then follow these simple steps to get it working.
ExpressionEngine 2.x:
1. Open /system/expressionengine/config.php and add the line
$config['use_forum_url'] = 'y';
2. From the ExpressionEngine Control Panel, click Addons > Modules > Discussion Forum. Then click the “Default Preferences” button under the “Forum Board Home” tab. Under General Settings, set the forum trigger word to “forums” and set the Forum URL to “http://forums.example.com”
3. In your .htaccess file, add a rule to redirect the forums.example.com subdomain to the ‘forums’ trigger page:
RewriteEngine On
RewriteCond %{HTTP_HOST} forums.example.com
RewriteCond %{REQUEST_URI} !index.php
RewriteRule ^(.*)$ index.php/forums/$1
ExpressionEngine 1.x:
1. Open /system/config.php and add the line
$conf['use_forum_url'] = 'y';
2. From the ExpressionEngine Control Panel, click Modules, click Discussion Forum, click Forum Management, click Default Preferences. Under General Settings, set the forum trigger word to “forums” and set the Forum URL to “http://forums.example.com”
3. In your .htaccess file, add a rule to redirect the forums.example.com subdomain to the ‘forums’ trigger page:
RewriteEngine On
RewriteCond %{HTTP_HOST} forums.example.com
RewriteCond %{REQUEST_URI} !index.php
RewriteRule ^(.*)$ index.php/forums/$1
Notes for all versions
If your host uses .htaccess to control subdomains (EngineHosting is one such host), the above .htaccess rules should handle everything sufficiently. Mapping the subdomain to a physical folder on your site in addition to this rule will cause Rewrite conflicts and not work properly.
Additionally, if you do not have mod_rewrite (Apache) or ISAPI_rewrite (IIS) on your server, you can use the standard method of running ExpressionEngine subdomains, however, you will have to run your forums through regular templates, which will increase the server overhead for running the forums.
If your host does not use .htaccess to control subdomains, and instead creates a new folder for each subdomain, then follow the directions on running ExpressionEngine subdomains instead of using .htaccess. This does NOT require that you run your forum through the regular template engine.