The forum software we use is vBulletin and made by Jelsoft Enterprises. While there are many products that look similar (some are free others are not), vBulletin is the most powerful in my opinion due to it's customization features, extensive options, powerful MySQL database, and fast PHP structure. The support staff is also top notch. A very promising new version of vBulletin will be released next year which will take the software to the next level.
What's most important for your server is that it can handle a fairly large MySQL database (several hundred megs as your site grows) and that you have PHP installed. If you have GZip enabled it will be a huge plus as well as it will help to reduce the high bandwidth that a forum can use on your site.
The software license is $160 lifetime and then $30 a year for updates or $85 for a yearly leased license. For info and questions about vBulletin check out their Web site. They have a very active forum there as well which has a wealth of helpful info.
http://www.vbulletin.com