Hey there!
Here is a few tips I could think of off the top of my head:
Well first of all you need to have a niche in the market. A topic that not many other forums have control off.
Another thing is being unique/original, if it's a gaming website, why should they join your website and not another website?
Don't have too many forums. As a new website you don't want to have a lot of dead forums. When starting out, ensure you don't have Call Of Duty Mw2 general discussion, Call Of Duty Mw2 Support, Call Of Duty Mw2 Modded Lobbies, just have one or 2 forums for it that will be actively used every day.
Don't limit guests to viewing the content. Obviously, your site will never get indexed on Google if your forum is setup to only allow registered members to view any of the site. A lot of traffic is from people searching for things and stumbling across your website.
Putting searchable keywords into your titles and posts, writing meta descriptions, and using alt tags on your images can all improve your SEO ranking.
Enable the feature on IP.Board (And I'm sure on vB/XenForo too) that enhances the topic title into the URL.
Free themes are geat but never pick the one that is the most popular unless you are going to be customizing it. The reason I say this is because a free theme can just make you look cheap, so if you are going to choose a free theme then make sure that it is one that many people haven’t seen. Plus, you don’t want to use a theme that other forums are using because it will cause confusion among visitors and it may cause some to think that the forums are the same. If you have a unique, clean and easy to navigate theme then that is what will keep people coming back (the content will too). If I go onto a site, I am always judging the forum on how it looks, we all do. If I don't like the layout/style of the website, the chances are I wont register.
Have a good staff base. Obviously you don't want the whole member base being staff, but having dedicated members to eliminate the spam and answer site related support is always a good thing.
Good luck!
Kingy