If you want to know if you "have" to upgrade or not look at IPS release notes, if there is security issues within a release you'll see a warning icon next to the release.
https://invisioncommunity.com/release-notes/
I'm running 4.3.6 on one site without any security issues. My other site is running 4.4.1 since that is the last "security" release.
It doesn't matter if you have a gaming board or a ton of members - the evil bots could care less about that stuff.
Best advice, upgrade to be safe because if something goes wrong you're going to be spending a lot of time cleaning up the mess. If you're frustrated with the upgrade process and your themes / plugins - simple, don't customize anything. Then you generally won't have an issue.
Accept the fact that if you customize, you're going to have to put in time to upgrade. That's just a fact of life.