The XF template system in XF is bad in my opinion. There really is no need to dive into FTP for templates (for IPB) unless you have a specific editor (like Notepad++) that you like to use, or you're uploading team icons or images for the style. XF's built in template editor feels messy and weird, and it just doesn't work for me. I can go and edit anything for my site template and be done within a matter of seconds with IPB, just doesn't happen with XF. Not to mention the fact that XF's default template is horrible and requires a ton of work if you want a nice looking style based off of it.
Regarding updates, I'd rather have massive updates than smaller updates. Sure, I might need to redo my stuff, but fixing things for every small update is a pain and just not something I have time for. Big updates are also nice because we get to see all of our requested features come at the same time, not scattered out between multiple updates.
Regarding vBulletin, 4.x.x is OK, but XF and IPB murder it in terms of functionality. vBulletin 3.8 has, and probably always will be the high point of that software. After that, it's just been a downward spiral. In terms of security, vBulletin isn't THAT bad, it's just many site owners have no idea how to properly secure their site, and vB requires a bit of extra work for security. It is known to have had some pretty well known exploits in the past though.
IPB is not bloated at all though, I have yet to find useless features in the software. Not to mention that IPB has other official suite addons, like Nexus, Downloads, Content, and Blog. XF only has Downloads, which is very disappointing to me.
Overall, if you want a complete suite, open source software, and a platform that is very flexible, go with IPB. If you want a "community" feel, go with XF. I've used both extensively, and IPB is just better IMO.