Yeah, unfortunately, if you upgrade from 3.x to 4.x without saving backups of your files or, at the very least, your database then you're out of luck. You can't downgrade.
4.x has a learning curve to it but I assure you, it's the best series Invision has released to date and 4.3 / 4.4 are the crown jewels to this point. 4.5 has a lot of potential and 4.6 is right around the corner.
Are the backups from the 3.x series? If not, then no. If yes, then simply removing existing files, deleting your current database, and replacing both with your backups will work just fine.; it's that simple.
You should expect data loss from anything post-backup.