I haven't been running this forum for very long so take these with a grain of salt (I have run them in the past, also around 2008-2012, but that's a long time ago now and the user count for those never exceeded 10-20):

• I think it's generally smarter to offload it to a VPS. It's fairly cheap and saves you a lot of work/maintenance. It's also slightly safer in that there's less chance of you exposing your home IP and getting DDoS'd, etc. VPS providers will often have better upload speeds than your home internet too.
• This is kind of a mid-level VPS and costs around $10/month. If there were a lot of users, we might need more, but for now this is handling perfectly fine.
• I think most security concerns boil down to proper configuration nowadays. Most software involved is battle-tested and thankfully either defaults to secure options (e.g. phpBB's bcrypt password hashing and token validation) or is fairly simple to set up securely (e.g. mysql_secure_installation, HTTPS, etc.) Using a password manager with long randomized unique passwords for everything will also help prevent people from getting into restricted areas.

I think the biggest challenge you'll have, as arizonapalms said, is just getting people to visit regularly. With more convenient and centralized ways to talk to people, forums are easy to forget about unless you have a way to keep people coming back.