The narrow smooth tyres on a road bike are pretty useless on loose gravel or soft surfaces and the wheels are too easily damaged by rough trails. I have ridden off road some on mine and so long as the track is reasonably firm and not too rough you can get by being very careful. Steep gravelly surfaces, rocky, muddy, sandy, etc then you want a mountain bike.
If you are mostly riding on the road and want to go fast and keep up with others, then a road bike is the better option. Cheap mountain bikes with knobbly tyres are much slower than road bikes on the road.
If you want to ride trails and the road, and don't want to get two bikes, get a quality lighter weight mountain bike with a second set of slick tyres for the road. A second set of wheels for the road would be a more expensive but more convenient option. You still want knobbly tyres for serious offroad work.
Pat