Hope it's not too late but have you considered gravel bikes? These bikes are competent enough for both road and trails / gravel.
A gravel bike is also safer to use on road with traffic than traditional road bikes. The wide tires allows you to pay more attention to the vehicles around you. Why? If riding a road bike with skinny tires, you are compelled to watch out for road imperfections like potholes or some poor repair work. Then you'd probably want to swerve around these and that leads to unsafe conditions in the presence of vehicular traffic.
You can either get hit by vehicle by swerving OR you may spook other drivers when you swerve near them, causing dangerous distraction, which may lead to accidents. Or it may be too late to swerved around a big pothole, there's a car beside you, you hit the brakes pretty hard which may lead to a crash or you ride over the pothole anyway at high speed and pray to God, nothing goes wrong. But a skinny tire of a road bike can get damaged if you do that. It can pinch flat and at high speed, can lead to a crash.
You can also 'bunny hop' over a pothole but honestly, such action can be quite harsh on a road bike. I wouldn't buy a used bike from someone who regularly bunny hops their bike even if they never crashed and competent in bike maintenance UNLESS I'm buying a used MTB from them. Plus, bunny hopping can also cause dangerous distraction to drivers so don't even bother unless it's absolutely the last option available to you like a huge crack opening on the ground.
A gravel bike on the other hand with wide tires will allow you to ride fast over potholes and other road imperfections without any problems as long as you're using the right tire pressures. That allows you to keep riding straight, safe, and be able to pay attention to traffic around you, instead of scanning the road for potholes and bumps. Some gravel tires are good for both paved and gravel rides. They tend to have flat center and knobby sides.
And a gravel bike don't really slow you down even on fast group rides on the road even if riding with wide tires. Your kit, riding position, training, and mental state will make a bigger difference on speed unless you're actually riding a heavy steel commuter bike with full fenders, racks, solid tires!!, and baskets. Heck, I've been on fast group rides pushing 25 mph on long flat sections where everyone is riding hard tail MTB's on wide knobby tires!