Sounds similar to my road bike usage (not racing, but quite often long rides...) I'm picking up my Cannondale Synapse on Friday.
I took one for a test-ride over some of the nastier road surfaces around town, and it was spooky just how smooth it was. After 50-odd km on these roads I had felt like I had barely done 20 on smooth tarmac. And this was on a bike that was one size too small, and not really fitted well to me at all!
I don't mean to confuse you by giving you yet more bikes to consider, but there's an emerging category of "performance/epic road" bikes out there for people like us, where the bikes still have a sporty feel but have the geometry and frame lay-up tweaked for long-term ride comfort. The Synapse is one, the Specialized Roubaix is another.. there are probably more out there, but they are the carbon ones I know of off the top of my head.
I didn't get a chance to test-ride a Roubaix, though most of the reason for the bike upgrade in the first place was being able to get a super good one-off deal on a Synapse through my LBS. Specialized is a bit thin on the ground here.. maybe it's something to do with that big misspelt word on the downtube