I ride road bikes through hardpack and loose-over-hardpack gravel trails a lot. The only thing I ever ruined was the rear axle on an old school freewheel hub. However...if you have expensive drivetrain components you may want to reconsider because they will get a lot more grit in them and will undergo accelerated wear. My sign-in name should offer a clue about why this factor doesn't concern me much.
Also, the bike will be much more prone to slide in the corners especially if you bounce a high pressure, skinny tire off a large piece of gravel.