For ANYONE, the Infinito series is a fast bike. It's a racing bike out-of-the-box. It may be a 'more comfortable' racing bike, but it's a racing bike none the less.
If you want to tweak it into a full blown, ballz to the wall racer you can lose the (assused OEM spec) compact crank and bolt on a 39-52, slam the stem to compensate for the head tube that's just a bit taller than the Oltre, maybe get an FSA flat bearing cover/top cap to lower the stem and then head to the races.
And anyone that turns their nose up at an Infinito on the starting line is a complete, hopeless moron.
Personally speaking, I would not enter such a high-end frame in a Cat. 4 crashfest criterium. Frankly, it's too good to risk with a field full of freds, posers and tri guys looking for sidewalk bragging material.
Did you get a Campagnolo equipped model?
There is nothing wrong with Spesh or Cervelo frames, but it is their experience that matters. Any results they achieve ares attributable far more to what they know, how they train and race than what they are astride.
Some ProTour riders sit on brick-stiff frames and some opt for that Domane or Roubaix type frame. It's a choice that should not provoke upturned noses. And if it does, it is their problem. Not yours, OP.
Good luck and stay safe in your races.