I reckon 6 of one half a dozen of the other. Frame size within +/- 10mm can fit properly by selecting bars/stem etc...
If you go smaller, then the stem will be longer which might give you less twitchy steering (conversly, you might want a shorter stem & more direct steering).
The other consideration is that in a time trial set up, you'll be sitting further forward & specific time trial bikes have very short chainstays.
I think I've convinced myself you should be on a smaller frame...