i suffer a similar fate. if i dont warmup correctly i will suffer a significant drop in performance. not completely sure on the science behind it, but i now follow a plan which works for me.
if for example i am racing a tt at about 2 pm i wil get up and have breakfast at about 8am. On the bike at about 9am for one hour easy at about 125hr (my rec zone), usually riding over the course if it is mostly flat. when i come back i eat my final pre race meal. eg pasta or what ever you prefer. This gives you time to change out of your cycling clothes and shower into casual clothes, leave the hotel or house and head to the race.
At the race about an hour out from start time, head out for 20 mins at e1 on the road. When you get back jump on the rollers or trainer to get a precise warmup. its hard to do it correctly on the road due to traffic, wind and hills etc.
The idea of a warmup before a TT is to get all of your energy systems working at maximum BEFORE you start. But to avoid putting heaps of lactic acid into your legs you must progressively warmup.
The time spent at each hr can be tailored around your own personal preferences but as an indication the warmup basically involves increasing your hr until you start to feel lactic and then recovering. I would first raise my hr to 150 and hold it for 2 mins before dropping to 130 for 3, then 160 for 2 mins and back to 130 for 3 etc. continue in this fashion until you reach AT then have 5 min recovery. Then i normally just do a 15 second sprint to make sure the anaerobic system is on. Spin for a few more minutes then it should be your start time.
this routine is time consuming but one that i will follow for really important races, as it really makes a difference.
