And what a fun vice it is.
Can't give you specifics as they relate to you, but I do own and ride Zipp 404s and Zipp Team Issue (Alu) wheels most of the time. Mostly ride the 404s for personal Time Trial training and the Team Issues for group rides and other general riding. Have had theses for a few years - the newer versions would be the 404 Firecrest and Zipp 101, but both have significantly different profiles (wider, etc.)
My typical ride group(s) target 19-21mph avg over rolling terrain (1,500-2,500 vert) usually in the 30-50 mile range. Most of the roads are in good condition, but some are potted and there are a few RR tracks along some routes. A couple of the RR x-ings are rough and there's no slowing - just hammer through. For that reason I rarely ride the 404s on group rides. Some do and I have not witnessed any failures, but they also turn over their wheelsets more frequently than I do. A little more work to take my turns at pulling with the Alu rims, but in the pack at our speeds/distance it isn't a major issue.
With regard to cross winds - my experience with the 404s is that they slice a pretty steady path over 20mph. Around 24-25mph it takes a really strong gust to push 'em around. ( I'm 185-190lbs, so that factors in also.) Under 20mph - especially 16-18 mph range - I do notice a more pronounced impact from strong crosswinds. I suspect it is related to less directional (forward) inertia to counter the side forces, but I haven't given it any great thought or research.
Many shops have demo programs for wheels. Some are free, some run in conjunction with special manufacturer events, and some cost $35-$50 to take a set out for a day. My suggestion would be to check with local LBSs and find one that has a demo program. Identify your sources and when a particularly breezy day comes along, go demo then. That's what I did with my 404s and was pleasantly surprised at the result.
Good luck!
