Okay, I didn't tighten the little 3-piece expansion thingy enough, that must be why tightening the top hex bolt didn't do anything to relieve the looseness.
Somehow I had thought the little expansion thing was just to keep the carbon steer from being crushed by the stem.