i built a fixed/ss from an old unknown brand very light roadbike found at a thrift shop for less than $20. rides great though not completely straight. this winter it's become my really bad weather bike.
also picked up an old raleigh grand prix, replaced the bent fork with a tange and laced up alloy rims w/ flip/flop hub. this is morphed into a really sweet fixed gear commuter. for just cruising around and commuting and running errands, it rocks.
i too always had my eyes on a track bike and after being dumped by my girlfriend for the umteenth time i took advatage of the situation and ebayed a Mercier break-up bike. i think it came with a 48x15 fixed and was a rocket with no brakes. the big gear required times of resting and great care in not smoking through familier turns. it's really tall feeling with very steep angles and light modern tubing. the top tube picked up a small dent when it lost a battle with the raleigh when one fell into the other one day. i use it for commuting and rain rides and just ripping around town. it's been re-geared with seat stem and bar changes with a front brake added and bigger tires.
point is that all the bikes are different bit simplicity of riding, maintenance and perfecting technique is the beauty of the fixed. sure, i still have thoughts of taking one to the track, though not nearly so often. buying a new frame and building from scratch really makes a bike yours. perfection does not exist and every choice has some trade off. the Gunnar sounds like what you'd really like. go for it.