Having no brakes is the main thing. It's for safety because everyone is going at very high speeds, and extremely close to one another, so of course with brakes you could stop a bike way faster than the reaction time of the rider behind you and that would cause a crash. And if you took the brakes off your geared bike, well now you have no way of slowing down because you will just coast uncontrollably, whereas on a fixed gear you can resist the spin of the cranks to slow it down, and yes it takes a while to fully slow down but same for everyone else, so when the rider infront slows, you slow. The key is that the rider infront can never stop suddenly. And yes when you are out on your road bike riding super close in groups you all have brakes, but out on the road it is safer with brakes, and you are generally going way slower than in the velodrome. In the velodrome it is safer without brakes because there are no obstacles or any reason to stop for something; the only time you need to stop is when you are finished riding, so it is safer for riders to just roll around the track slowing down until they leave the track and come to a complete stop.