can't add much to what people have said, but that won't stop me🙂
If you want to walk around, go with the type of cleat that fits in a mountain bike shoe. There are a lot of mountain bike shoes that will suit the needs of very serious road riding. I use a Specialized Comp MTB shoe and have been very happy. I happen to have Shimano spd pedals, but there are other brands with similar cleats. You can get spd-type cleats with float. They might not have as much float as others, but I've found them to have plenty for my needs. Some people don't like float because the connetion seems a little less solid. Of course I don't know what kind of knee problems you have, but the knee sorness I've gotten from cycling have been far more related to seat height and position than cleat float. Actualy, the new cleats I installed a few weeks ago have no float (I don't know where I got them, they were in my box of "extra bike parts".). My knees have absolutely not noticed the difference. I'll have to say, I slightly prefer the more solid feel, and they're easier to get out of.
For the time being, try the Power Grips straps. I think you might find them better and more useful than cages/straps.
Also ditto on the cycling shorts. Very important.
Finally, to get started, you don't have to spend a fortune on the shorts, pedals and shoes. I believe my pedals and shoes cost about $40 and $75 each, on sale; I wouldn't be surprised if you could get into a MTB/pedal combo for $100 and the quality would be fine. Not elite grade, but very good. I absolutely do recommend shoes with velcro straps and no expose laces. When i first bought shoes, this was one of the main reasons I liked them - no laces to deal with. The support I only noticed after I tried ridng in regular shoes later. (and that is a huge deal).
As for shorts, same thing. You could find some for $30 easily. You won't know which kind you like until you try some, so just try some. I have used the cheapest shorts from Performacebike.com, and have been happy. They're not as nice as the more expensive shorts (3X the price) my wife bought me, but they're fine and CERTAINLY better than no shorts.