There's certainly something to be said for your current inclination -- at this stage, your primary concern needs to be fit. Find the appropriate balance between what your local shop guys indicate an appropriate range of configurations to be (considering frame size, stem length, stem rise, and seat height, and the resulting effects on your back, neck, reach, pedal stroke, and so on), and your own personal sense of comfort. Sizing isn't a perfectly exact science... these guys should be able to offer you a range of fitting tweaks based on your preference and style.
Once you've got a good sense of fit down, feel will probably become less and less ambiguous to you. There are differences in handling; do the angles and stem selections contribute to a more responsive ride? Does it turn quickly, or is it more forgiving?
There are also differences in road feel -- an intriguing, tricky, and sometimes controversial issue which is always promised to be at least 65% completely subjective. It's more apparent when you've been riding longer and become more familiar with your ride(s), but different bikes convey different sensations. Some feel plush, or heavy; some seem to transmit every pebble and grain of gravel right into your skull; some seem soft and flexy; some feel rock hard, or stiff. Some can be described as lively, while others feel dead. Componentry plays as big a role as the frame does; wheels, tires, seatposts and saddles typically receive attention.
The terminology gets a bit too self-involved and hyperbolic, as with wine-tasting, but there's a reason some people are so passionate about considerations of "feel." If you love your bike, and you put money into it, you want to know it inside out -- you want to understand the subtle hums and whirrs that differentiate your bike from the next.
But all in good time. You're doing the right thing by stressing a little about this (though just a little). Keep test riding, and keep looking for a proper fit. In all likelihood, the right bike will jump out at you eventually.
And when all else fails, all else being equal, go by paint job.