I'll ride down to about 25 F if the sun is out. Rain only if is 60 or above. I don't ride with thunder/lightning, and I really don't like the wind, but can usually map a ride that minimizes the wind. I don't ride on snow/ice either- is simply too dangerous.
It is all about the riding gear, and SierraSlim there is nothing that says you have to buy gear that is "bike specific." Wicking base layers don't have to be $50 each made by ArmorAll (I get mine at Meijer for $10 and they work great). Watch for cycling shorts on sale/clearance and don't forget about retailers like LLBean - who has a great selection of affordable bike gear. Hat and gloves you already have as well as a scarf from your closet will keep you warm. I'm sure my LBS doesn't like to hear it, but when I can buy the same thing elsewhere that isn't designated for cycling, at a fraction of the price, I do.
Now some things cycling specific really are worth the cost. I fully appreciate cycling jackets and jerseys due to the longer tail, elastic bottom and rear pockets. Clipless pedals/cleats are great when you get to that point, but in the meantime a pair of your favorite shoes work fine. Cycling gloves are nice too, but I started out wearing a pair of baseball batting gloves and they worked fine.
Keep riding!