Although cycling specific clothing does a great job for the specific purpose of cycling, you may find that you already have some items, although not specifically cycling apparel, will do well. The thing to check is the materials of the clothing. Certain fibers are better for certain functions. I've adapted the following equipment as a part of my winter wear.
A nylon shell jacket works great as a wind barrier, cutting out the chill. also, although it is not 100% waterproof, it can keep a light rain from soaking you.
Fleece vests or jackets can add a layer for warmth below the nylon shell and aren't very restrictive for movement.
Thermal underwear can be a good base layer. This stuff is usually designed with the same functionality that cold weather cycling requires. Consider that this clothing is also used in the winter for skiing, snowmobiling, hunting, etc., so why not use it for cycling on a limited budget.
Warm socks that are NOT cotton. Cotton socks will hold the moisture, but not the heat. Cold toes don't warm easily when cycling.Wool or acrylic seem to work. My favorites are Smart Wool brand.
Headband to cover the ears that also fit under my helmet.
Some items you may have to buy, include gloves. As previously noted, there are different types for differnet temperatures. I've got a lined pair of neoprene, that weren't too expensive and served me well through the winter. If you already have other active winter gloves (skiing type), you may want to check them for funcitonality before byuying cycling gloves.
The one item that I couldn't find an equivalent from my non-cycling wardrobe were pants. I got a pair of neoprene pants that I could wear over padded shorts or tights. They cost around $50 dollars and were worth it.
Also, don't forget to wear Chapstik. Lips, nose, cheeks and other exposed skin areas need moisture protection from the cold and wind.
As previously noted, watch for treacherous road conditions that you will see in the winter: ice, potholes, the accumulation of cinders and other deicing materials which like gravel can effect traction.
Save your $$$$ for an indoor trainer and you'll be able to crank when hazardous conditions keep you indoors.
Good luck.