I'd say it all depends on your goals.
Do you want to race? What kind of races? Or are you riding purely for fitness, or to lose weight? Are you riding purely for the fun of it?
I do a combination of everything. Last weekend I raced a 50 miler. So for 2 weeks prior to the race, I concentrated on power. Shorter, very intense rides, with more rest in between.
I'm not in any events till late Sept. now, so I'm just riding for enjoyment and base miles. Today I'll ride a century, and probably won't beat myself up much with speed, because I'd like to ride tomorrow as well. Besides, it's a beautiful day, and I'd like to enjoy it to the max on the bike.
My next event is a century ride, so in the weeks prior to the ride, I'll concentrate on building power and average speed for that particular distance.
My advice is just ride. Mix it up. But it also depends on your goals, and also how you feel on any particular ride. If you feel like riding hard, ride hard. If you feel like riding long, ease up on the intensity.
It helps to keep a log. I always keep average speed on the back of my mind while riding, but don't get hung up on it. Average speed depends on lot on conditions and terrain. Plus, if you always try to up your average speed, it's hard to do recovery rides.