Brake tension? Brakes rub against rim

My problem is that whenever I use my front brake, the brakes wont stop rubbing against the rim. If I push the breaks outwards away from the rim, it rides fine, until I need to use the break. After I release the lever, the brakes pull away from the rim, but are still slightly rubbing against it. The break system is a side pull, single pivot system. Can anyone help please?

Do both brake shoes rub, or is it just on one side? In the former case, most likely it's a case of adding a bit of lube, either in the cable housing, the brake joint, or the brake lever, or all of the above. If that doesn't help, you may need to replace your cable and housing.

If it's rubbing on one side, then you may need to re-adjust the centering of the brake, and/or possibly adjust the length of the cable housing.

It sounds as if the caliper is slightly loose on the fork. You need to tighten the correct nut (typically the one on the back of the caliper, not the front). If you tighten the wrong nut, you will simply stiffen up or lock the brake.
Once you have tightened the nut, or as you do it, you can then rotate the caliper on the bolt to centre it.

__________________
"All that we see and seem is but a dream, within a dream..."

Thanks alot you guys! The caliper was loose on the fork as artemidorus said.

I just took apart the caliper and cleaned and relubed the parts. Then retightened it on the fork and centered the caliper and it worked! No more rubbing and the pads return to their positions.