What do you mean by "acting kinda strange"? Were they acting NOT strange at some time in the past? If so then consider reparing before replacing. Changing between different types of cable disc brakes won't provide huge differences in performance.
The most common complaint with disc brakes is dirt or contamination. The mechanisms are tighter than those of V-brakes and if they are not kept clean they will degrade with time. Contamination of the pads or discs (with certain types of cleaners, hydrocarbons, oils or brake fluid) ruins the surface, reduces performance and increases the propensity to squeal.
I once saw one guy who completely destroyed his near new brakes by chosing to clean his whole bike with petrol or kero or WD40 (something like that). He initially denied it even though I could smell it all over the macine and still see the residue of it on the discs. We cleaned up the discs but the pads are almost impossible to clean up as they are porous and soak up the contaminants.
Rotors are standardised on diameter (either 160 or 185mm in this case). An increase in diameter will yield extra stopping power but each size requires a correspondingly sized brake mounting.
I wouldn't use Avid rotors. They're really flimsy and never come straight out of the box. Unless they are dirty (in which case you can clean them) or warped, you can stick with the rotors you have.
If your heart is set on "upgrading" consider shimano cable discs or hydraulics rather than Avids. They are cheaper, far easier to set up, their rotors are more robust and they are no less powerful.