I have ridden easy with a low grade fever and even did some moderate training with a fever when my teammates needed me for a TTT practice. I ended up with a head cold, and was forced into a low volume week or so where I was mostly focused on getting well. This ended up being an unplanned taper leading to my best peak of the season!!! My first weekend back racing after being sick, I had my only win of the season.
So, listen to your body, if you are sick and need time off, take it, or just ride easy. Don't be afraid that it will derail your training altogether, because sometimes the body just needs a little break to come into form.