If you had the green and polka dot jersey at the same time there is a high probability that you would also be wearing the GC leader's yellow jersey.
I mean not wearing, but entitled to wear.
Would be rather stifling in the French summer to be wearing three jerseys at once!
But if you held the prestigious yellow jersey and were a genuine GC contender, you would be defending it rather than going for intermediate climbing or sprinting points and time bonuses.
For the polka dot jersey there are intermediate checks on hill/mountain tops for bonuses. Flat courses = no bonus points for climbers. Same for sprinters but on a mountainous stage there are no intermediate sprint bonuses.