I am not a physicist or an engineer, so this is just my take on the situation.
There are two aspects (I think) here.
1. Drag is proportional to wind velocity squared. So riding into the wind, you add your velocity to the wind speed and square it. With the wind, your velocity minus wind speed squared (if negative, you are being pushed) . Either way, the benefit of the tailwind is less than the loss to the headwind.
2. Because you are going slower into the wind, you spend more time on that leg, which would pull down your average speed compared to a windless ride at the same effort. Average speed depends on time at speed, not distance at speed.
(Now all you phsyicists and engineers can tell me why I'm wrong!)