It became a problem when when I started using helmet mirror. That's when I realized the huge safety benefits of having rear view.
Before I started using mirror, I always looked behind by turning my head before changing lane, etc. But will never be as good as having mirror on your helmet. On urban commutes, it's a big yes to have some sort of rearview than just turning your head backwards. Even if you never do urban commutes, there's still a chance your rides will take you in more crowded situations.
As well as occasionally an odd vehicle coming from behind who looks like he wants to drive you off the road. You'll never know unless you turn your head or has mirror. But do you turn your head to the back every time you hear a vehicle approaching from behind? I doubt everyone does it all the time.