I also believe that putting the smartphone in the pocket is the best idea. I also usually put the smartphone at the same place.I have a Smartphone which has 4,7 inches, it's not that big and often I manage to put it in my pocket.
Do you dislike keeping the phone in your pocket?I usually put my phone into my small backpack. It's the best solution for me. It's kind of inconvenient when the phone is in my pocket as riding requires a constant footwork.
I would not recommend you to listen to music while riding because it can distract your focus. It can be dangerous. You can definitely listen to the music with open speakers.I just put it in a clip on case and put it on my side. That way I can listen to tunes while I ride.