"higher pressure = faster?", no.
higher pressure = harsher ride?, yes
At your total bike + rider weight of <165lbs, you could lower your pressure a bit without increasing rolling resistance or risking pinch flats. The "plus factor" will be a less harsh, more comfortable ride.
BTW, the "max inflation presssure" spec on clincher tires is ~50% of the pressure it takes to blow the tire off an appropriate rim. Despite what you may read in manufacturer's propaganda or be told at your LBS, it _is not_ an indication of tire quality, per se.