Harry rethinks family UK trip