how do you tell him to back off

I agree with Random.
Also, if you're not comfortable telling him on face to face, next time he rings you, let him know that way.
Just say that you won't be able to see him again, it's not for you. You can also then make your excuse to get off the phone, so as not to be held up talking when no point.

Hope you get there with this one.