Good question

So yesterday I was taking a final. When I finished my test, I pulled out my phone and saw about ten missed alerts. Some of them were expected, and most of them were from people I know. However, there were two missed calls from the same number, which I didn't know. I decided that I should probably call them back, so I did. Here is the ensuing conversation.

Grumpy Man (GM): Hello?
Me: Hi, I saw that you had called me a few times, so I'm calling back.
GM: Who is this?
M: Rachel Bohman.
GM: I don't know any Rachel Bohman. Why are you calling?
M: I saw that you had called me twice in the last two hours, so I'm calling you back.
GM: This is a private residence. How did you get this number?
M: Your number was in my phone. I had two missed calls from you.
GM: Don't call this number.

Was he not listening? Was I insufficiently clear? Why?