"Well we'll see, won't we" feels like the appropriate thing for you to say to him if the power dynamics were reversed — if you were the one in behind and he was possibly trying to coyly ask where you lived (and you wanted to coyly not respond). Makes no sense in this context — either he is a creep, or he was totally reading the wrong thing into your question. Or both.