It's much easier to be prejudiced against someone you don't know personally.

And it's funny, because a lot of the people I've been in the world with, so to speak, do come from some of the big organized religions that require, if they were to practice strictly, that they think of me as lost, inferior, etc. either because I wasn't born into X religion or because I've rejected the one I was born into.

That's never bothered me. If any of my friends/classmates/coworkers/in-laws actually have thought that (or, more likely, felt it, because those kinds of knee-jerk reactions do come with the tradition in a lot of cases), they keep it private, or it's just not relevant, or they get over it.