36 years old here and still watching anime. Started when I was three!
Don't feel bad regarding anime, or animation in general, my friend. Remember: Animation are NOT cartoons. It's rather the other way around. Try watching Akira, or Ghost in the Shell, or anything from director Satoshi Kon and you'll get my meaning.
Bow, about feeling lonely: I'm also sort of a loner, but being alone gets kinda depressing to me. Some advice? Try getting in some sort of extra curricular activity that you like: A sport (I practice martial arts, myself), an art course, extracurricular school or college programs, heck, a cooking course, anything you like! You will almost invariably meet some people with similar tastes as you.
And remember: Nobody is perfect. NOBODY. So a little tolerance and, yes, a smile, will get you a long way. Variety is what keeps things interesting, and respect is what keeps us together as a society.