Don't fit in with my boyfriend's family and (some) friends?

My boyfriend and I have been together for 3.5 years. He is a single dad and 13 years older than me. His sisters and "the gang" that grew up together back home are all about 10 years older than me. They all are married with kids (except for one). I feel that I don't fit in with them because sometimes they speak/joke around in their own language, or they talk about memories from when they were growing up. Or they talk about kids and marriage. My bf and I have gone through a lot because it was hard for me to adjust to a relationship with kids and him having little time for a relationship. We used to live together, but it was too difficult for me so I moved out after a year and slowly started disengaging in certain areas and to focus on us instead of everything. I feel that one sister has a problem with me based on some things her brother might have said to her (although she has never said anything to me but I can tell). His parents are great. I just feel that whenever they all get together they have a blast and are drinking and joking around. I don't drink, I'm a little shy sometimes and either I don't understand their jokes or I don't know the stories that they are referring to. When his family comes over, it's almost like he forgets me and has a lot of fun with them. He is always busy with everyone else and doesn't even look at me or talk to me. I'm not an insecure person, I'm totally fine with myself and the way I am. I know there is nothing wrong with me (I don't have socialization issues). I just feel that I don't fit in there and I don't feel like my bf and I are a couple when his kids, family and/or friends are around. I'm tired of not fitting in or feeling like I'm not in a relationship. I want to throw in the towel but maybe it won't take much to turn things around?