If I were a young woman, and I found out the fellow I was dating had slept with someone his brother slept with, I think that would be enough to call it quits.
If there is no real reason to be with him besides feelings, the break up may have been the best thing. This is just totally messed up. He wants to be friends with his friend and his family, and his brother has a connection with them. These people could be a part of your life and a constant source of temptation if you marry this guy.
Some people think soap operas are entertaining, but I'm pretty sure it won't be fun living in one.