Have you talked to the guy about the distress hes causing you?
If you haven't, do so. Maybe he doesn't know hes hurting you - when women are sad and upset men will distance themselves and automatically go to more cheerful girls [or their friends.]
If you talk to him, make sure not to start a fight. You want him to understand your point and view and let him know this is hurting you. Heres a good starter, my favorite I statements:
"I feel [feeling] when [event] because [this happened] and I want [whatever.]"
Heres an example:
"I feel [angry] when [you make me do the dishes everyday] because [you said I could have a break] and I want [to work on my homework.]"
You see, its much better than flat out accusing the person. I statements will help this person understand your perspective and be able to put themselves in your shoes.
-------------------
However, if you have in fact talked to this guy and he has continued with his ways; there are two options in my opinion.
1) Try again with talking to him and make sure he knows
2) Leave him
The second option, as you said, is going to be very hard. If it comes to this at some point, it can become easier by spending less and less time with this guy. That way, you can get him out of your life if thats wanted.
Example:
Monday - Spend 4 hrs together
Tuesday - Spend 3 hrs together
Wednesday - Spend 2 hrs together
Or it can be in halves. It can also be flexible, but remember to keep the goal in check!
Best of luck, I hope I helped you a little. < 3