Hmmmm.... opinions pleeeaaaeee?

You have answered your own question. You would rather spend lunch with your girlfriends and you are unwilling to give up your activities for him. It sounds like you don't want to spend time with him at all. Tell him this is my life, I'm not changing it, if you can live with that great, if not bye.