I think it's important to recognize the limitations of our Humanity. Not being able to disconnect from full-time ministry every once in a while, reconnect with family, spouses, and kids, and simply enjoy the fruit of your hard labor can cause distress, depression, burnout, and so much more.
Remember, God rested too.
I definitely think you should consider making your vacation time a priority when you have conversations with those in leadership. It's a tough conversation, but I would approach it in a way that amplifies the fruit of a good rest, and how mismanaged vacation time can deter that fruit.
Praying for your situation!