You didn't start your business with dreams of working all the time doing stuff you hate and answering to a ton of bosses, better known as clients, who don't understand what you do but want to tell you how to do it anyway. (whew – run-on sentence!)
Unfortunately, most people who start a business end up in exactly that position, especially if it's a service business. I'll bet you have, too. It's just so easy to get stuck in a rut that it's hard to take a step back and think things through.
Nobody else has the knowledge and experience that you do, though. They may have similar, but they'll never have exactly the same. That's what makes you so valuable to the clients who are the best fit for you to work with and not valuable to those who aren't. And that's great! You only want to work with the ones who fit, right? Unless you're a masochist, I already know the answer is yes.