My standard is to say something like "I don't have any cash to share, good luck to you though." I work in homeless services and know a ton of folks who survive on panhandling. In my area, people have no problem finding food, there are a ton of social service orgs and churches that provide food daily.
The following is not a judgment and is a generalization that is far from universal. This is just a description of what I commonly observe. The unfortunate reality is that much of the money people get from panhandling goes to purchasing cigarettes, alcohol, meth, K2, crack, and/or fentanyl. For this reason I avoid giving people money directly.
You don't need to explain yourself to anyone, and if you feel bad for not giving someone money because you feel the inequality, consider donating your time or money to organizations in your area that are doing the work to help people gain employment or housing, meet their basic needs, or treat their physical or mental health needs.