The trouble is that enough panhandlers are not actually poor downtrodden homeless that it affects how the others are viewed.
For example, my city is a generally poor area. Businesses closing, unemployment high. I am very sure there are MANY genuinely homeless folks out there. However, there are two very very prominent panhandlers who patently are trying to take advantage of tourists, and so that colors my perception. The one has a sign explaining that as a result of a house fire he lost everything. Might have even once been true...but he's been using that sign for over four years. It's even been tacked onto a wooden backing. And if someone brings him inside to buy him lunch(I work at a Mcdonalds) he does his level best to get them to just give cash once in line or failing that, buy a gift card. If he is forced to be bought food, he waits until they leave and then asks for his money back. I've seen his wallet(he buys a cup of coffee in the morning) and he has PLENTY of money. Telling people who wish to give him money that he isn't one of needy typically falls on deaf ears. It took police intervention to get him to stop harassing my drive thru line.
The second one works the 'late shift' after the first one leaves. He pulls up in a nice car, parks it at the grocery store nearby and then sets up shop by the stoplight. He uses different signs. Some have him as a veteran, some kicked out by wife, some that he wants a home, just random. Same guy though, and when he came in I asked which bases he'd been stationed at, since I was in the Air Force. "I think I passed trhough one a few years ago," he said.
So these two have made me unbelievably doubting.
There have been similar high profile cases around where I live. But there's a distinct difference between that kind of person, and the clearly distressed dude wandering around by the food trucks multiple days asking people if they had any food, or the guys near me who sleep under bridges and have sometimes multiple poorly cared for prosthetics. The slick guys giving you a line are easy to spot in my experience. In large part, they're the ones with lines. The genuine homeless and needy I've met (not always the same, New Orleans for example has a high percentage of homeowners in extreme poverty) don't have stories to tell people about tragic luck. They just ask for help.