Littering your yard with food attracts things like rats, raccoons, squirrels, etc, which destroy property and infrastructure, spread disease, and cause injury to people and pets. I'm not saying I'm against fruit trees, but I do understand people who are. It's a legitimate concern. Some areas even have things like boars or bears which are extremely dangerous.
I'm also curious with the way you can sue people in the US what would happen if someone becomes sick after eating one of your fruits. I imagine it varies by state.