If the drone does not cross the property line, it's not much different from being in a tree, a camera in a tree or even just being on your roof or looking out your upstairs window. Are any of those illegal o even wrong?
If someone wants to see what is going on, they can hire a helicopter and just fly. I believe they have to be 500 feet up, but with modern cameras that's no limitation. So what is new about drones?