Of course you are being disrespected. They intentionally removed the survey flag. What were they thinking - that you would not notice? I'm sure they knew they could trigger a confrontation just by removing the flag, but you did not react, so they sort of won round one. You should have the corner staked again, this time with a permanent stake that has an official surveyors license number on it and take a picture of it. At some point you have to be direct and ask them not to encroach. If they remain disrespectful and continue to encroach, why then would you care about risking any kind of relationship? Like most have said above, a fence is your right and makes a statement even if you don't want to confront them. Even if they come over and confront you, you can simply point to the surveyors stake.
Q