My biggest problem with sending food back is that now you have to wait while everyone else at your table eats. Then when your food arrives, everyone needs to wait for you while you eat.
This usually isn’t too much of an issue if your table has a lot of people because you can just chat and the length of time to eat a meal is higher since you all are talking. But when you’re on a date, it’s super annoying and awkward.