Absolutely. I played and managed in a amateur team in the nibneties, here in northern Germany, and our team had a lot of international students, from Morocco, Argentina, Korea, Uganda, all over the world. We heard a LOT of nasty shit, but we made a popint of telling the captains of the opposing team and the referees, that we would walk off the field, if this didn't stop. We gave up some games like that, some referees told us, that is a three goal plus three point win for the other team, but eventually, this made a change.
I think, the players have to take the initiative, make it clear up front, that they won't come back out off the cabin for the second half, if such things happen