Speaking in a language other than English is at most mildly irritating when it occurs in public. However, irritating is a personal thing, and is only a personal foible.
There is NO national language in the US. There is NO requirement to speak English in public. It does help to communicate with the rest of the citiznry, but it is not mandatory to only speak English.
My father would have been in that fellows face so fast... and then he would have given him a civics lesson.