How to Distinguish Between American English and Taiwanese Mandarin

In Singapore, people always get quite confused because they hear me speak English with an American accent. They might hear me speak Chinese with a Taiwanese accent. So more than once people in the classroom or on taxis or in public spaces in Singapore, they hear my wife and I switching between American English and Taiwanese Mandarin. And at some point they throw up their hands and say, what are you?