(What you did was the polite/correct thing.)
It's a bad habit to answer it, in general - your opponent doesn't know what's being said, and you easily be coached that way. If I was a judge being called over, I'd tell you to turn it off, and call them back between rounds.
If you were expecting a call from...say your Mom, who you've been trying to reach all day for the ride home, tell your opponent before the match starts what you're expecting, and that you don't want to be impolite during the match if she calls...that's reasonable, right? And probably okay, but I'd bring it up before it happened. (It would not be polite at that point to answer just ANY call...the matches aren't that long.)