The biggest barrier, beyond the translation itself, is speech recognition. Moreover, in a recent test he took part in, the system was 'near-perfect' with certain language combinations (English to Portuguese is specifically cited).
It sounds like Google's Babel fish-esque instant translation solution is making progress - Android VP Hugo Barra told The UK Times that Google's got hardware prototypes ( in the form of mobile phones) already working.
