Large Language Models (LLMs) offer transformative potential across diverse settings, enhancing quality and user experience. At KUDO, we've pioneered the integration of LLMs into our AI Simultaneous Speech-to-Speech Translation system, marking a significant milestone as the first genuine application of Generative AI in a practical speech translation tool. This enhancement led to a notable improvement in translation quality, increasing accuracy and fluency by approximately 30 percentage points. However, implementing LLMs isn't without its challenges, including the need to mitigate model hallucinations and unanticipated behaviors. Such complexities necessitate a rigorous integration and testing methodology to ensure translation quality aligns with expectations, with the application on speech data adding another layer of complexity.

This presentation delves into the intricacies of our LLM integration process, shedding light on the integration's objectives and our testing methodology tailored for speech translation. We've crafted an innovative yet straightforward evaluation method for speech translation, yielding dependable insights into tool quality. Grounded in human assessment, our methodology balances cost-effectiveness with reliability of results. Furthermore, our exploration into automatic evaluations, leveraging LLMs, has revealed a certain degree of alignment with human evaluations, which might open up the possibility to increase testing at scale. Emphasis will be placed on the challenges and constraints inherent in assessing speech translation. LSPs seeking to independently and knowledgeably evaluate speech translation vendors may find this presentation particularly insightful.

Host organization: Globalization and Localization Association