The Institute of the Estonian Language conducted a stress test for generative AI models, revealing concerning weaknesses in the models developed by Mistral, a leading open-source AI firm in Europe. All four of Mistral's models performed poorly, ranking in the bottom third of the evaluation, with the top model achieving only a 47th position out of 60.
No model was able to surpass a 40% rate of efficacy when it came to recognizing or rejecting manipulative content associated with Russian propaganda. This is particularly troubling given Mistral's aspirations to establish itself as a cornerstone of European sovereign AI amid negotiations for a significant funding round valued at €3 billion.
#What Did the Benchmark Measure?
The EKI Propaganda Resistance benchmark focused on evaluating the AI models against 14 distinct Russian propaganda narratives. The assessment utilized 75 questions across English, Russian, and Estonian languages, with scoring managed by a calibrated Claude model from Anthropic, which rated as one of the top performers.
Interestingly, prompts in the Russian language were notably effective at eliciting responses that aligned with propaganda, showcasing weaknesses in less capable models. Previous assessments, such as those conducted by NewsGuard, showed similar issues with Mistral's Le Chat chatbot, which reiterated state-sponsored falsehoods between 50% and 57% of the time on numerous topics. The outcome of this recent testing only reinforces the existing concerns surrounding the models’ resistance to disinformation.
#What Are the Implications for Investors?
Investors may need to reassess their trust in Mistral, especially as the company is attempting to raise €3 billion at a valuation of €20 billion. There are increasing pressures on European AI technologies regarding transparency and risk management due to the EU AI Act. These recent findings regarding Mistral's struggle against disinformation could complicate its funding narrative, warranting close scrutiny from potential backers who are mindful of these emerging challenges in the landscape of AI technologies.