LLMs as Information Warriors? Auditing Chatbots on Russia-Ukraine Disinformation
Full title: Auditing how LLM-powered chatbots tackle disinformation about Russia’s war in Ukraine
Authors: Mykola Makhortykh, Ani Baghumyan, Victoria Vziatysheva et al.
Published: 2024-09-16
arXiv: 2409.10697
Source: arXiv (cs.CY)
Abstract
The rise of large language models (LLMs) has a significant impact on information warfare. By facilitating the production of content related to disinformation and propaganda campaigns, LLMs can amplify different types of information operations and mislead online users. In our study, we empirically investigate how LLM-powered chatbots, developed by Google, Microsoft, and Perplexity, handle disinformation about Russia’s war in Ukraine and whether the chatbots’ ability to provide accurate information on the topic varies across languages and over time. Our findings indicate that while for some chatbots (Perplexity), there is a significant improvement in performance over time in several languages, for others (Gemini), the performance improves only in English but deteriorates in low-resource languages.
Why This Work Matters
This paper provides an empirical baseline audit of how major commercial LLM-powered chatbots perform on one of the most contested disinformation environments: the Russia-Ukraine war. The longitudinal and multilingual design captures both temporal drift (do models improve over time?) and the low-resource language vulnerability that the “Friend or Foe?” paper later theorized more formally.
The finding that performance improvements are language-dependent — Perplexity improving broadly, Gemini only in English — has direct implications for information environment governance in non-English-speaking populations that are primary targets of Russian IO.
Core Concepts and Contributions
Chatbot audit methodology: Systematic evaluation of Google Gemini, Microsoft Copilot, and Perplexity AI against a structured set of Russia-Ukraine disinformation claims. The audit design is replicable and establishes a methodological template for LLM IO-performance monitoring.
Temporal performance tracking: The longitudinal dimension is analytically significant — it captures whether safety alignment and knowledge updates improve chatbot resistance over time, or whether improvements are uneven across languages. This is operationally relevant for analysts relying on LLM tools in contested information environments.
Low-resource language vulnerability: Gemini’s performance deteriorated in low-resource languages while improving in English — confirming that English-centric training biases create structural gaps that can be exploited in multilingual IO environments. This extends the theoretical finding from the Kamińska & Klynina paper (Fine-Tuning Paradox) with empirical commercial-LLM evidence.
Dual-use implication: The same capability gap (LLM uncertainty in low-resource language disinformation contexts) that weakens defensive performance can be exploited offensively by state actors targeting non-English-speaking populations.
Connections
- Disinformation Campaign — the domain under study
- Russia-Ukraine War — the specific conflict whose disinformation ecosystem is audited
- Friend or Foe? Language as Ideological Switch in LLMs — theoretical companion; the Fine-Tuning Paradox formally explains the pattern this paper observes empirically
- Cognitive Warfare: Definition, Framework, and Case Study — contextual framework
- Russian Federation — primary IO actor whose narratives are tested