AI Helps Document the Impact of Attacks on Civilians in Ukraine

Artificial intelligence and machine learning help identify Telegram posts about harm to civilians in Ukraine more quickly. This approach accelerates OSINT investigations, information verification, and the documentation of the impact of Russian attacks.

Why This Research Matters

Between February 2022 and September 2025, researchers and volunteers collected, geolocated, and documented more than 2,500 incidents involving civilian casualties following Russia’s full-scale invasion of Ukraine.

As part of this work, they tested a new machine learning model that ranks Telegram posts based on the likelihood that they contain information about harm to civilians.

This approach significantly reduced the time needed to find and filter relevant posts, allowing researchers to focus more on verifying and documenting incidents rather than searching for them.

The study explains the methodology behind the model, discusses the ethical considerations of using this approach, and presents findings that may be valuable for researchers, journalists, and other professionals working in similar fields.

Research into harm to civilians using open-source intelligence (OSINT) is still a relatively new field. One of its biggest challenges is processing vast amounts of user-generated content quickly enough to identify genuinely important reports.

This is where machine learning can make a significant difference. By analyzing large datasets, identifying patterns, and automatically prioritizing the most relevant content, it helps researchers work far more efficiently.

Given the ongoing conflicts causing significant civilian harm in Ukraine, Sudan, and the Middle East, this methodology could serve as a practical example of how machine learning can help identify and prioritize reports related to the consequences of armed conflict.

In this context, the term “harm to civilians” extends beyond deaths and injuries. It also includes psychological trauma, forced displacement, loss of livelihoods, damage to civilian infrastructure, and other long-term consequences of war.

The Initial Telegram Dataset

Every Telegram post containing a verified case of harm to civilians and confirmed by researchers was included in the initial dataset. In machine learning, these are known as positive examples. In total, the dataset contained 5,848 unique Telegram post URLs.

During the manual data collection process, researchers reviewed posts from relevant Telegram channels in chronological order, from the oldest to the newest. Once a post was confirmed as containing information about harm to civilians, the posts immediately before and after it were assumed to have already been reviewed and found not to contain relevant incidents.

For each confirmed case, researchers also selected ten neighboring posts as examples without signs of harm to civilians. After removing duplicates and unavailable posts, this produced a dataset of 48,545 posts classified as negative examples.

The large number of negative examples intentionally reflects real-world conditions, where reports of harm to civilians make up only a small fraction of all available content. It also helps improve the quality of the machine learning model during training.

Metadata was also collected for every URL through the Telegram API, including publication time, message text, reactions, and other available information. If individual posts had been deleted, missing data was restored from previously archived copies whenever possible.

Feature Engineering

Machine learning models require numerical data because all predictions are based on mathematical calculations.

To create this data, information from the original dataset—such as keywords that may indicate harm to civilians—was converted into numerical values known as features. These features allow the model to identify patterns and classify Telegram posts.

This process is called feature engineering. It improves model accuracy by combining mathematical analysis with the practical expertise of researchers familiar with the subject.

The complete list of features used to train the model is available in the code published alongside the study. Many of these features were developed from the researchers’ own experience manually reviewing Telegram channels, verifying individual posts, and identifying reports related to harm to civilians.

Features Derived from Telegram Metadata

Some features were extracted directly from the metadata of each Telegram post. These included the type of media attached, the day of the week, and whether the post had been forwarded, edited, or published as a reply to another message.

The model also took user engagement into account by analyzing the number of views, forwards, reactions, and even the use of specific emojis. For example, it separately measured how often particular reactions appeared beneath individual posts.

Converting Text into Numbers

To incorporate knowledge gained from manually reviewing Telegram channels, the researchers compiled a list of Ukrainian and Russian keywords that may indicate reports of harm to civilians. These included terms such as “Shahed” and “KAB.” For each post, the model counted how frequently these keywords appeared in the text.

The researchers also used a set of English key phrases related to the consequences of war, such as “injured,” “school hit,” and “hospital hit.” Rather than serving as simple search terms, these phrases were used to measure how closely the meaning of a post matched known descriptions of civilian harm.

To achieve this, the researchers applied a semantic similarity method that measures how closely words or entire phrases are related in meaning.

First, the text of each Telegram post and the selected keywords were converted into numerical representations, known as vectors, using a natural language processing model. The cosine similarity between these vectors was then calculated, one of the most widely used methods for measuring the similarity between two pieces of text. The higher the similarity score, the more likely the post was to relate to harm to civilians.

Since the model operates on vector representations of text, semantic similarity is measured on a scale from -1 to 1. Scores close to 1 indicate that words or phrases have similar meanings, while values close to -1 suggest little or no semantic relationship.

For example, the words “hurt” and “injured” would produce a high similarity score because they convey closely related meanings. In contrast, “residential” and “injured” would receive a low or even negative score because they are not semantically related.

To estimate how closely each post relates to harm to civilians in Ukraine, the entire text of every Telegram message was converted into a 768-dimensional vector using a multilingual BERT-based language model. This approach enables the algorithm to work across multiple languages and understand the meaning of a text rather than simply matching individual words. As a result, the same sentence written in Ukrainian, English, or another language produces very similar vector representations.

At the same time, the first version of the model analyzed only textual data. Features extracted from images or videos were not included, although incorporating them in future versions could significantly improve the system’s accuracy.

Model Selection, Training, and Evaluation

After preparing a dataset containing 54,393 records and 893 numerical features for each one, the researchers selected four machine learning algorithms to train their models.

Logistic regression was chosen as the baseline model because it is one of the simplest and most interpretable classification algorithms. In addition, three more advanced models were evaluated: Random Forest, XGBoost and LightGBM.

These algorithms were selected for two main reasons. First, they perform well on tabular datasets of this size. Second, they make it possible to identify which features contribute most to the model’s predictions. Neural networks were not used because they are considerably more difficult to interpret and generally achieve their best performance on much larger datasets.

To evaluate the models objectively, the dataset was divided into three subsets:

• Training set — 60% of the data, used to train the models.

• Validation set — 20% of the data, used for intermediate evaluation and hyperparameter tuning.

• Test set — the remaining 20% of the data, which the models never saw during training and which was used for the final evaluation.

Rather than splitting the data randomly, the researchers used stratified sampling. This ensured that the proportion of positive and negative examples remained consistent across all three subsets. In this case, confirmed reports of harm to civilians accounted for approximately 11% of each dataset.

During training, the models were optimized to maximize the PR-AUC (Average Precision) score, a metric that summarizes precision across all levels of recall. Although PR-AUC incorporates both precision and recall, it places greater emphasis on recall. For this study, that makes it a more appropriate metric because it helps minimize the number of reports of harm to civilians that the model fails to identify.

The table below ranks the models from best to worst based on their PR-AUC scores and compares their performance with that of a random baseline classifier. It also includes two additional evaluation metrics: ROC-AUC and F1 score.

• ROC-AUC measures how well a model distinguishes between positive and negative examples.

• The F1 score combines precision and recall into a single metric and is calculated using the optimal classification threshold.

Based on the evaluation results, XGBoost was selected as the final model because it consistently achieved the best performance across all of the primary evaluation metrics.

Model Interpretation

Because these models are interpretable, it is possible to identify which features contributed most to predicting whether a post contains information about harm to civilians. The table below shows the ten most important features used by the XGBoost model when making its predictions.

• semantic_keywords_similarity — the semantic similarity between the post’s text and a manually curated set of keywords related to casualties, damage, and harm to civilians.
• BERT — semantic text embeddings that enable the model to capture the meaning of a message. Notably, three BERT-derived features ranked among the model’s ten most important features.
• reaction_crying_face — the number of crying face emoji reactions a post received.
• group_of_messages — an indicator showing whether a post contains multiple media files.
• keywords_in_text — the number of Ukrainian or Russian keywords related to harm to civilians found in the text.

Overall, these findings are consistent with expectations. Posts describing harm to civilians were more likely to contain characteristic keywords and to elicit strong emotional reactions from readers. However, not all models identified the same features as being the most important. For example, Random Forest ranked the number of crying face emoji reactions as its single most influential feature—a pattern that the researchers had already observed during the development of the methodology.

Results and LLM Comparison

The table above shows that LLMs are indeed capable of using the engineered features to improve their predictions. All four models outperformed the baseline logistic regression model, but none surpassed the performance of the purpose-built machine learning models. XGBoost continued to achieve the highest PR-AUC score.

Interestingly, Gemini 2.5 Flash outperformed the newer Gemini 3.5 Flash and even achieved a slightly higher maximum F1 score than any other model. Nevertheless, PR-AUC remains the most important metric for identifying reports of harm to civilians, as it best reflects a model’s ability to detect rare positive cases while minimizing false positives.

Ethical Considerations

Using an automated system to identify reports of harm to civilians inevitably raises ethical concerns. These include automation bias, where people may place excessive trust in an algorithm’s recommendations, and algorithmic bias, where a model reproduces patterns present in its training data, potentially underrepresenting or overrepresenting certain types of harm.

The decision to develop an automated methodology was driven by limited resources for both identifying potential incidents of harm to civilians and verifying them afterward. Over time, a large backlog of unverified incidents had accumulated because researchers spent much of their time monitoring events in real time to ensure that potential evidence was collected and preserved as quickly as possible.

Automating this process also reduced researchers’ exposure to large volumes of distressing textual and visual material, helping to lower the risk associated with prolonged exposure to traumatic content.

To mitigate ethical risks, the project incorporated several safeguards. Researchers randomly reviewed posts that had not been selected by any model, analyzed the features that influenced the models’ decisions, and compared historical patterns within the data to identify potential biases.

In addition, the initial version of the model did not use features extracted directly from images or videos. Although incorporating visual content could improve predictive performance, applying AI to analyze such material introduces additional ethical challenges, including the risk of model bias.

The use of LLMs raises its own ethical concerns. Because many of the companies developing these models have opaque ownership structures, and because LLMs are inherently generative, their use presents potential privacy and data security risks, particularly when sensitive information is involved. For this reason, the primary production model did not rely on LLMs, and they were evaluated only retrospectively.

How the Model Fits into the Overall Workflow

Once the final model had been selected, the researchers built an interface that allowed investigators to review Telegram posts ranked by their estimated likelihood of containing information about harm to civilians. The interface was designed for rapid triage and seamless integration with other tools. When a researcher confirmed a post, it was automatically sent to Auto Archiver, Bellingcat’s tool for preserving digital content, and then forwarded to ATLOS, the organization’s internal collaborative verification platform. From there, Bellingcat staff and volunteers could manually verify each incident. Every decision made by researchers was recorded so that it could later be used to further improve the model.

Initial results showed that the model was effective in practice. It not only reduced the time required to review dozens of Telegram channels containing wartime reports but also helped reduce researchers’ workload. At the same time, the posts prioritized for review consistently contained a wide range of genuine reports of harm to civilians.

The authors note that the model is still under active development and has considerable room for improvement. Despite its promising performance, further testing, feature engineering, and continuous evaluation will be necessary before it can be deployed more broadly.

Although the system was designed to identify reports of harm to civilians on Telegram—a platform widely used in both Ukraine and Russia—its architecture is general enough to be adapted for other conflict-monitoring tasks. How easily this can be achieved depends largely on the openness of the target social media platform and the ability to access its posts. The system also makes it straightforward to incorporate new features and data sources while automatically retraining, testing, and deploying updated models as additional verified data become available.

As armed conflicts continue to generate vast amounts of information, analyzing that data at scale will remain a significant challenge. The authors hope that the proposed methodology will help newsrooms, conflict-monitoring organizations, and other investigative teams more effectively balance ethical considerations with available resources when conducting OSINT investigations related to harm to civilians and human rights violations.

This article is based on information published by Bellingcat.