March 4, 2020
Between December 2019 and January 2020, a group of journalists from ANSA, Civio & VRT, Fandango’s consortium partners and end-users of the platform, tested the first functional iteration of Fandango’s platfom.
The pilot was executed on three domains: ANSA, the leading wire service in Italy, focused on the use case concerning European Context; Civio, an small non-profit newsroom in Spain specilized in investigative reporting, dealt with the use cases regarding Immigration; and VRT, the national public-service broadcaster for the Flemish Community of Belgium, was in charge of the the Climate use case.
We chose a two-phased approach. During Phase 1, professional journalists and journalism students collected a set of metrics (KPI’s) without the use of the Fandango services, whereas Phase 2 was evaluation stage for the Fandango application, and the same metrics were collected with the help of the platform for comparison. The main goal was to acquire quantifiable feedback, evaluate applicability of the solution in real scenarios and assess the impact of Fandango in the process.
Each partner provided at least three different people and a total of 64 hours of effort, broken down into 5 different real-world scenarios were journalists would use the tool to:
- Verify a claim
- Verify a video
- Verify an article
- Verify an image
Let’s take article verification for example. A journalist investigating public health systems might need to verify an article claiming that immigrants are congesting it and that they use the sanitary facilities more than people born in Spain. She would visit Fandango’s site, enter the article URL in the search box and press “Verify article”. In this scenario, Fandango would return analysis from diverse modules: its Open data dashboard, Article verification, Text similarity and Authors analysis. Fandango would provide a set of sources to assist the journalist in her verification process: a selection of similar and already verified claims and articles, a Scoring of the author regarding their trustworthiness based on graph analytics and a list of public databases on immigration and health. This information would enable her to take better informed decisions whether the article is verified or should be considered disinformation.
Both without Fandango and with Fandango, the users gathered a set of KPIs related to the following areas of impact:
- Number of Verifications
After the first pilot iteration, results were promising for Fandango as support tool during the news verification process. Nevertheless users suggested usability improvements and better clarifications to understand better how scores of text linguistic characteristics and image and video analysis are calculated. Also a broader time frame for pre-ingested articles, claims and open data, so that information returned by the platform would be more relevant for the user.
This feedback and the pilot results are the basis under which project partners are currently improving the quality of the implementation of Fandango ahead of the second pilot iteration, in which disinformation around the COVID-19 pandemic will be added as a new scenario.
For more information check out our Pilot Execution and Evaluation Plan