Speaker Diarization Challenge

IBERSPEECH-RTVE SPEAKER DIARIZATION EVALUATION

Alfonso Ortega, Ignacio Viñals, Antonio Miguel, Eduardo Lleida (UZ)
Virginia Bazán, Carmen Pérez, Alberto de Prada (RTVE)

Description

The IBERSPEECH-RTVE SPEAKER DIARIZATION is a new challenge in the ALBAYZIN evaluation series. The evaluation is supported by the Spanish Thematic Network on Speech Technology (RTTH) and Cátedra RTVE Univesidad de Zaragoza and is organized by Vivolab – Universidad de Zaragoza.

The Speaker Diarization evaluation consists of segmenting broadcast audio documents according to different speakers and linking those segments which originate from the same speaker. For this evaluation, the evaluation database has been donated by RTVE and labelled thanks to the Spanish Thematic Network on Speech Technology (RTTH) and Cátedra RTVE de la Universidad de Zaragoza. Around sixteen hours of two different TV shows will be used for development and another sixteen hours from another two different shows will be used for testing. For training, the Catalan broadcast news database from the 3/24 TV channel proposed for the 2010 Albayzin Audio Segmentation Evaluation [1-3] and the Corporación Aragonesa de Radio y Televisión (CARTV) database proposed for the 2016 Albayzin Speaker Diarization evaluation will be provided [4].

No a priori knowledge is provided about the number or the identity of the speakers participating in the audio to be analyzed. In the provided training data, information regarding the presence of noise, music and speech will be annotated but not in the development or test partitions. The Diarization Error Rate will be used as scoring metric as defined in the RT evaluations organized by NIST [5]. Two different conditions are proposed this year, a closed-set condition in which only data provided within this Albayzin evaluation can be used for training and an open-set condition in which external data can be used for training as long as they are publicly accessible to everyone (not necessarily free). Participants can submit systems in one or both conditions in an independent way.

The RTVE data is available to the evaluation participants only and subject to the terms of a licence agreement with the RTVE. The license agreement can be downloaded from Cátedra RTVE-UZ web page (http://catedrartve.unizar.es)

More details will be given in the evaluation plan.

Schedule

  • June 18, 2018: Release of the evaluation plan, training and development data
  • July 15, 2018: Registration deadline
  • September 24, 2018: Release of evaluation data
  • October 21, 2018: Deadline for the submission of system outputs and description papers
  • October 31, 2018: Results distributed to participants
  • November 21-23, 2018: Evaluation Workshop at Iberspeech 2018
  • Registration

Interested groups must register for the evaluation before July 15th 2018, by contacting the organizing team at ortega@unizar.es with CC to ALBAYZIN 2018 Evaluations Organising Committee. The contact should contain the following information:

  • Research group (name and acronym)
  • Institution (university, research center, company, …)
  • Contact person (name)
  • Email

To download the RTVE data, you will need to sign this data license and return it to the IberSpeech-RTVE team.