This is the abstract submitted to this conference which takes place in November.
Unlike most technical and scientific conferences there are no papers required, only an abstract. However as the word limit was quite generous, this abstract turned into a short paper.
All the figures are monochrome as required by the conference.
Automated Monitoring of UK SDH Quality – a research tool exploring methods and results
Following retirement from BBC R&D I began to watch more television with the subtitles turned on. These were UK TV public service broadcast programmes in English with English language subtitles. I began to notice some quite severe delays, particularly in news programmes, especially on pre-scripted material where subtitles were being keyed out rather than respoken. This was a technique developed to reduce subtitle delay, but it was now sometimes having the opposite effect. Along with delay in live subtitles there were also archive programmes broadcast with the original subtitles where the editing was so severe that it was easier to watch the programme without them. This was because the original UK guidelines for subtitling specified a maximum word rate of only 120 words per minute [1, 2].
In order to persuade the broadcasters to improve their subtitles I needed clear evidence of these problems. This meant analysing many days of TV broadcasts to identify and characterise the issues. My focus was on delay and word loss because these were the most obvious problems and they are also the issues which are regularly highlighted in audience surveys [3, 4].
Amongst my previous work on subtitles [5, 6], I discovered a clear, linear relationship between subtitle delay and perceived subtitle quality and established that subtitles are effectively useless when delayed by ten seconds or more [7]. This led to changes to the systems for delivering live subtitles at the BBC, using pre-prepared subtitle blocks for scripted speech and pre-recorded packages along with reduced encoding delays for subtitles generated by respeaking [8].
Around the same time Ofcom, the UK’s communications regulator, in partnership with the University of Roehampton, carried out an exercise in monitoring live subtitle quality. This involved both broadcasters and subtitling students manually assessing the quality of 10-minute clips of live subtitles once every 6 months over a period of two years [9]. As a response to the sparse sampling and high costs, we developed a prototype tool capable of continuously monitoring various aspects of DSAT Teletext subtitles [10]. In a later piece of work, I utilised a speech to text engine to enable the recovery of subtitles for video clips on the BBC web videos that had been clipped from broadcast TV programmes [11].
In order to collect evidence of the problems with subtitle delay and word loss I combined the approach of off-air monitoring of DSAT teletext subtitles with the use of a speech to text engine [12, 13, 14]. The major challenge was aligning the words in the subtitles with the automatic transcript, especially where there was a significant degree of word loss and/or delay. These problems were caused by:
- Subtitles containing many types of non-speech information
- The timing in the subtitles and transcript not matching.
- The subtitles omitting many of the spoken words.
- Word errors in both the subtitles and the transcript.
- Spelling differences between the subtitles and the transcript.
- Compound words and contractions vs separate words.
- Numerals vs words for numbers.
- Words in the subtitles being in the wrong order.
- Long sections of subtitles being repeated.
- Subtitles getting stuck.
- Live subtitles containing corrections – in varying formats.
- Non-speech utterances, especially in children’s TV.
- Speech or singing containing lots of repetition.
In order to get round most of these issues, the software uses a series of stages which start by separating out the non-speech elements from the subtitles and then attempts to match long strings of words. It then iterates over the process, matching shorter and shorter strings of words to fill the gaps. A final process then deals with variations in spelling, numerals, compound words and contractions while ignoring the 20 most common words as these often cause false matches [12, 13, 14].
Once the subtitles have been aligned with the transcript, the “start” time for each subtitle can be compared with the “begin” time for the first word of the subtitle in the transcript to give a measure of the subtitle delay. A scatter plot of delay against the start time of the subtitle is then created (figure 1). Note that not all subtitles can be aligned, but those that can give a good indication of the general trend. The average subtitle delay for each minute is also rendered as a line plot which can give a clearer indication of general trends (figure 2).
Figure 1
Figure 2
A further plot shows the number of words in each minute for the subtitles and the transcript. This can reveal changing patterns in the number of words missing from the subtitles (figure 3) which mostly occurs with live programmes and archive content. Figures 1, 2 & 3 all show the same 1 hour recording of a TV channel with commercial breaks. The first 38 minutes is a live programme which is followed by a pre-recorded programme.
Figure 3
It soon became apparent from early results (late 2024) that subtitle delays of 30 seconds or more were commonplace and caused by many different factors. Figure 4 is a plot of delay on Sky News where the subtitles pause then restart over a minute late. The system also captured examples of subtitles turning up early. Figure 5 is an example from BBC News where the subtitles are keyed out too fast to the point where they are nearly sixty seconds early then stop for two minutes, resuming over a minute late.
Figure 4
Figure 5
Figure 6 is a plot of word rates from daytime ITV1 where, for the first five minutes, over half the spoken words are missing from the subtitles because the subtitler is trying to cope with often rapid speech and multiple voices all at the same time. Figure 7 is a prerecorded programme from BBC One which had been edited without the subtitles being changed, resulting in the subtitles being nearly 4 seconds late for the last few minutes of the programme. Figure 8 is a plot of word rate for an episode of the programme Red Dwarf from 1988 which has recently been rebroadcast on the BBC complete with its original subtitles. It shows how the subtitles have been edited to keep the word rate under 120 words per minute, resulting in word loss of over 20%. These plots along with many others have been shared with the broadcasters concerned.
Figure 6
Figure 7
Figure 8
To reduce the time spent looking through one-hour plots I began creating 24 hour plots of delay, averaged over each minute, and word rates averaged over 5 minutes. Figure 9 shows a remarkable change in subtitle delay on BBC News which took place at 2pm on 13th November 2025. This plot shows how the introduction of Automatic Speech Recognition (ASR) for the BBC News channel resulted in a very consistent delay of around 4 seconds, rather than the varying delay in the hours before. This was initially a trial, but has continued in use. However, the delay has now increased to 7 seconds.
Figure 9
A further enhancement to the 24 hour plots was the addition of Electronic Programme Guide (EPG) data. Figure 10 is a plot of subtitle delay for BBC One on the 25th August 2026. The EPG data gives a clear indication of the way subtitle delay varies between live and prerecorded programmes. It also shows how the subtitle delay is now being kept below 10 seconds except for one problem during a news bulletin.
Figure 10
Figure 11 is a word rate plot for ITV1 on the 17th September 2025. Apart from the early morning programming which does not contain subtitles, the most noticeable issue is the large difference in word rate between the subtitles and transcript between 09:00 and 13:30. This is because the magazine style of programming contains many different voices often talking rapidly and sometimes over each other. Similar programming on BBC One shows the same pattern. This level of word loss on magazine programming is a significant problem which would benefit from new research.
Figure 11
This work was conceived as an exercise in demonstrating automated subtitle monitoring and providing feedback to broadcasters on the issues of delay and word loss. While the accuracy of the measurements tends to decline as the quality of the subtitles decline, the results are still a good indication of quality failures that require attention. The results have been fed back to the broadcasters concerned and, in some cases, this has resulted in noticeable improvements subtitle timing. In the past two years there has also been an increased deployment of ASR for news output, notably on BBC News and Sky News, which, while not perfect, does keep delay and word loss within reasonable bounds.
Acknowledgements
The work has been carried out as an unfunded, independent research project, but with the support and encouragement from Dr Michael Crabb, Head of Science at the University of Dundee and Dr Carol O’Sullivan from the University of Bristol and the UK Subtitling Audiences Network [15].
1. Robert G. Baker. Guidelines for the Subtitling of Television Programmes: A Research report from Southampton University for Independent Television (UK). 1981.
2. Robert Baker, Andrew Lambourne & Guy Rowston. Handbook for Television Subtitlers. Independent Broadcasting Authority. 1982.
3. Carol O’Sullivan & Sara Ramos Pinto. Reading Between the Lines: Audience Preferences in Subtitling. Languages & the Media, 2026.
4. Action on Hearing Loss (RNID). User Experience Survey 2012.
5. BBC R&D. Subtitle Quality: Measuring and improving subtitle quality. BBC R&D Project Page. January 2012. https://www.bbc.co.uk/rd/projects/live-subtitle-quality
6. Michael Armstrong, Andy Brown, Michael Crabb, Chris Hughes, Rhianne Jones, James Sandford. Understanding the Diverse Needs of Subtitle Users in a Rapidly Evolving Media Landscape. 2015/9, IBC2015, IET. https://www.bbc.co.uk/rd/publications/whitepaper307
7. Michael Armstrong. “The development of a methodology to evaluate the perceived quality of live TV subtitles.” In IBC2013 Conference, pp. 11-1. London UK: IET on behalf of IBC, 2013. https://digital-library.theiet.org/doi/abs/10.1049/ibc.2013.0044
8. Trevor Ware & Matt Simpson. Live subtitles re-timing proof of concept. BBC R&D White Paper WHP 318. https://www.bbc.co.uk/rd/publications/whitepaper318
9. Pablo Romero-Fresco. “Accessing communication: The quality of live subtitles in the UK.” Language & Communication 49 (2016): 56-69. https://www.sciencedirect.com/science/article/abs/pii/S0271530916300398
10. Samuel Bason & Michael Armstrong. TVX2014 Short Paper – Subtitle Monitoring. https://www.bbc.co.uk/rd/blog/2014-10-tvx2014-subtitle-monitoring
11. Michael Armstrong. Automatic recovery and verification of subtitles for large collections of video clips. BBC R&D White Paper WHP 323, Oct 2016. https://www.bbc.co.uk/rd/publications/whitepaper323
(Also published in SMPTE Motion Imaging Journal, October 2017 Digital Edition.)
12. Michael Armstrong and Michael Crabb. Using AI-based tools to monitor subtitle quality. Presentation at the Conference on Disability, Accessibility and Representation in the Creative Industries, September 2025. https://enhancingaudiodescription.com/darci-conference
13. Michael Armstrong . Using generative-AI speech-to-text output to provide automated monitoring of television subtitles. IBC2025 Technical Conference September 2025. https://www.ibc.org/accelerating-innovation/reports/using-generative-ai-speech-to-text-output-to-provide-automated-monitoring-of-television-subtitles/22454
14. Michael Armstrong, Using Generative-AI Speech-to-Text Output to Provide Automated Monitoring of Television Subtitles. SMPTE Motion Imaging Journal, Volume 135, Issue 4, July 2026. https://journal.smpte.org/periodicals/SMPTE%20Motion%20Imaging%20Journal/135/4/13/
15. The Subtitling Audiences Network. https://subtitlingaudiences.org/