These are the slides for this conference which are to be presented on 5th November 2026.
This is the abstract submitted to this conference which takes place in November.
Unlike most technical and scientific conferences there are no papers required, only an abstract. However as the word limit was quite generous, this abstract turned into a short paper.
All the figures are monochrome as required by the conference.
Automated Monitoring of UK SDH Quality
A research tool exploring methods and results
Following retirement from BBC R&D I began to watch more television with the subtitles turned on. These were UK TV public service broadcast programmes in English with English language subtitles. I began to notice some quite severe delays, particularly in news programmes, especially on pre-scripted material where subtitles were being keyed out rather than respoken. This was a technique developed to reduce subtitle delay, but it was now sometimes having the opposite effect. Along with delay in live subtitles there were also archive programmes broadcast with the original subtitles where the editing was so severe that it was easier to watch the programme without them. This was because the original UK guidelines for subtitling specified a maximum word rate of only 120 words per minute [1, 2].
In order to persuade the broadcasters to improve their subtitles I needed clear evidence of these problems. This meant analysing many days of TV broadcasts to identify and characterise the issues. My focus was on delay and word loss because these were the most obvious problems and they are also the issues which are regularly highlighted in audience surveys [3, 4].
Amongst my previous work on subtitles [5, 6], I discovered a clear, linear relationship between subtitle delay and perceived subtitle quality and established that subtitles are effectively useless when delayed by ten seconds or more [7]. This led to changes to the systems for delivering live subtitles at the BBC, using pre-prepared subtitle blocks for scripted speech and pre-recorded packages along with reduced encoding delays for subtitles generated by respeaking [8].
Around the same time Ofcom, the UK’s communications regulator, in partnership with the University of Roehampton, carried out an exercise in monitoring live subtitle quality. This involved both broadcasters and subtitling students manually assessing the quality of 10-minute clips of live subtitles once every 6 months over a period of two years [9]. As a response to the sparse sampling and high costs, we developed a prototype tool capable of continuously monitoring various aspects of DSAT Teletext subtitles [10]. In a later piece of work, I utilised a speech to text engine to enable the recovery of subtitles for video clips on the BBC web videos that had been clipped from broadcast TV programmes [11].
In order to collect evidence of the problems with subtitle delay and word loss I combined the approach of off-air monitoring of DSAT teletext subtitles with the use of a speech to text engine [12, 13, 14]. The major challenge was aligning the words in the subtitles with the automatic transcript, especially where there was a significant degree of word loss and/or delay. These problems were caused by:
In order to get round most of these issues, the software uses a series of stages which start by separating out the non-speech elements from the subtitles and then attempts to match long strings of words. It then iterates over the process, matching shorter and shorter strings of words to fill the gaps. A final process then deals with variations in spelling, numerals, compound words and contractions while ignoring the 20 most common words as these often cause false matches [12, 13, 14].
Once the subtitles have been aligned with the transcript, the “start” time for each subtitle can be compared with the “begin” time for the first word of the subtitle in the transcript to give a measure of the subtitle delay. A scatter plot of delay against the start time of the subtitle is then created (figure 1). Note that not all subtitles can be aligned, but those that can give a good indication of the general trend. The average subtitle delay for each minute is also rendered as a line plot which can give a clearer indication of general trends (figure 2).
Figure 1
Figure 2
A further plot shows the number of words in each minute for the subtitles and the transcript. This can reveal changing patterns in the number of words missing from the subtitles (figure 3) which mostly occurs with live programmes and archive content. Figures 1, 2 & 3 all show the same 1 hour recording of a TV channel with commercial breaks. The first 38 minutes is a live programme which is followed by a pre-recorded programme.
Figure 3
It soon became apparent from early results (late 2024) that subtitle delays of 30 seconds or more were commonplace and caused by many different factors. Figure 4 is a plot of delay on Sky News where the subtitles pause then restart over a minute late. The system also captured examples of subtitles turning up early. Figure 5 is an example from BBC News where the subtitles are keyed out too fast to the point where they are nearly sixty seconds early then stop for two minutes, resuming over a minute late.
Figure 4
Figure 5
Figure 6 is a plot of word rates from daytime ITV1 where, for the first five minutes, over half the spoken words are missing from the subtitles because the subtitler is trying to cope with often rapid speech and multiple voices all at the same time. Figure 7 is a prerecorded programme from BBC One which had been edited without the subtitles being changed, resulting in the subtitles being nearly 4 seconds late for the last few minutes of the programme. Figure 8 is a plot of word rate for an episode of the programme Red Dwarf from 1988 which has recently been rebroadcast on the BBC complete with its original subtitles. It shows how the subtitles have been edited to keep the word rate under 120 words per minute, resulting in word loss of over 20%. These plots along with many others have been shared with the broadcasters concerned.
Figure 6
Figure 7
Figure 8
To reduce the time spent looking through one-hour plots I began creating 24 hour plots of delay, averaged over each minute, and word rates averaged over 5 minutes. Figure 9 shows a remarkable change in subtitle delay on BBC News which took place at 2pm on 13th November 2025. This plot shows how the introduction of Automatic Speech Recognition (ASR) for the BBC News channel resulted in a very consistent delay of around 4 seconds, rather than the varying delay in the hours before. This was initially a trial, but has continued in use. However, the delay has now increased to 7 seconds.
Figure 9
A further enhancement to the 24 hour plots was the addition of Electronic Programme Guide (EPG) data. Figure 10 is a plot of subtitle delay for BBC One on the 25th August 2026. The EPG data gives a clear indication of the way subtitle delay varies between live and prerecorded programmes. It also shows how the subtitle delay is now being kept below 10 seconds except for one problem during a news bulletin.
Figure 10
Figure 11 is a word rate plot for ITV1 on the 17th September 2025. Apart from the early morning programming which does not contain subtitles, the most noticeable issue is the large difference in word rate between the subtitles and transcript between 09:00 and 13:30. This is because the magazine style of programming contains many different voices often talking rapidly and sometimes over each other. Similar programming on BBC One shows the same pattern. This level of word loss on magazine programming is a significant problem which would benefit from new research.
Figure 11
This work was conceived as an exercise in demonstrating automated subtitle monitoring and providing feedback to broadcasters on the issues of delay and word loss. While the accuracy of the measurements tends to decline as the quality of the subtitles decline, the results are still a good indication of quality failures that require attention. The results have been fed back to the broadcasters concerned and, in some cases, this has resulted in noticeable improvements subtitle timing. In the past two years there has also been an increased deployment of ASR for news output, notably on BBC News and Sky News, which, while not perfect, does keep delay and word loss within reasonable bounds.
Acknowledgements
The work has been carried out as an unfunded, independent research project, but with the support and encouragement from Dr Michael Crabb, Head of Science at the University of Dundee and Dr Carol O’Sullivan from the University of Bristol and the UK Subtitling Audiences Network [15].
1. Robert G. Baker. Guidelines for the Subtitling of Television Programmes: A Research report from Southampton University for Independent Television (UK). 1981.
2. Robert Baker, Andrew Lambourne & Guy Rowston. Handbook for Television Subtitlers. Independent Broadcasting Authority. 1982.
3. Carol O’Sullivan & Sara Ramos Pinto. Reading Between the Lines: Audience Preferences in Subtitling. Languages & the Media, 2026.
4. Action on Hearing Loss (RNID). User Experience Survey 2012.
5. BBC R&D. Subtitle Quality: Measuring and improving subtitle quality. BBC R&D Project Page. January 2012. https://www.bbc.co.uk/rd/projects/live-subtitle-quality
6. Michael Armstrong, Andy Brown, Michael Crabb, Chris Hughes, Rhianne Jones, James Sandford. Understanding the Diverse Needs of Subtitle Users in a Rapidly Evolving Media Landscape. 2015/9, IBC2015, IET. https://www.bbc.co.uk/rd/publications/whitepaper307
7. Michael Armstrong. “The development of a methodology to evaluate the perceived quality of live TV subtitles.” In IBC2013 Conference, pp. 11-1. London UK: IET on behalf of IBC, 2013. https://digital-library.theiet.org/doi/abs/10.1049/ibc.2013.0044
8. Trevor Ware & Matt Simpson. Live subtitles re-timing proof of concept. BBC R&D White Paper WHP 318. https://www.bbc.co.uk/rd/publications/whitepaper318
9. Pablo Romero-Fresco. “Accessing communication: The quality of live subtitles in the UK.” Language & Communication 49 (2016): 56-69. https://www.sciencedirect.com/science/article/abs/pii/S0271530916300398
10. Samuel Bason & Michael Armstrong. TVX2014 Short Paper – Subtitle Monitoring. https://www.bbc.co.uk/rd/blog/2014-10-tvx2014-subtitle-monitoring
11. Michael Armstrong. Automatic recovery and verification of subtitles for large collections of video clips. BBC R&D White Paper WHP 323, Oct 2016. https://www.bbc.co.uk/rd/publications/whitepaper323
(Also published in SMPTE Motion Imaging Journal, October 2017 Digital Edition.)
12. Michael Armstrong and Michael Crabb. Using AI-based tools to monitor subtitle quality. Presentation at the Conference on Disability, Accessibility and Representation in the Creative Industries, September 2025. https://enhancingaudiodescription.com/darci-conference
13. Michael Armstrong . Using generative-AI speech-to-text output to provide automated monitoring of television subtitles. IBC2025 Technical Conference September 2025. https://www.ibc.org/accelerating-innovation/reports/using-generative-ai-speech-to-text-output-to-provide-automated-monitoring-of-television-subtitles/22454
14. Michael Armstrong, Using Generative-AI Speech-to-Text Output to Provide Automated Monitoring of Television Subtitles. SMPTE Motion Imaging Journal, Volume 135, Issue 4, July 2026. https://journal.smpte.org/periodicals/SMPTE%20Motion%20Imaging%20Journal/135/4/13/
15. The Subtitling Audiences Network. https://subtitlingaudiences.org/
I will be giving a presentation on my subtitle monitoring work at the SMPTE Media Broadcast Technology Conference Europe which is part of MPTS10 at London Olympia.
My presentation, Investigating the Viability of Automated Quality Control for Subtitles, is scheduled for 15:35 on 13th May 2026.
There is currently no paper to acompany the presentation. However, I have had interest from SMPTE for their Motion Imaging Journal, so one may follow.
We were watching last night’s episode of Weatherman Walking, Dylan’s Walks: 2. New Quay and Laugharne when I noticed the subtitles seemed to be consistently late by a couple of seconds.
Checking back on iPlayer to measure the timing, the scatter plot shows a distinct jump in subtitle delay at around ten and a half minutes into the programme.
If you watch the programme you can narrow the place where the timing changes down to 10:32. You can hear the edit in the sound as it jumps from seascape into music.
What has clearly happened here is that someone has cut just under 2 seconds out of the programme without updating the subtitles.
It is also possible to more accurately measure the amount cut out of the programme by looking at the timing histogram, where the two peaks are around 1.8 seconds apart.
The programme was first broadcast back in 28 Apr 2014, so that edit could have happened any time in the past 12 years, but the impact on the subtitles is quite annoying.
On 27th February I gave a talk on my subtitle quality measurement work to the wonderful STEM, LGBTQ & You 2026 conference.
Photo by @nathanstylesporter
One of the problems with plotting out subtitle data over a 24 hour period is working out which programmes are causing problems. The answer is to add EPG data to plots and working out how to get the text onto the plots in a readable form.
The result isn’t particularly elegant, because each plot is different so the positioning of the text can overlap with the plot. In the end I changed the colour of the plot and overlayed black text to make it readable. This mostly works, except for programmes lasting 10 minutes or less where the text overlaps.
The above plot is a record of subtitle delay on BBC One for the 7th January 2025. It shows approximately zero delay up till 01:30 when BBC One joins BBC News and the characteristic 4 second delay of the BBC News ASR trial is seen. Then at 06:00 the delay becomes larger and more variable because Breakfast is subtitled by humans using a mixture of autocue and scripts along with respeaking. The delay then drops slightly for Morning Live and then back to zero for the prerecorded programmes. You can also see the delay go back up during the live News bulletins at 13:00, 18:00 and 22:00, the One Show at 19:00 and Match of the Day running up to midnight.
Plotting the average word rates over 5 minute segments for the subtitles and transcript reveals some interesting contrasts in the plot above. While in the subtitles are seen to roughly keep track with the speech most of the time, three programmes stand out as causing problems, Match of the Day, Breakfast and most noticeably, Morning Live where the speech rate far exceeds the subtitle word rate.
Because the programme boundaries are now available it is also possible to plot the average word rates for each programme, see above. In this plot you don’t see the very high peak speech rates in sections of Morning live, but you can clearly see that while the average speech rate is around 200 wpm, the subtitles only manage about 160 wpm. This means that in total around 20% of the words spoken are missing from the subtitles and in some segments of the programme it is much higher.
The delay plot for ITV 1 on the 19th January shows some similar patterns and some key differences. There are no subtitles between midnight and 03:00 during Shop on TV, and Unwind with ITV from around 03:50 to 05:00 contains no speech. The programmes at 03:00 and 05:00 are prerecorded and the delay spikes are likely to be caused by anomalies caused by the commercial breaks. Good Morning Britain goes live at 06:00and the delay is around 6 seconds, dropping to zero when adverts are subtitled and there is a large delay peak of over 30 seconds just after 09:00.
By looking at the scatter plot of delay for 09:00 to 10:00, above, it would appear this peak was caused by a technical fault, because the delay rises almost linearly before dropping back. This must be some form of buffering or rate limiting downstream of the subtitler.
ITV1 live programming continues with Lorraine at 09:30, This Morning at 10:00 and then Loose Women at 12:30 where the subtitle delay averages around 12 seconds for around 15 minutes from 13:00 onwards. This might be down to the delay in the programme feed reaching one of the subtitlers working on the programme, or some other technical issue. The delay drops back below 10 seconds for the lunchtime news before prerecorded programmes take over at 14:00. at 18:00 the subtitle delay jumps to a very consistent 5 seconds for the regional news. This is consistent with the use of ASR generated subtitles for all their regional news bulletins. The delay is more variable for the national news that follows and again for the news at 22:00.
The word rate plots for commercial channels have to be interpreted with caution, because most TV adverts do not contain subtitles. This reduces the average subtitle word rate below that of the programme itself and makes the programme averages less useful. However, from the 5 minute word rate averages, it is clear that the subtitles fall behind the speech most noticeably during This Morning.
The similarity with BBC One’s Morning live is striking. These are similar programmes and clearly both present a very difficult challenge for subtitlers. These programmes are both factual and aimed at an audience that is quite likely to rely on subtitles. These programmes and others like them should be the focus for research into improving subtitle quality, because any techniques that could be found to improve the quality of subtitles for this material would benefit many other less challenging live programmes. This is the kind of problem that might benefit from the use of programme specific machine learning along with helper material like scripts alongside the used of cutting edge speech to text. However, because this is a good example of a messy, hard problem I suspect it is one many academics will avoid because of the high risk of failure.
So the 24 hour plots are made far more informative by the addition of the EPG data. The EPG data helps highlight which types of programming present the biggest challenges for subtitling and are starting to indicate where future research should focus.
The introduction of ASR subtitles for BBC News on the 13th November prompted me to produce plots covering a 24 hour period, from midnight to midnight. The main reason was I spent a couple of weeks looking at one hour plots following the change to check of glitches and this became very boring. Grouping the data into 24 hour plots made the task more reasonable as I could spot any problems and then drill down to the one hour segments where the faults occurred. The most dramatic of these one day plots is of the 13th November where the wildly fluctuating subtitle delay, typical of BBC News up to that point becomes an almost flat line.
To make the plots readable on this scale I have taken the one minute average of the subtitle delay which does not show the maximum extent of any delay (or advance) but gives a good indication of when any problems occurred.
I have also plotted the different word rates for the 24 hour period, comparing the subtitle word rate with the speech to text transcript word rate to expose times when the subtitles omit a significant proportion of the speech content. This proved more difficult, as the word rates fluctuate quite significantly from minute to minute. Eventually I settled on five minute averages for the word rates which shows the difference up reasonably well. Here is the plot of word rates plotted for the same BBC News day.
The issue with plotting a 24 hour new channel is that it has very little variation of content across the day. By contrast a mainstream channel like BBC One has a varied output with both live and pre-recorded content as can be seen from these day plots from the 16th December.
The delay plot shows the channel going over to BBC News on ASR at about 01:30 and then the start of BBC Breakfast at 06:00 where the human subtitlers take over until about 10:45 where the channel goes over to pre-recorded programmes. The News at one and six pm both have live subtitling. The peak just before 7pm is caused by the regional weather forecast and live subtitles continue until 7:30pm with the One Show. Live subtitles then reappear for the 10pm news.
The word rate plot shows how the BBC News channel ASR tracks the spoken word rate, but once the respoken subtitles from 6am onwards fall behind. The most notable problems with word loss are shown between about 9:30 and 10:30am where there is around a 30% word loss in the subtitles. The word loss between 7 and 7:30pm is also notable but less severe. These are indicative of the problems with subtitling live magazine programmes where the speech rates are quite high and there is a great deal of interchange between the speakers.
The one thing that is lacking in these plots is any indication of the programme schedule. That is my next task.
At 14:00 on 13th November my monitoring system showed a dramatic change in the form and quality of the subtitles on the BBC News channel. The subtitles went from a mixture of block and snake subtitles to being entirely snake and the distribution of subtitle delay went from -3 to +15 seconds in the period 13:00 to 14:00 to +3 to +7 seconds in the period from 14:00 to 15:00.
Before 14:00
After 14:00
Now I have been monitoring BBC News subtitles, on and off, for several months and have fed some of the worst problems back to the BBC and RedBee, their subtitle provider. I had seen examples of subtitle delay that looked like they were caused by one or more technical faults along with a few incidents of subtitles turning up early were more likely to have been caused by the subtitler. In one case I saw subtitles being buffered to the extent that ended up over 3 minutes late before catching up and another where pre-prepared blocks of subtitles were triggered around a minute early. I was beginning to suspect that there had been some improvements as a result of my feedback, but this recent change is clearly a complete technology refresh that must have been planned well in advance.
The other noticeable change my monitoring has picked up is the subtitles are now more or less verbatim. Where subtitles are generated using respeaking it is generally the case that the maximum word rate that can be achieved is 180 words per minute, often as low as 160 words per minute. In this plot from 13:00 to 14:00 you can see the subtitle word rate is in places well below the speech rate, indicating that the respeaker has left words out
By contrast between 14:00 and 15:00 the subtitles keep up with the speech. The two gaps where there is speech but no subtitles are programme trails which were not subtitled.
All of this indicates that BBC News is now trialling direct speech to text subtitles. There is no official word yet from the BBC, but having watched the subtitles for a while they are pretty impressive. They have managed to retain the colour changes between speakers, which is an improvement over ITV regional news where the speaker change is indicated by a new line beginning with “>>”, though the colour change is not entirely reliable. One other issue with these subtitles is that the punctuation is not as good as that produced by a respeaker, something that could do with more work. On the positive side, while my system cannot directly measure word errors, from what I have watched so far the accuracy is very high, even managing to get names and places correct, which suggests that the system is being fed with script and other helper data.
This is a really interesting development and it will be fascinating to see if there is any response from the audience as the change is quite obvious. There is has been no official statement about this yet, as it currently has the status of a trial, but given that it would appear to have eliminated many other problems it is likely to continue into a full deployment at which point there should be a press statement or blog post.
211
A Lightning Talk
On 14th October I gave a brief talk about measuring subtitle quality as a lightning talk at Dorkbot Bristol. It was a severely cutdown version of my IBC presentation for an audience of technically and artistically minded people. When asked the question, who here uses subtitles, not quite half of the audience raised their hands and almost all of them use same language subtitles, so I had a receptive audience.
This was the first time I’d given a talk on this work where it wasn’t being recorded so I felt free to give more details about which TV channels were being represented in the plots and led to audible gasps in one case. The audience response has given me confidence in the relevance of the topic for a non-specialist audience and will be seeking out more opportunities to talk about this work and gather feedback.
One topic that came up in the later conversations was the issue of censorship of swear words in subtitles. This is a longstanding issue, dating back to the early days of subtitles. Indeed, Prof Alan Newell who led the early work on subtitling has claimed that “A further interesting finding was the difference between the impact of the spoken and written word, the most obvious example being the use of swear words. These have a greater impact when read as subtitles rather than when heard.”[1] This attitude, was also espoused by the likes of the infamous reactionary Mary Whitehouse and when combined with other patronising attitudes towards disability this led to the censorship of swear words subtitles. These attitudes are now returning with YouTube censoring words in automatic captions, though this can be disabled.[2] Netflix has also been called out for its censorship of subtitles with their audience complaining about the service misrepresenting, censoring and simplifying dialogue from a variety of shows.
One of the issues which both YouTube and mainstream broadcasters face is speech to text systems outputting swear words in error when there are similar sounding words in the speech. In the case of broadcasters, these systems replace swear words, pre-watershed but are switched off post-watershed. The exception the BBC makes is during coverage of the Glastonbury Festival where censorship of the subtitles causes a barrage of complaints, so the system has to be switched off for the duration.
As a result of this feedback, I’m intending to add a check for censorship in subtitles, though the initial challenge is to check that the speech to text system I am using does not have censorship trained into it.
[1] Design and the Digital Divide: Insights from 40 Years in Computer Support for Older and Disabled People. Alan F. Newell, Morgan & Claypool Publishers 2011 (page 33) https://link.springer.com/book/10.1007/978-3-031-01592-2
[2] How to Uncensor the Automatic Captions. Gabe the Slacker. YouTube. https://www.youtube.com/watch?v=fTdHXCXETcQ
54
Now I have presented my paper at IBC2025 I can share a version of the slide deck and the paper.
81
