Greg Armstrong --
This BBC Podcast brings indicator discussions to life. BBC radio's weekly podcast More or Less: Behind the Stats presents an entertaining array of examples of how intelligent people can differ in their interpretations of indicator data, and how to apply common sense to the claims made using statistics, or other quantitative data.
[edited to update links May 2013]
Level of Difficulty: Moderate and entertaining
Primarily useful for: Policy makers, senior managers, Project stakeholders, donors, project managers, project monitors, and politicians of any nationality
Length: 28 minutes (mp3 format)
Limitations: 22 recent programmes are available for download but many older programmes are available only for listening online.
Who this is for
For project managers, donors, project monitors and policy makers who need to maintain a watchful eye on how indicator data are used, More or Less, provides some interesting examples of why it pays to be skeptical of indicator validity and reliability, the claims and the interpretations put on indicator data.
Background: Surveying Results-Relevant Radio
This is the third in a series of posts discussing how audio podcasts can reinvigorate thinking on indicators and results.
BBC is not the only source of intelligent programming available on the internet - there are, as I noted in my June 2011 post, several excellent programmes available from ABC, National Public Radio, and other sources. But BBC radio has by far the widest range of podcasts to choose from.
The BBC podcast website the last time I looked (August 2, 2011) had 287 programmes available for download. Of these, roughly half usually fall into categories such as music, comedy, sports, religion or children's programming but 129 programmes fell into the "factual" podcast category. Each of these programmes has multiple -- some dozens, some hundreds -- of individual episodes available for download, or listening online.
The BBC Radio 4 website in early August 2011 listed more than 9,600 individual programme episodes, with almost 9,000 of them still available for listening in some format. This is in the "factual" category alone. Some are BBC news programmes, and despite recent cutbacks to the BBC foreign language programming, news is still available in many languages(see links at the end of this article). News programmes often cease to be available more quickly than other documentaries, for obvious reasons of topicality, but the last time I looked, there were roughly 90 available.
Some of the nearly 9,000 available factual podcasts on radio 4 focus on consumer affairs, arts history or travel.
But there are several which provide useful but also entertaining insights into the kind of work we do when we think about results and how to describe, measure or report on them.
Of all the programmes I have found online, the most directly and consistently relevant to results-based management is More or Less: Behind the Stats. It has been hosted, since October 2007, by economist Tim Harford, the engagingly skeptical author of The Undercover Economist, The Logic of Life, most recently Adapt and many articles for The Financial Times and other publications. Tim Harford brings a common sense and clear language approach to determining whether claims for results, and the use of statistics to support such claims, are credible.
In this, he continues the work of his irreverent predecessor, Andrew Dilnot, currently the chair of a U.K. Commission examining long-term health care for the elderly. Andrew Dilnot set the tone for More or Less with his no-nonsense approach to data. As The Guardian recently wrote about the Dilnot report, it is "Rich in evidence and pithy in prose" - and this is, after all, what we need more of, in all of our reporting.
Dilnot wrote a few years ago that what is important in judging political leaders' claims is "respect for data over wishful thinking", something that could be said equally of results claimed for development projects.
In a world where political leaders exhort aid workers to base -- and justify -- their programming decisions on indicator evidence, but themselves use evidence as the basis for policy only when it suits their political needs, this is something worth remembering.
More or Less has been produced since 2005 in association with the Open University and their site has some supplementary written material. Additional interesting material is available also on Tim Harford’s other websites.
The programme is updated for downloads every Friday during its broadcast seasons, and appears to have 2-3 broadcast seasons of 7-9 episodes each year.
The More or Less website had, as of August 2, 2011, 22 episodes available for portable listening in the standard and (easiest to download) format - MP3. (A very rudimentary introduction on how to download and listen to podcasts, is available in my May post.) Each current episode of More or Less is 28 minutes long, and they cover the period between September 2010 and May 2011, the most recent broadcasts. An additional 28 streaming episodes of More or Less are available for listening -- but not downloading -- using the BBC Player, covering the period between January 2009 and August 2010. A new season of More or Less begins in the first week of August 2011, and the September 2010 episodes will probably be archived before the new season ends.
There are also older More or Less episodes, going back to February 2003, with Tim Harford or Andrew Dilnot, but there is no point in trying to access those by clicking on the "previous programmes by year" link, because some work, and some don't. But if you go to the More or Less Archives you can get 73 more episodes. What is curious about these older archived programmes is that some - such as the earliest available episode of More or Less in February 2003, do have audio you can listen to, albeit in the sometimes problematic ram format, while others, including some later programmes, simply have a written description, but do not, as far as I can see, have an audio component. All six of the More or Less episodes broadcast between February and March 2003 have audio, for example, but none of the six episodes broadcast in January and February 2004 appear to have audio.
Nevertheless, all things considered, I estimate that that there are probably about 120 episodes of More or Less that you can listen to one way or the other and as the new season arrives there will be more.
Each episode of More or Less usually deals with discussions of 5-6 indicator issues. Recent episodes, for example, have included discussions on indicators related to, among many other subjects
While many of the items on More or Less use examples from the United Kingdom, it is easy to see how lessons from the debunking of claims about problems or results, could be applied to other issues, and other countries.
A useful starting point for anyone wanting to test the More or Less range of issues, is the December 31, 2010 downloadable episode. The MP3 version of this episode can currently be downloaded from the main More or Less website, with 21 others, but it will soon be archived, and then you will need to go to the BBC player version of the programme, where currently the most recent 48 episodes are available, and listen online.
"The meat and drink of More or Less are the errors and connivances embedded in the statistics which fill each news bulletin" Tim Harford noted in the final programme of 2010. And this episode illustrates his point, as it produces the most important numbers of 2010, as seen by 7 people who work regularly with indicator interpretation.
Examining one episode in slightly more detail provides an example, I hope, of how we can use podcasts to stimulate ideas and, with a little effort, further research on indicators. As there are no clickable links in podcasts (at least not in this one), the references below are to the time (in minutes and seconds) into the podcast where you can find the reference.
Included in the 2010 Year-End summary of indicators in the news:
Daniel Franklin Executive Editor of The Economist, and editor of The World in 2010 and The World in 2011, discusses the difference between David Cameron's claims on crime and a "broken society" and what the statistics on crime rates, teen pregnancy, smoking and other issues say. Some of this is reflected also in the data on percentage of births to teenage mothers, suggested by National Statistician Jil Matheson (08:24-10:19)
It is easy to see how these discussions on falling crime rates and the implication for public policy could apply to other countries, such as Germany the U.S. or Canada
Cathy Newman, former correspondent for the Financial Times, now political correspondent for Britain's Channel 4 news, and author of the Factcheck Blog contrasts former Prime Minister Gordon Brown's claims that defence spending rose in real terms in recent years, while data on inflation - adjusted spending suggested defence spending had fallen. She points out the difference between "cash spending" and "real spending", and she also notes that the Conservative - Liberal Democrat coalition government will be reducing the defence budget by about 8% in real terms over the next four years.
Tim Harford highlights the need to check indicator data sources, before making extravagant claims. He shows how Liberal Democratic party leader Nick Clegg's election debate claim that 80% of immigrants to the UK came from the European Union, was based on his party's misreading of an Economist article, which referred to students, not immigrants. The real figure Harford says, and a report in the Daily Telegraph appears to confirm, is about 39%. It is not known from any of these sources if the use of the faulty data was sloppy party research or careless use of the data in the debate.
David Spiegelhalter, The Winton Professor of the Public Understanding of Risk at Cambridge University, talks about the need to put risk indicators in perspective. He examines risks for the military in Afghanistan, and compares these to the risk of riding a motorcycle on a major highway in the U.K.
In the longest discussion in this episode, Hans Rosling, co-founder of the Gapminder Institute, who will be familiar to many people from his entertaining presentations on indicators , talks about how indicator data on issues such as child mortality and economic growth differ widely country to country in sub-Saharan Africa.
When questioned about the reliability of indicator data from under-funded African statistical offices, he explains why some indicator data such as child mortality and fertility rates are reliable, while other data on indicators such as maternal mortality and unemployment are not.
But, Hans Rosling says:
There is much more in this interview with Rosling, on:
Finding an indicator for the "slippery concept" of well-being
(18:37-20:24)
Statistician Michael Blastland former producer of More or Less, co-author with former host Andrew Dilnot of The Numbers Game: The Commonsense Guide to Understanding Numbers in the News, in Politics, and in Life , and author of a regularly published column on the BBC News online Magazine on making statistics relevant to non-statisticians, considers whether insomnia can be taken as an indicator for wellbeing - and concludes "...Well-being: this is a good illustration of how slippery a concept that is and the number of things that might have to go into it".
Dealing with slippery concepts is something development aid workers will be familiar with.
Robert Peston, Business Editor at the BBC talks about how the amount of cash banks have on hand as a percentage of what they borrowed, is an indicator of possible bank failures. Comparing British banks at the end of 2008, with British banks during the Great Depression, he comes up with some surprising information. It is presumably still relevant in 2011, but the interview never quite makes clear how
Ben Goldacre, a physician, the author of the Bad Science blog, a book of the same name, and a columnist for the Guardian uses the case of incompletely reported data on the drug reboxetine to illustrate how published studies on drug effectiveness provide unreliable indicators for safety and effectiveness because of the way data are both withheld, and then reported. While this brief comment focuses on this one drug, his blog has discussed several other examples of how inaccurately reported data on drug trials can be, to put it mildly, unreliable. These include, within the past year, reviews of a medicine for schizophrenia that may cause diabetes, a diabetes medication that may increase risks of heart attacks and several critiques on claims for homeopathic medicine.
This short session of More or Less with Ben Goldacre is of relevance for those involved in results reporting for any field. Checking the methods and context for any research providing us with indicator data, is something that is often neglected as indicators are used in reports on development projects. Failing to check out the sources for published results supporting results indicators in development projects, however, rarely has such immediately dangerous implications as does careless - or malevolent - use of indicators in the field of medical research.
For an entertaining infographic on Bad Science see the Bad Science Infographic.
Limitations:
Unbundling....other programmes do it:
As I noted in my previous survey of a number of results-relevant podcasts, several -- such as ABC's Counterpoint unbundle the programmes -- break them down into components that can be downloaded separately. ABC Radio's The Science Show makes the case for this when it says:
Some (but not all) episodes of the National Public Radio show RadioLab also do this quite well, dividing an episode which has its own internal coherence, into 3-4 components available separately and with their own references. You can see examples of these, among many others, with the May 31, 2011 episode on talking to machines or the June 2009 episode on randomness and data patterns (stochasticity). There may be some BBC programmes that do unbundle their podcasts by themes, but I haven't found them yet.
Time limits:
While the 22 recent episodes (this will undoubtedly change in August 2011 with the new season) are available for download and portable listening, and readers can listen to, but not download, many others dating back to April 2003, some of the programmes prior to 2005 do not open easily. The easiest to download format - the MP3 versions currently available for the last 22 episodes - are usually only available in that format for a year. So if you want to be sure you can download and save some of the interesting episodes, it is worth skimming the site first, identifying potentially interesting episodes, and downloading them before they are archived or go to the BBC Player site. The September 2010 episodes will probably be archived soon.
Weak research links:
More or Less is relevant to those who work with results and indicators, but of course it provides just tantalizing summaries of the issues, not the whole picture. It is natural that many of us would want to do further research online to follow up on the issue summaries in each episode. This is the whole point of being online - to get access to multiple sources of data. We can, of course, do our own research, and many of the links I provided earlier in this post when I discussed the 2010 New Year's episode of More or Less, were links I did find myself, through some time-consuming research.
While the More or Less website has improved since January 2011, and particularly since the April 2011 season, it is still a disappointment that there is so little assistance on the site for further research, particularly for some of the programmes that predate 2011. The More or Less website does indeed now provide more links that it previously did, both directly and through the Open University link, but there remain occasional problems even in these links. In at least one case I noted in January 2010 the site misspelled the name of one of the guests (Ben Goldacre) and that error remains as I write this in July. Such things prove distracting, and complicate attempts to follow the story online with further search. While it is easy to make errors in writing (and this blog undoubtedly has some) there is a certain irony in finding such an error on the website of a podcast dealing with data integrity.
Other podcasts such as RadioLab have websites that provide much more in terms of links to further research, and in RadioLab's case, even have a reading list. None of these other programmes has the direct and frequent relevance to results and indicators that More or Less has, but they are better organized.
More or Less is, as I have said, the most useful of all of the programmes I have found so far, to those of us who work with results and indicators, and it is disappointing that the website does not make it easier to capitalize on the good work they have done. More links to background material on the More or Less website would be useful as would even the inclusion of metadata accessible to listeners who might want to go further.
The Bottom Line:
More or Less: Behind the Stats, is the most useful programme on the internet for people interested in how indicators are used in daily life, in policy analysis, politics and assessing results. It is worth listening to, for most people involved with results-based management. A new season of More or Less begins on August 5, 2011.
For our colleagues working in other languages - BBC foreign-language news
BBC News in Persian
BBC News in Mandarin
BBC News in Cantonese
BC News summaries and commentary from the press in Turkish
BBC News in Burmese
BBC News and commentary in Russian
BBC News Analysis in Ukrainian
BBC News in Indonesian
BBC News in Spanish
At some associated BBC sites, languages such as Swahili are available, but not as regular news broadcasts.
This BBC Podcast brings indicator discussions to life. BBC radio's weekly podcast More or Less: Behind the Stats presents an entertaining array of examples of how intelligent people can differ in their interpretations of indicator data, and how to apply common sense to the claims made using statistics, or other quantitative data.
[edited to update links May 2013]
 |
| BBC Radio's More or Less Podcast |
Level of Difficulty: Moderate and entertaining
Primarily useful for: Policy makers, senior managers, Project stakeholders, donors, project managers, project monitors, and politicians of any nationality
Length: 28 minutes (mp3 format)
Limitations: 22 recent programmes are available for download but many older programmes are available only for listening online.
Who this is for
For project managers, donors, project monitors and policy makers who need to maintain a watchful eye on how indicator data are used, More or Less, provides some interesting examples of why it pays to be skeptical of indicator validity and reliability, the claims and the interpretations put on indicator data.
Background: Surveying Results-Relevant Radio
This is the third in a series of posts discussing how audio podcasts can reinvigorate thinking on indicators and results.
The first post in this series discussed the mechanics of downloading and listening to podcasts. The second post surveyed the wide range of available podcasts from the BBC, ABC and National Public Radio of potential use to people working on results and indicators.
This post reviews what I think is the single most useful programme on world radio, for people who work with indicators: BBC's More or Less: Behind the Stats.
BBC Podcasts
BBC is not the only source of intelligent programming available on the internet - there are, as I noted in my June 2011 post, several excellent programmes available from ABC, National Public Radio, and other sources. But BBC radio has by far the widest range of podcasts to choose from.
Roughly 9 Thousand available BBC podcasts
The BBC podcast website the last time I looked (August 2, 2011) had 287 programmes available for download. Of these, roughly half usually fall into categories such as music, comedy, sports, religion or children's programming but 129 programmes fell into the "factual" podcast category. Each of these programmes has multiple -- some dozens, some hundreds -- of individual episodes available for download, or listening online.
The BBC Radio 4 website in early August 2011 listed more than 9,600 individual programme episodes, with almost 9,000 of them still available for listening in some format. This is in the "factual" category alone. Some are BBC news programmes, and despite recent cutbacks to the BBC foreign language programming, news is still available in many languages(see links at the end of this article). News programmes often cease to be available more quickly than other documentaries, for obvious reasons of topicality, but the last time I looked, there were roughly 90 available.
Some of the nearly 9,000 available factual podcasts on radio 4 focus on consumer affairs, arts history or travel.
But there are several which provide useful but also entertaining insights into the kind of work we do when we think about results and how to describe, measure or report on them.
More or Less: Behind the Stats
Of all the programmes I have found online, the most directly and consistently relevant to results-based management is More or Less: Behind the Stats. It has been hosted, since October 2007, by economist Tim Harford, the engagingly skeptical author of The Undercover Economist, The Logic of Life, most recently Adapt and many articles for The Financial Times and other publications. Tim Harford brings a common sense and clear language approach to determining whether claims for results, and the use of statistics to support such claims, are credible.
Respect for Data
In this, he continues the work of his irreverent predecessor, Andrew Dilnot, currently the chair of a U.K. Commission examining long-term health care for the elderly. Andrew Dilnot set the tone for More or Less with his no-nonsense approach to data. As The Guardian recently wrote about the Dilnot report, it is "Rich in evidence and pithy in prose" - and this is, after all, what we need more of, in all of our reporting.
Dilnot wrote a few years ago that what is important in judging political leaders' claims is "respect for data over wishful thinking", something that could be said equally of results claimed for development projects.
"If you'd prefer to be flattered by bogus numbers, to believe that the world changes when you play statistical games, or at least to act as if it does, wrote Dilnot "you are, let's be blunt, delusional and dangerous."
In a world where political leaders exhort aid workers to base -- and justify -- their programming decisions on indicator evidence, but themselves use evidence as the basis for policy only when it suits their political needs, this is something worth remembering.
120 Available episodes of More Or Less
More or Less has been produced since 2005 in association with the Open University and their site has some supplementary written material. Additional interesting material is available also on Tim Harford’s other websites.
The programme is updated for downloads every Friday during its broadcast seasons, and appears to have 2-3 broadcast seasons of 7-9 episodes each year.
The More or Less website had, as of August 2, 2011, 22 episodes available for portable listening in the standard and (easiest to download) format - MP3. (A very rudimentary introduction on how to download and listen to podcasts, is available in my May post.) Each current episode of More or Less is 28 minutes long, and they cover the period between September 2010 and May 2011, the most recent broadcasts. An additional 28 streaming episodes of More or Less are available for listening -- but not downloading -- using the BBC Player, covering the period between January 2009 and August 2010. A new season of More or Less begins in the first week of August 2011, and the September 2010 episodes will probably be archived before the new season ends.
There are also older More or Less episodes, going back to February 2003, with Tim Harford or Andrew Dilnot, but there is no point in trying to access those by clicking on the "previous programmes by year" link, because some work, and some don't. But if you go to the More or Less Archives you can get 73 more episodes. What is curious about these older archived programmes is that some - such as the earliest available episode of More or Less in February 2003, do have audio you can listen to, albeit in the sometimes problematic ram format, while others, including some later programmes, simply have a written description, but do not, as far as I can see, have an audio component. All six of the More or Less episodes broadcast between February and March 2003 have audio, for example, but none of the six episodes broadcast in January and February 2004 appear to have audio.
Nevertheless, all things considered, I estimate that that there are probably about 120 episodes of More or Less that you can listen to one way or the other and as the new season arrives there will be more.
A wide range of indicators
Each episode of More or Less usually deals with discussions of 5-6 indicator issues. Recent episodes, for example, have included discussions on indicators related to, among many other subjects
- measuring child poverty
- calculating civilian deaths in war zones
- comparing international data on student achievement
- measuring well-being
- abuses of statistical significance claims
- measuring the "fiscal multiplier"
- how luck, and regression to the mean, can bias data interpretation
- whether celebrity (or royal) marriages - or crime - lead to jumps in marriage rates
- calculating the real costs of military interventions
- how differences in data definitions can bias international comparisons
- distinguishing between correlation and causation
- how different methods of calculating "averages" can affect indicator data
Using More or Less as a research tool: What the data said about 2010
While many of the items on More or Less use examples from the United Kingdom, it is easy to see how lessons from the debunking of claims about problems or results, could be applied to other issues, and other countries.
A useful starting point for anyone wanting to test the More or Less range of issues, is the December 31, 2010 downloadable episode. The MP3 version of this episode can currently be downloaded from the main More or Less website, with 21 others, but it will soon be archived, and then you will need to go to the BBC player version of the programme, where currently the most recent 48 episodes are available, and listen online.
"The meat and drink of More or Less are the errors and connivances embedded in the statistics which fill each news bulletin" Tim Harford noted in the final programme of 2010. And this episode illustrates his point, as it produces the most important numbers of 2010, as seen by 7 people who work regularly with indicator interpretation.
Examining one episode in slightly more detail provides an example, I hope, of how we can use podcasts to stimulate ideas and, with a little effort, further research on indicators. As there are no clickable links in podcasts (at least not in this one), the references below are to the time (in minutes and seconds) into the podcast where you can find the reference.
Included in the 2010 Year-End summary of indicators in the news:
Indicator data on crime and social change
(00:56)Daniel Franklin Executive Editor of The Economist, and editor of The World in 2010 and The World in 2011, discusses the difference between David Cameron's claims on crime and a "broken society" and what the statistics on crime rates, teen pregnancy, smoking and other issues say. Some of this is reflected also in the data on percentage of births to teenage mothers, suggested by National Statistician Jil Matheson (08:24-10:19)
It is easy to see how these discussions on falling crime rates and the implication for public policy could apply to other countries, such as Germany the U.S. or Canada
Indicators on defence spending
(02:30 into the episode)Cathy Newman, former correspondent for the Financial Times, now political correspondent for Britain's Channel 4 news, and author of the Factcheck Blog contrasts former Prime Minister Gordon Brown's claims that defence spending rose in real terms in recent years, while data on inflation - adjusted spending suggested defence spending had fallen. She points out the difference between "cash spending" and "real spending", and she also notes that the Conservative - Liberal Democrat coalition government will be reducing the defence budget by about 8% in real terms over the next four years.
Misleading indicators on immigration
(04:25)Tim Harford highlights the need to check indicator data sources, before making extravagant claims. He shows how Liberal Democratic party leader Nick Clegg's election debate claim that 80% of immigrants to the UK came from the European Union, was based on his party's misreading of an Economist article, which referred to students, not immigrants. The real figure Harford says, and a report in the Daily Telegraph appears to confirm, is about 39%. It is not known from any of these sources if the use of the faulty data was sloppy party research or careless use of the data in the debate.
Compared to what? Risk indicators
(07:30)David Spiegelhalter, The Winton Professor of the Public Understanding of Risk at Cambridge University, talks about the need to put risk indicators in perspective. He examines risks for the military in Afghanistan, and compares these to the risk of riding a motorcycle on a major highway in the U.K.
Hans Rosling on the quality of indicator data
(10:30-18:10)In the longest discussion in this episode, Hans Rosling, co-founder of the Gapminder Institute, who will be familiar to many people from his entertaining presentations on indicators , talks about how indicator data on issues such as child mortality and economic growth differ widely country to country in sub-Saharan Africa.
When questioned about the reliability of indicator data from under-funded African statistical offices, he explains why some indicator data such as child mortality and fertility rates are reliable, while other data on indicators such as maternal mortality and unemployment are not.
But, Hans Rosling says:
"It is not countries that have weak indicators. It is certain indicators that are weak for methodological reasons". (13:23)
There is much more in this interview with Rosling, on:
- Indicators on fertility;
- Rates of change to indicators such as child mortality, and economic growth in China;
- Disaggregating indicator data and defining geographic focus for indicators;
- His conclusions about the relationship between economic improvement, good governance and democracy;
- The difference between ideals and advocacy, and facts.
Finding an indicator for the "slippery concept" of well-being
(18:37-20:24)
Statistician Michael Blastland former producer of More or Less, co-author with former host Andrew Dilnot of The Numbers Game: The Commonsense Guide to Understanding Numbers in the News, in Politics, and in Life , and author of a regularly published column on the BBC News online Magazine on making statistics relevant to non-statisticians, considers whether insomnia can be taken as an indicator for wellbeing - and concludes "...Well-being: this is a good illustration of how slippery a concept that is and the number of things that might have to go into it".
Dealing with slippery concepts is something development aid workers will be familiar with.
Cash as an indicator of bank viability
(20:26)Robert Peston, Business Editor at the BBC talks about how the amount of cash banks have on hand as a percentage of what they borrowed, is an indicator of possible bank failures. Comparing British banks at the end of 2008, with British banks during the Great Depression, he comes up with some surprising information. It is presumably still relevant in 2011, but the interview never quite makes clear how
How Incomplete Data reporting undermines indicator utility
(22:50)Ben Goldacre, a physician, the author of the Bad Science blog, a book of the same name, and a columnist for the Guardian uses the case of incompletely reported data on the drug reboxetine to illustrate how published studies on drug effectiveness provide unreliable indicators for safety and effectiveness because of the way data are both withheld, and then reported. While this brief comment focuses on this one drug, his blog has discussed several other examples of how inaccurately reported data on drug trials can be, to put it mildly, unreliable. These include, within the past year, reviews of a medicine for schizophrenia that may cause diabetes, a diabetes medication that may increase risks of heart attacks and several critiques on claims for homeopathic medicine.
This short session of More or Less with Ben Goldacre is of relevance for those involved in results reporting for any field. Checking the methods and context for any research providing us with indicator data, is something that is often neglected as indicators are used in reports on development projects. Failing to check out the sources for published results supporting results indicators in development projects, however, rarely has such immediately dangerous implications as does careless - or malevolent - use of indicators in the field of medical research.
For an entertaining infographic on Bad Science see the Bad Science Infographic.
Limitations:
Unbundling....other programmes do it:
As I noted in my previous survey of a number of results-relevant podcasts, several -- such as ABC's Counterpoint unbundle the programmes -- break them down into components that can be downloaded separately. ABC Radio's The Science Show makes the case for this when it says:
"The whole program cut up into separate stories - allows easy skipping from one story to the next so you can pick and choose".What is also useful is that the Science show also makes transcripts available for many episdoes, so having listened to it, it is relatively easy to go back, check data and follow up.
Some (but not all) episodes of the National Public Radio show RadioLab also do this quite well, dividing an episode which has its own internal coherence, into 3-4 components available separately and with their own references. You can see examples of these, among many others, with the May 31, 2011 episode on talking to machines or the June 2009 episode on randomness and data patterns (stochasticity). There may be some BBC programmes that do unbundle their podcasts by themes, but I haven't found them yet.
Time limits:
While the 22 recent episodes (this will undoubtedly change in August 2011 with the new season) are available for download and portable listening, and readers can listen to, but not download, many others dating back to April 2003, some of the programmes prior to 2005 do not open easily. The easiest to download format - the MP3 versions currently available for the last 22 episodes - are usually only available in that format for a year. So if you want to be sure you can download and save some of the interesting episodes, it is worth skimming the site first, identifying potentially interesting episodes, and downloading them before they are archived or go to the BBC Player site. The September 2010 episodes will probably be archived soon.
Weak research links:
More or Less is relevant to those who work with results and indicators, but of course it provides just tantalizing summaries of the issues, not the whole picture. It is natural that many of us would want to do further research online to follow up on the issue summaries in each episode. This is the whole point of being online - to get access to multiple sources of data. We can, of course, do our own research, and many of the links I provided earlier in this post when I discussed the 2010 New Year's episode of More or Less, were links I did find myself, through some time-consuming research.
While the More or Less website has improved since January 2011, and particularly since the April 2011 season, it is still a disappointment that there is so little assistance on the site for further research, particularly for some of the programmes that predate 2011. The More or Less website does indeed now provide more links that it previously did, both directly and through the Open University link, but there remain occasional problems even in these links. In at least one case I noted in January 2010 the site misspelled the name of one of the guests (Ben Goldacre) and that error remains as I write this in July. Such things prove distracting, and complicate attempts to follow the story online with further search. While it is easy to make errors in writing (and this blog undoubtedly has some) there is a certain irony in finding such an error on the website of a podcast dealing with data integrity.
Other podcasts such as RadioLab have websites that provide much more in terms of links to further research, and in RadioLab's case, even have a reading list. None of these other programmes has the direct and frequent relevance to results and indicators that More or Less has, but they are better organized.
More or Less is, as I have said, the most useful of all of the programmes I have found so far, to those of us who work with results and indicators, and it is disappointing that the website does not make it easier to capitalize on the good work they have done. More links to background material on the More or Less website would be useful as would even the inclusion of metadata accessible to listeners who might want to go further.
The Bottom Line:
More or Less: Behind the Stats, is the most useful programme on the internet for people interested in how indicators are used in daily life, in policy analysis, politics and assessing results. It is worth listening to, for most people involved with results-based management. A new season of More or Less begins on August 5, 2011.
Further Reading, viewing and listening on indicators and results
- The Open University on the BBC: More or Less Inside Interview with Tim Harford
- Straight talk on Statistics
- Understanding risk and uncertainty
- "Professor Risk" on Youtube
- The Guardian Data Store: Facts are Sacred
- The Bad Science Blog
- The Factcheck Blog
- The Numbers Guy on the Wall Street Journal
- ABC's Counterpoint
- The Gapminder Institute
For our colleagues working in other languages - BBC foreign-language news
BBC News in Persian
BBC News in Mandarin
BBC News in Cantonese
BC News summaries and commentary from the press in Turkish
BBC News in Burmese
BBC News and commentary in Russian
BBC News Analysis in Ukrainian
BBC News in Indonesian
BBC News in Spanish
At some associated BBC sites, languages such as Swahili are available, but not as regular news broadcasts.
_____________________________________________________________
