On the DfE Academies and Autonomy Report Part 1: Poor Methodology

In this series of blogposts I review 'Do academies make use of their autonomy?' which is a research report published by the Department of Education in July 2014. I quote liberally from the report which can be found in its entirety here.

The first thing I noticed about this report is that, considering a title that is quite limiting in scope, the authors made some quite sensational findings:

'Academies have used their freedoms to innovate and improve

  • 79 per cent have changed or plan to change their curriculum
  • 90 per cent have procured or planned to procure services previously provided by the LA
  • 84 per cent are now linking pay to performance
This is helping them raise standards for their pupils

  • Two thirds believe these changes have improved attainment
  • The most important changes were seen to be those to the curriculum and leadership

It is also helping them to raise standards for pupils in other schools via collaboration

  • 87 per cent of academies support other schools (72 per cent support schools they did not support before becoming academies)
  • 96 per cent of outstanding academies support other schools

So academies have used their freedoms and said freedoms are implicitly a good thing for students whose attainment has improved – this includes students in other schools who have benefitted from collaboration with the academy. Grandiose, in fact so grandiose that the sceptical educationalist might expect some rigorous evidence to back all of this up yet a look at the research methodology reveals some pretty big holes:

'15 minute online survey was sent to the 2919 academies open on 1st May 2013.

Academies which converted after the 1st May 2013 were excluded from the survey. The survey focused on changes made since becoming an academy so it was not appropriate to ask a number of the questions to recent converters. The survey was initially issued to academies on 24th February 2014 and fieldwork closed on 30th March 2014. A total of 720 academies replied (a response rate of 25 per cent).'

The sample size here is very impressive and a pertinent survey seems to be a reasonable method of identifying whether academies are actually using their freedoms and/or collaborating with other schools – if you have a control group of non-academies to compare the responses with:

'A short survey was also sent to a sample of LA maintained schools to act as a comparison group. Only sixty schools responded so the results have not been included in this report'

So no control group then not that it mattered to the DfE who made their conclusions anyway. Now I'll accept that control groups aren't always needed if what one has observed is obviously something that only an academy can do. There is, for example, qualitative commentary that suggests some academies are now able to do things that the LA objected to when they were in charge:

'We have opened an academically selective sixth form. We would not have been able to open a 6th form as an LA school. The LA is firmly opposed to school-based 6th forms.”

However the majority of the exemplar evidence is, well, eyebrow raising at best – for example are changes to the 'ethos, values, aspirations of all students and staff' or 'rigorous quality assurance…allied to performance management and CPD' or a 'zero tolerance approach to poor behaviour' really manifestations of academy freedoms? Is LA status really an insurmountable barrier to providing 'effective partnership, sharing of resources for the benefit of staff and our children'? The DfE doesn't actually know because without a control group there is no evidence that non-academy schools are, or are not, doing the same things.

If you are going to claim that changing a behaviour policy is evidence of academies using their freedoms then you may as well slide all the way down the slippery slope and believe that any change made by an academy is good for student outcomes – this is what the DfE have done. Despite a third of academies claiming otherwise the DfE included the sweeping headlines 'this is helping them raise standard for pupils' and 'Academies have used their freedoms to innovate and improve' while in the extended conclusion we find the following, rather cocksure, pronouncement:

'Some of the changes which were made by a minority, such as lengthening the school day, were often felt to be the most important change made by academies. The department will use this research to develop examples of how changes made have led to school improvement and communicate these messages within the system.'

Now this might be fine if the people telling to DfE that these changes raise standards were not, er, the academies themselves (well two thirds of them anyway):

'We are now able to recruit high quality Russell Group trained teachers. Academic standards are very high. We are now the highest performing school in [the LA] and by far the most oversubscribed school!'

'The needs of the children are now more effectively addressed and the whole community has benefitted from the high quality teaching and learning to be found in both establishments.'

'By becoming a sponsor academy we could invest further in our own school and impact further on standards but also use our successful model in other failing schools which had issues.'

'Two thirds of academies believe that the changes they have made have improved attainment. This is especially the case for sponsored academies. The longer an academy has been open, the more likely they are to say the changes have substantially improved attainment.'

In terms of scientific validity this research is below standard. Are academies using their freedoms to the extent suggested in the report? It is difficult to tell without a control sample of non academy schools to compare them with. Are academies actually improving students outcomes? Most people would be more inclined to believe it if an unbiased source told them so. The tradegy here is that the DfE a) has a lot of data from every school to compare student outcomes and b) despite the incredibly dubious effort actually had enough responses (sixty) from LA schools to form a control group and attempt some statistical analysis of academy behaviour. They didn't and it is high time they did – if it is favourable good but if it isn't maybe they'll think twice about publishing puff pieces about academies and the academy policy.


Mixed Bag: The Paradoxical Tenure of Education Secretary Gove

Following his departure from the Department of Education a lot of bloggers were compelled to voice their opinions about his four year reign as Secretary of State for Education. In the name of originality I've decided to do exactly the same thing!

If nothing else no one can ever accuse Gove of not standing up for his beliefs regardless of their popularity with the electorate and within the teaching profession. Gove was contemptuous of the fashionable pedagogy that spawned things like learning styles and learning pyramids. He seemingly ignored the bureaucratic monoliths created by his ministerial predecessors: Every Child Matters and Assessing Pupil Progress. His focus on subject knowledge and academic rigour meant that, for the first time in years, 'traditional' teachers had a champion at the summit of the educational establishment – although the battle against child centred progressivism was only beginning Gove endorsed intelligent critiques of Ofsted and the rise of teacher led blogging and networking through conferences such as researchED. In terms of policy Gove's biggest impact was to change school accountability measures to tackle the scandalous rise of the equivalent qualification – this will probably do more to improve the educational lot of disadvantaged children, which was important to Gove, than anything else.

Despite the aforementioned changes Gove's departure from the Department for Education was generally well recieved by the downtrodden rank and file of the teaching profession. Gove seemingly had a monetarist view of management believing that most people are essentially lazy and self-interested and need to be pushed along by extrinsic forms of motivation; his very worst policies – psuedo competition between schools, using broad averages as benchmarks and performance related pay for teachers are the product of this thinking. The irony here is that Gove's ideas about people meant that he found idealogical kinship with the school managers responsible for the problems he identified in the first place.

All of this meant Gove was a paradoxical minister. For every collaborative effort (aka researchED) Gove admired there was an edu-brand like Harris to be praised. Every time Gove reminded us about standards he'd name check an educational 'hero' who'd benefited from lowering them. For every 'traditional' teacher blogger Gove praised in public power would be handed to a non-teaching monetarist behind closed doors. Gove introduced a pay system that relies on individual self interest then bemoaned the selfish nature of teachers taking industrial action. Gove admired Army style discipline yet turned a blind eye to the behaviour crisis. Gove never really articulated what he thought schools were for; sometimes it was all about experts providing a liberal education while at other times it was about free child care provided by, well, anyone the school managers feel like.

I mentioned earlier that Gove stood up for his beliefs – this would have been a good thing if those beliefs, when taken together, actually made sense.

Other articles about Gove that I really liked:

Goodbye Mr Gove

Michael Gove and the bitter pills of education reform

Micheal Gove: loved by the media loathed by the public

Other articles about Gove that I really didn't like:

How will I remember Mr Gove



The Long Walk 1st Anniversary Scrapbook

My name is Leonard James, I am a secondary Science teacher at a comprehensive school in England. I have been teaching for eight years and have been promoted to middle leadership during this time. The Long Walk is an outlet for my personal concerns and frustrations about British education. Here I outline said concerns and provide links to the archive of blogposts.

The educational establishment

The educational establishment sits at the summit of the British education systems managemental hierarchy. It is a toxic mix of empire building CEOs, consultants, gurus, bureaucrats and politicians. They are divorced from, and therefore ignorant about, the everyday business of schools. They manage from afar via endless specifications, targets and compliance checking. The following series of blogs highlight some of the unintended consequences of this management style:

Educational Platitudes #1: 'Kayliegh wants to do well'

What do Ofsted have to say about Science curriculums?

Ofsted Data Dashboards: Spot the Difference

Methods for Attaining League Table Altitude

%5 A*-C inc EM

Mandatory Optimism

ERSC Miss the Elephant in the Room

Minions of Ofsted Part 1: High Stakes Lesson Observations

Minions of Ofsted Part 2: Learning Walks

Minions of Ofsted Part 3: The Consultants

The New Ofsted Data Dashboards Reward Science Departments who Game the Examination System

School choice, competition and the Free Market

The problems described above are exacerbated by personalities within the educational establishment who can't get along. Some members of the educational establishment are monetarists (mainly CEOs and some politicians) pushing for an education system based on free market principles. The following posts describe how teachers and their work are or might be affected by this sort of thinking:

On the Staffroom Debate

Educational Platitudes #2: 'I'm passionate about…'

Injecting Competition: A response to Gabriel Sahlgren and Julian Le Grand

Teaching is a McJob Part 1: Low Pay

Teaching is a McJob Part 2: Low Prestige

Teaching is a McJob Part 3: No Training

Progressive education

Many of those within the educational establishment (who are described by Michael Gove as 'the blob') reject liberal education in favour of child centered 'progressive' pedagogy. This pedagogy undermines the authority of adults, has contributed to a national behaviour crisis and is detrimental to learning:

A Response to Mitra Part 1: Education and Employability

A Response to Mitra Part 2: Classroom Pedagogy

A Response to Mitra Part 3: The Past


Injecting Competition: A response to Gabriel Sahlgren and Julian Le Grand

Gabriel Sahlgren (research director of the Centre for Market Reform of Education) and Julian Le Grand (professor at the London School of Economics) co-authored an interesting, if ultimately unconvincing, comment piece for the Financial Times advocating increased competition between schools. In this response I have quoted liberally from the article of which can be found in full here. I would encourage all to read it before going any further.

The article coincides with media coverage of yet more anxiety inducing evidence from PISA about relative achievements of western and Asian children and Liz Truss' trip to Shanghai to find out how to teach Maths properly. The article actually starts quite sensibly with a warning from the authors, apparently to Truss, about her jaunt and the pitfalls of copying apparently successful education systems before thinking things through:

“Yet in their rush to copy the winning formula of high-performing countries in east Asia, politicians risk drawing the wrong conclusions.

Schools in Shanghai are very different from those in Ms Truss’s constituency in southwest Norfolk. But not all of those differences play a role in Shanghai’s superior performance. Some are irrelevant. Some may even be harmful. And some will be idiosyncratic features of the school she happens to visit, rather than representative of the system.”

The authors even expand on this by pointing out that the gap between the best and worst students is larger in the educational holy lands than anywhere else in the world. So should we, which is what the authors must be implying about Shanghai, raise standards by focusing on the best students to drag up average outcomes? The answer appears to be a resounding no as there is apparently 'no evidence that the inequality of outcomes in east Asian schools contributes to raising standards'. It is with this, quite bizarre, statement that Sahlgren and Le Grand's argument begins to unravel.

Contrary to what is said a successful focus on any particular sub group of students will raise standards in some way if one presumes that the focus group is being taught better. The only argument against this is ethical which would be fine if the same ethical objections to inequality weren't subsequently dismissed by Sahlgren and Le Grand as an argument against competition between schools. Competition, you see, can 'force laggards to improve' – the assumption being that the laggards will improve faster than anyone else and close the gap accordingly. It takes some nerve to wag the inequality finger at high outcomes Shanghai while predicting their own medicine will raise standards and close the gap here yet this is exactly what the authors seem to be doing:

“One policy that can increase general standards of accomplishments while reducing inequality is injecting competition into the system.”

One would assume there is some evidence for all of this beyond the usual snake oil offered by educational economists about academies and charter schools and, to their credit, this is not lost on Sahlgren and Le Grand:

“Much of the debate centres around whether academies perform better than other schools on the most important measures of student attainment.

These arguments miss the point. It is difficult to be sure whether one school performs better than another because it is an academy, because it is funded more generously, or because it attracts students who would have performed well wherever they went to school. Pointing to the success of a small number of UK academies is no better than indiscriminately copying from the countries that have the highest grades.”

Kudos to that but the problem here is that Sahlgren and Le Grand's alternative evidence (West & Woessmann 2010) isn't much of an improvement. In short West & Woessmann have linked the 'the legacy of the Vatican's effort' in some countries which is 'competition in schools' to high attaining systems. If this is going to be useful how does one quickly recreate the legacy effects of a phenomena that started in the 19th century? And is a pseudo Protestant and pseudo Catholic education movement represented by corporate, modern academy chains even palatable in the first place? One might live with these practical doubts if the authors weren't guilty of yet more finger wagging this time about the virtue of 'copying the country with the highest grades' while justifying their own position on the basis of, er, high grades.

So what is one to make of this apparent cherry picking? Perhaps the message is don't copy the systems or schools with the highest grades unless there is a legacy of competition in that system in which case do copy because, hey guess what, those systems have the strongest attainment? Or perhaps the message is to avoid high attaining systems with high levels of educational inequality unless there is competition between the schools in which case, relax, the laggards are sure to catch up eventually? Sahlgren and Le Grand have a tendency for over optimism and, given their earlier criticism of 'highest grades', have not justified the evidence offered to support their position in enough detail. What is most depressing is the absence of any analysis at all about the impact of competition that already exists between schools in some Western countries. These things leave me with the impression that the article is little more than a thinly vieled broadside against education systems that have not adopted free market principles rather than a compelling argument for market based reform.


% 5 A*-C EM

One of the drawbacks for unaffiliated teachers who use the internet and social media for professional interaction is exposure to the virtual presence of edu-brands and ambitious members of the educational establishment. For most of the year their contributions seem limited to vague descriptions of visions and bland aspirational statements but it is results season, and the arrival of quantitative data, that herald a wave of insufferable self aggrandisement and promotion.
Although there are many performance measures that a school can use to quantify their examination results the most important (because of league tables), and by far the most popular, is the percentage of students gaining at least 5 grades at A*-C including English and Maths (% 5 A*-C EM) – the following is relatively toned down but a fairly typical communication:

Some communications are more visceral and use % 5 A*-C EM data to send implicit messages about the quality of a service or brand:
The worst communications are where % 5 A*-C EM data is used to add value to one brand whilst simultaneously devaluing the competition:
Screen shot 2013-08-22 at 15.05.25
There are several reasons to reject this sort of celebratory zeal. The main reason is that % 5 A*-C EM isn't very useful because it doesn't consider the prior attainment of a cohort and is easily manipulated through nefarious means. It is obvious then, that without a considerable amount of accompanying context, a schools % 5 A*-C EM tells us very little about its performance. The second reason is that it is unpalatable when one educational organisation starts declaring it's superiority over others – there are more than enough people queuing up to give less fortunate schools a kicking without edu-brands like Harris joining in.
This devotion to % 5 A*-C EM simply leads to schools using increasingly desperate, increasingly unethical methods so they can compete with each other over a number. The best we can say for the addicts is that they are well meaning but out of their depth; there are SMT out there who consider 5 A*-C EM to be, not a crude performance measure, but a form of pastoral care that is within the grasp of any 'deserving' child if their school is good enough. Such beliefs either undermine the value they attach to the data or cruelly suggest that intellectually weak children are 'undeserving' of anything beyond their own limitations. The very worst are charlatans who promote themselves using % 5 A*-C EM data believing, correctly, that many of those outside the educational establishment (and I include politicians in this group) are ignorant of its practices (for example boasting about high GCSE pass rates while neglecting to mention that a proportion of the cohort actually passed BTEC) and will accept the data at face value.