The Pareto principle and the great divide

I just scared myself looking at my last post...it was at the end of January! So far this academic year has seen just 5 posts to my blog - in previous years I averaged 1 post per week during term time. In fact I've just passed the 2 year anniversary of starting this blog and I've now written less since September 2014 than I have at any time since I started it. I think it's about time I explained why...

SLT role
Good grief it's busy as SLT! I wrote a post (here) back in October about my first month as SLT and the multitude of unexpected pressures that suck away at your time. In all honesty I've had to stop posting to my blog so regularly because I've simply been that busy that I couldn't justify any more time sat in front of a computer writing a blog post each week. I've wanted to post, in fact I've even started a decent number of them and then saved them in draft because I wasn't able to finish them to a point where I'd be happy to share them publicly. In fact half of this post was written months ago, but seems to fit with what I wanted to say today.

This isn't intended to be a whinge about workload or that kind of thing so don't worry about digging out the violins, however I do have some observations about the tensions caused by the SLT role.

Is there a great divide?
As I got closer to and then started my new role as assistant head and I became increasingly aware of the perceived divide between SLT and middle leaders/mainscale teachers. I know it's not just at my school, I've seen it at every school I've been in but it becomes really clear as you get up close to it and then jump across into SLT, I think it's particularly visible to me as I stayed at the same school.

It started with the jibes from my fellow Friday afternoon staff footballers who were counting down the number of games I was likely to be at before I become "too important" to play with them. I'm pleased to say this has stopped as I've carried on playing!

It continues with discussions about timetables where as an assistant head I teach less than half the number of lessons I did as a head of department, leading to jokey conversations about me having plenty of time to think up jobs & initiatives to keep the middle leaders and mainscale teachers busy.

The thing is the structure of a week for a member of SLT is so massively different to that of a middle leader they look like completely different jobs. Compare SLT to a classroom teacher and it's even more different - I actually teach only slightly more lessons than a main scale teacher gets for PPA.

I'm not necessarily saying its wrong, but it changes your perspective of the job massively. A marking policy that is completely manageable on an SLT timetable can become tough to manage as a middle leader, and completely impossible as a main scale teacher. Teaching a difficult group when you have loads of time to prepare/recover, plus a level of seniority to trade with, is very different to having a similar group amidst full days of teaching.

Of course much of the time not teaching as SLT is spent doing management & leadership functions, so the "loads of time" is a fallacy, but the perception is there from those outside of SLT even if it's not the reality.

I like to think I'm fairly approachable and try to make sure that I spend time talking to colleagues at all levels of the organisation, and have great working relationships across the school. A great part of that, and the fact that I wasn't always SLT at this school, means that I think some are more open and honest with me than they might be with the other members of the leadership team. I know for a fact that that some will say things to me that they wouldn't dream of saying to others in our SLT. This gives an interesting insight at times...

No time for perfection
I think the biggest actual disconnect I've discovered as part of all of this is the perception from many staff in general that all actions from SLT are deliberate, and that SLT have complete control over everything that happens in the school. Now I'm not suggesting that we go around blundering into things and have no control over what goes on, but we are a team of humans and with that comes limitations. Similarly we work in a setting that has a massive number of stakeholders with a vast array of needs and agendas, and are subject to legistative, judgemental, procedural and time constraints that limit or shape things in all manner of ways.

Sometimes the size of a team means that not everyone can be consulted properly in advance. Sometimes the speed a decision needs to be made means that only the most central consequences can be considered (see comments about Pareto below). Sometimes an element of planning falls between areas of responsibility meaning it gets missed. Sometimes a task is simply not done quite as perfectly as it might have been because a human has done it or they ran out of time. All of these are issues to be guarded against, and even planned to be mitigated, but none would be done deliberately. However I know from that the consequences of what I know to be a small human error at SLT level can be seen as a direct decision to undermine or cause issues for other staff.

I've seen a situation where a given member of SLT has been working as hard as they could on something, but due to the realities of life in schools it ended being delivered to a wider audience slightly before it was completely finished. The reception from others in the school was frosty "it's a half baked Idea", "they've not considered the impact on...", "why couldn't they have told us that weeks ago?" and so on. The expectation from the school as a body is that the SLT have all the answers and have the time to plan everything out fully. The reality is that there is so much going on that there is too often no time to complete any task fully, sometimes it just has to be good enough to get the job done.

Pareto principle for leadership
If you've not heard of the Pareto principle it stems from the observations by Italian Economist Vilfredo Pareto that 20% of Italians own 80% of the land. This has been extended to business in various ways with assertions that 80% of profit comes from 20% of your clients, or 80% of sales come from 20% of the sales team. It is also used in health and safety where 80% of injuries come from the 20% most common incidents and so on.

In my experience it can be fairly safely applied that about 80% of the behaviour incidents in a school come from just 20% of the students (you know which ones!). Similarly about 80% of the high grades come from 20% of the students. 80% of absences come from about 20% of staff, I could go on...

Furthermore another aspect of Pareto in terms of leadership is that if you consider 20% of the stakeholders in a given decision then you'll probably expose 80% of the potential issues. Due to time pressures and general day to day management constraints it is common for leaders to have to revert to this 80/20 rule in all sorts of situations in order to see the wood from the trees. Many leaders do this unknowingly, or some knowingly but rarely is it applied in a cold, deliberate way, however the Pareto Principle can be used to prioritise all sorts of things in all sorts of ways. Yes it's a rule of thumb but it actually fits reasonably well in many situations, and does force a level of perspective that can help get the job done.

Of course I'm not saying it is a good thing to be forced to prioritise like this, and certainly if you're one of the 80% of the stakeholders that is not consulted who then raises one of the 20% issues then I completely understand that you'd feel aggrieved, but that's not really what this is about. What I'm trying to say is that SLT sometimes set themselves up a little as infallible, and are often expected to perform that way by wider staff. However often what they are doing is their best given the situation they are in and the conflicting demands they have on their time. Often this means prioritising and that then leads to some people feeling overlooked.

For SLT the key thing is to acknowledge that we're doing this and to communicate more clearly with those involved. Be willing to acknowledge that at the bottom of all decisions is a human that has done their best, but may not have done it perfectly. For those outside of SLT looking in, perhaps consider if it was physically possible to do it better, or if the compromise you need to make is actually the right one for overall progress (are you one of the 80% or the 20%).

Yes SLT are paid more in order to make decisions, yes SLT have more time in the week in order to make decisions, but that doesn't change the laws of physics, and it certainly doesn't make anyone perfect or infallable. Time is finite for all of us. We all have constriants to manage, and differing perspectives.

I think the biggest thing I've learnt as a member of SLT is that I can't always take the time to be a perfectionist, but I can do the best I can given the time and resources I have available.

No, I'm wrong... the BIGGEST thing I've learnt is that once I've had to commit to something that I know might not be perfect I need to avoid beating myself up over it and constantly revisiting it. I've spent too long doing that at various points during the year and it's done me no favours. You don't needs to be perfect in order to get it right for the vast majority of the time; the important thing is to make sure that even if something isn't quite right it is still one of the least bad options!

This is just me rambling again - sharing the thoughts bouncing round my head - comments always welcome!

RAG123 as defined by the students

Our head of maths has done a piece of work recently on RAG123 that I think I just have to share...

Firstly if you've never heard of RAG123 then look here for my original post on it, and then see here for all the others I've done...

Pupil voice
So far with RAG123 I've seen teacher definitions of what R,A,G mean, and what 1, 2, 3 mean, and we've occasionally had a go at doing a student oriented set of descriptors. However it surprises me to admit it but we've never previously asked the students to define it themselves!

With my stepping up to SLT this year I have had the pleasure of welcoming and line managing a new HoD to run the maths department. Simon Preston arrived at our school, inherited RAG123 from me and then he embraced it and used it for a while. Then he had a brainwave that is so obvious I don't know why nobody had thought of it before... He asked the students to define their understanding of all the RAG123 ratings...

Simon did this by issuing a sheet like the one below and asked the students to fill in the boxes with a description of what types of work or attitudes warranted each rating. Notably he didn't just ask them to define R, A, G and then 1, 2, 3, but he got them to define all 9 combinations of letters and numbers.

What did the students say?
I have been fascinated by the responses that the students gave. Having collated their inputs and drawn together the common themes Simon has compiled the following grid, which for me now seems to be the definitive RAG123 rating grid.

I think the nuance that the students have highlighted between R2 and R3 in the root cause for the low effort is really interesting. Also like the A1 "just enough". Overall I am really pleased by the clear linkage between effort and understanding. It all comes back to the basic position where students on 3 for understanding need clear input from the teacher to move them, and those on R for effort also need a decision from the student to improve.

Involving parents
Also this week we held a parent information evening for our yr 11 students where we were briefing them on revision techniques and ideas to improve home-school partnerships. This RAG123 grid was shared with parents and students in this session. We suggested that parents could work with students to RAG123 their revision processes at home in order to help figure out whether a session was effective or not. This was really well received and we have had several positive comments from parents about this giving them the tools to help review progress with revision, particularly in subjects that they have no expertise in.

Have you done something similar?
The idea of asking the students is so obvious I'm amazed I or someone else haven't already done it - does anyone else have a similar student perspective on RAG123? If you have I'd be really keen to see it.

Once again - if you've not tried RAG123 you don't know what you're missing in terms of building linkage between marking and planning, building dialogue with students and the promoting growth mindset type linkages between effort and progress. Give it a try and let me know how it goes!

A ragging birthday!

Time flies!

Almost exactly a year ago I wrote my first blog post about RAG123 (find it here) which followed a single week trial of an idea that seemed illogical... Mark more often, but write less, improve feedback and reduce workload. Remarkably it worked - students responded positively, I felt more in control of my marking workload and my lessons were more effective. I still haven't taken a single pupil book home to mark since I started RAG123 over a year ago, but ALL of my books are marked up to date.

I've since written loads of posts on RAG123 (all found here), and tweeted prolifically on it over the past year. I know I am guilty of being a bit evangelical about it, but I do feel justified in my enthusiasm. The evidence suggests that using this approach to marking and feedback (and planning) really does have a beneficial impact both on the students and the teachers involved. I get fantastic feedback like this on a regular basis:

No going back!

As I know I can be a bit biased on this throughout the year using and developing RAG123 I have regularly asked for negative feedback or stories of people that have tried RAG123 but stopped. From the responses I have received there are only a couple of people that have stopped once tried. In these cases it was never because they didn't think RAG123 was beneficial, it was due to some external factor such as illness or a change in role. In all cases where someone said they'd stopped they followed up with a comment that they would start again as soon as their circumstances allowed. I remain open and receptive to constructive criticism of rag123 and want to retain balance on it. To be honest though negatives only really come from people who have never tried it, or haven't really understood the idea. To date the overwhelming evidence is that once you try it you will see such benefits that you won't want to go back. 

Going national and international
As well as individual teachers using RAG123, there are whole departments adopting it, and I know of a couple of schools that have adopted RAG123 as a central part of their marking policies (one has even reported it to me as a contributing factor in their schools journey out of special measures). I'm constantly being contacted by people who are sharing it within their departments, their schools or via teachmeets across the country. In fact it's also gone international, and not just in English speaking countries. I know it's been translated into Welsh (#COG123), and also is in the process of translation into Swedish...
 

So... a year later what have I learnt?
I've written, thought and learnt a lot about RAG123 over the past year. While the core idea remains exactly as described in the original post, there are a number of subtleties that I have seen and picked up over the last year. I've probably tweeted most of them at some point or other, but it's also about time I shared them all in one place. Along the way there are a couple of confessions I should make too...

Top 10 tips to get the most out of RAG123:

1. There are no strict rules for RAG123! Each teacher should take the core principle and make it work for them, their students, their school, their workload.

It makes no real difference if you use the colours or numbers for understanding or effort. It also doesn't matter if you need more than 3 levels for each aspect to fit with some other system (I know of at least one RAG1234 system being used, and there is also a RAGB123 out there). Actually you could call it anything, ABC123 would work just as well

However I do personally think colours are emotive and therefore can add to impact, which is why my preference remains RAG for effort as that's the bit I want the students to identify with the most (though for a cautionary note on colours see point 5).

2. While the process I put forward for RAG123 involves marking every day, there is no actual necessity to mark every day or every lesson. However without a doubt the more often you can manage it the more effective it will be.

Personally I try to RAG123 between every lesson but don't manage it all the time (still true even now I'm on an apparently empty SLT timetable). What you gain from doing it after every lesson is the opportunity for RAG123 to feed into planning for the next lesson, thereby improving differentiation and the impact of the next phase of teaching (for more on RAG123 as formative planning see here). I now find it much harder to plan if I've not had chance to RAG123 my books.

3. It's the 2 dimensional nature of RAG123 that brings its strength. Separating effort (student controlled) from understanding (teacher influenced) is really important.

If a student is not trying then even the best teacher will struggle to help them learn. Conversely if the student is working as hard as they can but not learning then it is the teacher that needs to do something different. This is why it's simply not the same as a plain traffic light assessment of understanding (more on that in this post). Highlighting the impact of their effort is important to students and makes direct links with other powerful things like growth mindset.

I often get asked how to measure effort, or how I decide exactly what constitutes a "green" "amber" or "red" effort? My answer is always the same - the rating should be scaled to the message you want that individual student to receive. If you think they're cruising then it's amber, if they're going flat out then it's green. It doesn't matter that one student has done half a page vs another doing four pages... If you know from the lesson that the half page struggled and persisted the whole lesson then it's green, if the four pages are all well within the ability of the student then it's amber. The brightest, best behaved students can certainly get reds if they are cruising (and they really don't like it so improve almost instantly!).

4. RAG123 doesn't and can't completely replace more detailed feedback, and I've never said that it should. Students need this, you still need to write extra at times. To help this it's good practice to aim to write an extra comment in 10-15% of books each time you mark. This hardly takes any extra time and after a week or so you can easily cover the whole class. Alternatively perhaps that feedback is verbal - which is fine too, though fails a little fouler of the dreaded "evidence for inspection". For me if you and the students are able to talk to an inspector about the feedback given (verbal or otherwise) and how it helps them to improve then that's perfectly valid feedback, but I do acknowledge that it takes a bit of confidence to fly without the safety net of written evidence. 

5. There is likely to be a colourblind student in every class group.... This was a big penny that dropped part way through the year, and I give thanks to @colourblindorg for the pointers on this. Clearly this causes tension for a system that has colours at its heart. However there is NO barrier to using RAG123 with colourblind students so long as symbols (e.g. "R", "A" or "G") are used and not simply coloured blobs/dots or even different coloured ink. Colourblindness is a big limitation for the various "purple pen of progress" or "green for good, pink to think" concepts that abound across teaching policies and #chat discussions. Using different coloured pens becomes irrelevant if colourblind students (and teachers) can't reliably tell the difference.

Colourblindness can render unlabelled R,A,G unintelligible to an average of 1 student in every classroom

The key message here is that all RAG123 posters, stickers, guidance must always have a way for colourblind people to distinguish between the colour designations - simply labelling R, A, G does this perfectly. Colours are still powerful and useful for the non-colourblind majority so I'm still in favour of using colours, but it's important we make them accessible to those that can't distinguish between them.

Just labelling R,A,G as shown above retains full accessibility for colourblind students.

6. RAG123 is absolutely a leap of faith, and sceptics take a lot of convincing!
Perhaps my biggest confession here is that despite sharing RAG123 nationally (& internationally) and even having it adopted by whole schools in other parts of the country I have not yet got it embedded across my school, or even widely used outside of the maths department.

The reasons for this are many... Perhaps I have been  being a little more shy about pushing RAG123 within my school with people who may not be actively looking for new ideas (compared to people at teachmeets, on Twitter or reading blogs who are clearly looking for and open to new ideas). There's also the fact that until September I was 'only' a head of maths and my influence only reached so far within school. Even now I'm on SLT there is someone else on the team that has the clear remit of improving marking and feedback and I don't want to step on their toes. I've spoken to them about it and actually they like the idea, but can't quite build it into a whole school position yet due to other priorities. While I do find this a little frustrating I want to emphasise that this is not a criticism of my colleague(s) across my school. They are all working immensely hard and have a real desire to do the best for the children in our care, they simply choose to do this in a different way to me and I have yet to fully do he hard sell on RAG123.

This also in no way suggests that I don't have faith in RAG123. Personally I feel my teaching would suffer massively if I had to stop, and think most people's teaching would benefit from adopting it, but I also recognise that change is difficult and it's not easy to try something like this. I know I'm not the only one that faces this challenge, Damian Benney who is the author of probably the second most read blog about RAG123 is a Deputy Head at his school but has struggled to get colleagues to try it, as detailed here. We're both completely sold on RAG123, and have had success sharing it across the country but changing minds more locally can be really hard.

7. Students need support with RAG123 to make the self reflection aspect meaningful. I've written before about how difficult reflection is so won't go into it again for this post (find more here and here), however I will emphasise that the provision of sentence starters or other scaffolding to prompt more meaningful comments really does help. It's also vital that students are given the time in lesson to review and respond to comments - if you don't demonstrate it's important they won't treat it as important.

8. Relating to the last sentence in the paragraph above... Marking & reviewing books as regularly as using RAG123 allows becomes a really powerful way to demonstrate to the students that you care what they do every lesson. This is a big point and shouldn't be underestimated. There are groups of students who don't like RAG123, when you ask them it's usually because they have nowhere to hide in terms of effort. The vast majority of students REALLY like RAG123, when you ask them it's because they know for certain that the teacher is taking an interest in what they do each day.

9. Even bad RAG123 is still quite good. I'll be absolutely honest, compared to the examples I've seen on Twitter my own practice of RAG123 is nowhere near the level that some people have adopted. In all honesty I don't know where some of the teachers that do this find the time to do anything other than school work, maybe they don't? The detail some go into with RAG123 marking is almost to the level you'd expect from a more traditional marking methodology. For me this is awesome but a little overwhelming and I wouldn't want others to think that if they can't sustain that level they are doing it badly.

What I do know is that my books are basically marked and I know the students in front of me extremely well as a result of talking to them in lessons and using RAG123 with them regularly. I also know that the lessons I plan are tuned to the progress that the students make each lesson, and therefore the marking that I do isn't pointless (see more on my thoughts about pointless marking here). I'll gladly argue my case that the progress students make is evidence that my marking and feedback is effective, even if it only results in a better planned next lesson rather than reams of written evidence in books. This will be a contentious point for many, and some may disagree completely, but that's true of so many aspects of teaching.

10. RAG123 as with all good teaching simply comes down to promoting good levels of effort from the students and good planning from the teacher. Initial users of RAG123 will often ask if a student can get a R1 (low effort, excellent understanding), or a G3 (high effort, low understanding). The answer in both cases is of course they can. For me the effort ratings should provoke the students to question what they are doing (can they try harder, can they maintain their current effort across a sequence of lessons) and the understanding should provoke the teacher to question their support/extension/differentiation for the student or planning for the class as a whole.

RAG123 and the future
So a year in and what's next. For me it's simply keeping using RAG123, I would be a worse teacher without it; I know other users feel the same. 

Sceptics will often ask for evidence that it works before trying it. I understand this but am also frustrated by it. I've tried to put together some evidence (see here) but it gets confounded by other factors, and as a result the relatively small sample size and other influences makes this limited sample ripe for taking shots at in terms of robustness of data. To accumulate enough hard data to support it (with a robust control group for comparison) would take a spectacularly long time and frankly I think it's simpler than that...

• RAG123 costs nothing - there are no subscription fees!
• RAG123 can be started and stopped overnight, all it takes is a decision to do it.

As such I'll reiterate the challenge that I issue whenever I present this at a Teachmeet... Try RAG123 with a class for 2 weeks. If you don't see a benefit then stop... If you do stop then that's absolutely fair enough, but please get in touch to tell me why as I'm keen to understand if it has limitations! Similarly if you find it useful then please spread the word by challenging others!

Comments are always welcome, happy Ragging!

Managing with colours - SLTeachmeet presentation

These are the slides I presented at #SLTeachmeet earlier today. Click here

The info shared in the presentation picks up on aspects covered in these posts:
Using measures to improve performance

Using seating plans with student data

RAG123 basics

As always feedback is always welcome...

Powerful percentages

Numbers are powerful, statistics are powerful, but they must be used correctly and responsibly. Leaders need to use data to help take decisions and measure progress, but leaders also need to make sure that they know where limitations creep into data, particularly when it's processed into summary figures.

This links quite closely to this post by David Didau (@Learningspy) where he discusses availability bias - i.e. being biased because you're using the data that is available rather than thinking about it more deeply.

As part of this there is an important misuse of percentages that as a maths teacher I feel the need to highlight... basically when you turn raw numbers into percentages it can add weight to them, but sometimes this weight is undeserved...

Percentages can end up being discrete measures dressed up as continuous
Quick reminder of GCSE data types - Discrete data is in chunks, it can't take values between particular points. Classic examples might be shoe sizes where there is no measure between size 9 or size 10, or favourite flavours of crisps where there is no mid point between Cheese & Onion or Smoky Bacon.

Continuous data can have sub divisions inserted between them, for example a measure of height could be in metres, centimetres, millimetres and so on - it can keep on being divided.

The problem with percentages is that they look continuous - you can quote 27%, 34.5%, 93.2453%. However the data used to calculate the percentage actually imposes discrete limits to the possible outcome. A sample of 1 can only have a result of 0% or 100%, a sample of 2 can only result in 0%, 50% or 100%, 3 can only give 0%, 33.3%, 66.7% or 100%, and so on. Even with 200 data points you can only have 201 separate percentage value outputs - it's not really continuous unless you get to massive samples.

It LOOKS continuous and is talked about like a continuous measure, but it is actually often discrete and determined by the sample that you are working with.

Percentages as discrete data makes setting targets difficult for small groups
Picture a school that sets an overall target that at least 80% of students in a particular category (receipt of pupil premium, SEN needs, whatever else) are expected to meet or exceed expected progress.

In this hypothetical school there are three equivalent classes, let's call them A, B and C. In class A we can calculate that 50% of these students are making expected progress; in class B it's 100%, and in class C it's 0%. On face value Class A is 30% behind target, B is 20% ahead and C is 80% behind, but that's completely misleading...

Class A has two students in this category, one is making expected progress, the other isn't. As such it's impossible to meet the 80% target in this class - the only options are 0%, 50% or 100%. If the whole school target at 80% accepts that some students may not reach expected progress then by definition you have to accept that 50% might be on target for this specific class. You might argue that 80% is closer to 100% so that should be the target for this class, but that means that this teacher as to achieve 100% where the whole school is only aiming at 80%! The school has room for error but this class doesn't! To suggest that this teacher is underperforming because they haven't hit 100% is unfair. Here the percentage has completely confused the issue, when what's really important is whether these 2 individuals are learning as well as they can?

Class B and C might each have only one student in this category. But it doesn't mean that the teacher of class B is better than that of class C. In class B the student's category happens to have no significant impact on their learning in that subject, they progress alongside the rest of the class with no issues, with no specific extra input from the teacher. In class C the student is also a young carer and misses extended periods from school; when present they work well but there are gaps in their knowledge due to absences that even the best teacher will struggle to fill. To suggest that either teacher is more successful than the other on the basis of this data is completely misleading as the detailed status of individual students is far more significant.

What this is intended to illustrate is that taking a target for a large population of students and applying it to much smaller subsets can cause real issues. Maybe the 80% works at a whole school level, but surely it makes much more sense at a class level to talk about the individual students rather than reducing them to a misleading percentage?

Percentage amplifies small populations into large ones
Simply because percent means "per hundred" we start to picture large numbers. When we state that 67% of books reviewed have been marked in the last two weeks it conjures up images of 67 books out of 100. However that statistic could have been arrived at having only reviewed 3 books, 2 of which had been marked recently. The percentage give no indication of the true sample size, and therefore 67% could hide the fact that the next step better could be 100%!

If the following month the same measure is quoted as having jumped to 75% it looks like a big improvement, but it could simply be 9 out of 12 this time, compared to 8 out of 12 the previous month.  Arithmetically the percentages are correct (given rounding), but the apparent step change from 67% to 75% is actually far less impressive when described as 8/12 vs 9/12. As a percentage it suggests a big move in the population; as a fraction it means only one more meeting the measure.

You can get a similar issue if a school is grading lessons/teaching and reports 72% good or better in one round of reviews, and then sees 84% in the next. (Many schools are still doing this type of grading and summary, I'm not going to debate the rights and wrongs here - there are other places for that). However the 72% is the result of 18 good or better out of 25 seen, the 84% is the result of 21 out of 25. So the 12% point jump is due to just 3 teachers flipping from one grade to the next.

Basically when your population is below 100 an individual piece of data is worth more than 1% and it's vital not to forget this. Quoting a small population as a percentage amplifies any apparent changes, and this effect increases as the population size shrinks. The smaller your population the bigger the amplification. So with a small population a positive change looks more positive as a percentage, and a negative change looks more negative as a percentage.

Being able to calculate a percentage doesn't mean you should
I guess to some extent I'm talking about an aspect of numeracy that gets overlooked. The view could be that if you know the arithmetic method for calculating a percentage then so long as you do that calculation correctly then the numbers are right. Logic follows that if the numbers are right then any decisions based on them must be right too. But this doesn't work.

The numbers might be correct but the decision may be flawed. Comparing this to a literacy example might help. I can write a sentence that is correct grammatically, but that does not mean the sentence must be true. The words can be spelled correctly, in the correct order and punctuation might be flawless. However the meaning of the sentence could be completely incorrect. (I appreciate that there might be some irony in that I may have made unwitting errors in this sentence about grammar - corrections welcome!)

For percentage calculations then the numbers may well be correct arithmetically but we always need to check the nature of the data that was used to generate these numbers and be aware of the limitations to the data. Taking decisions while ignoring these limitations significantly harms the quality of the decision.

Other sources of confusion
None of the above deals with variability or reliability in the measures used as part of your sample, but that's important too. If your survey of books could have given a slightly different result if you'd chosen different books, different students or different teachers then there is an inherent lack of repeatability to the data. If you're reporting a change between two tests then anything within test to test variation simply can't be assumed to be a real difference. Apparent movements of 50% or more could be statistically insignificant if the process used to collect the data is unreliable. Again the numbers may be arithmetically sound, but the statistical conclusion may not be.

Draw conclusions with caution
So what I'm really trying to say is that the next time someone starts talking about percentages try to look past the data and make sure that it makes sense to summarise it as a percentage. Make sure you understand what discrete limitations the population size has imposed, and try to get a feel for how sensitive the percentage figures are to small changes in the results.

By all means use percentages, but use them consciously with knowledge of their limitations.


As always - all thoughts/comments welcome...

RAG123 is not the same as traffic lights

I've written regularly about RAG123 since November 2013 and since starting it as an initial trial in November I still view it as the single most important thing I've discovered as a teacher. It's now absolutely central to my teaching practice, but I do fear that at times people misunderstand what RAG123 is all about. They see the colours and they decide it is just another version of traffic lighting or thumbs up/across/down AFL. I'm sure it gets dismissed as "lazy marking", but the reality is that it is much, much more than marking.

As an example of this uncertainty of RAG123 at a surface level without really understanding the depth I was recently directed to the Ofsted document "Mathematics made to measure" found here. I'd read this document some time ago and it is certainly a worthwhile read for anyone in a maths department, particularly leading/managing the subject, but it may well provide useful thoughts to those with other specialisms. There is a section (paragraphs 88-99) that are presented under the subheading "Marking: the importance of getting it right" - it was suggested to me that RAG123 fell foul of the good practice recommended in these paragraphs, even explicitly criticised as traffic lighting and as such isn't a good approach to follow.

Having read the document again I actually see RAG123 as fully in line with the recommendations of good practice in the Ofsted document and I'd like to try and explain why....

The paragraphs below (incl paragraph numbers) are cut & pasted directly from the Ofsted document (italics), my responses are shown in bold:

88. Inconsistency in the quality, frequency and usefulness of teachers’ marking is a 
perennial concern. The best marking noted during the survey gave pupils 
insight into their errors, distinguishing between slips and misunderstanding, and 
pupils took notice of and learnt from the feedback. Where work was all correct, 
a further question or challenge was occasionally presented and, in the best 
examples, this developed into a dialogue between teacher and pupil. 
RAG123 gives a consistent quality, and frequency to marking. Errors and misunderstandings seen in a RAG123 review can be addressed either in marking or through adjustments to the planning for the next lesson. The speed of turnaround between work done, marking done/feedback given, pupil response, follow up review by teacher means that real dialogue can happen in marking.

89. More commonly, comments written in pupils’ books by teachers related either 
to the quantity of work completed or its presentation. Too little marking 
indicated the way forward or provided useful pointers for improvement. The 
weakest practice was generally in secondary schools where cursory ticks on 
most pages showed that the work had been seen by the teacher. This was 
occasionally in line with a department’s marking policy, but it implied that work 
was correct when that was not always the case. In some instances, pupils’ 
classwork was never marked or checked by the teacher. As a result, pupils can 
develop very bad habits of presentation and be unclear about which work is 
correct.
With RAG123 ALL work is seen by the teacher - there is no space for bad habits to develop or persist. While it can be that the effort grading could be linked to quantity or presentation it should also be shaped by the effort that the teacher observed in the lesson. Written comments/corrections may not be present in all books but corrections can be applied in the next lesson without the need for the teacher to write loads down. This can be achieved in various ways, from 1:1 discussion to changing the whole lesson plan.

90. A similar concern emerged around the frequent use of online software which 
requires pupils to input answers only. Although teachers were able to keep 
track of classwork and homework completed and had information about 
stronger and weaker areas of pupils’ work, no attention was given to how well 
the work was set out, or whether correct methods and notation were used.
Irrelevant to RAG123

91. Teachers may have 30 or more sets of homework to mark, so looking at the 
detail and writing helpful comments or pointers for the way forward is time 
consuming. However, the most valuable marking enables pupils to overcome 
errors or difficulties, and deepen their understanding.
Combining RAG123 with targeted follow up/DIRT does exactly this in an efficient way.

Paragraphs 92 & 93 simply refer to examples given in the report and aren't relevant here.

94. Some marking did not distinguish between types of errors and, occasionally, 

correct work was marked as wrong.

Always a risk in all marking, RAG123 is not immune, but neither is any other marking. However given that RAG123 only focuses on a single lesson's work the quantity is smaller so there is a greater change that variations in student's work will be seen and addressed.

95. At other times, teachers gave insufficient attention to correcting pupils’ 

mathematical presentation, for instance, when 6 ÷ 54 was written incorrectly 

instead of 54 ÷ 6, or the incorrect use of the equals sign in the solution of an 

equation.

Again a risk in all marking and RAG123 is not immune, but it does give the opportunity for frequent and repeated corrections/highlighting of these errors so that they don't become habits.

96. Most marking by pupils of their own work was done when the teacher read out 

the answers to exercises or took answers from other members of the class. 

Sometimes, pupils were expected to check their answers against those in the 

back of the text book. In each of these circumstances, attention was rarely paid 

to the source of any errors, for example when a pupil made a sign error while 

expanding brackets and another omitted to write down the ‘0’ place holder in a 

long multiplication calculation. When classwork was not marked by the teacher 

or pupil, mistakes were unnoticed.

With RAG123 ALL work is seen by the teacher - they can look at incorrect work and determine what the error was, either addressing it directly with the student or if it is widespread taking action at whole class level.

97. The involvement of pupils in self-assessment was a strong feature of the most 

effective assessment practice. For instance, in one school, Year 4 pupils 

completed their self-assessments using ‘I can …’ statements and selected their 

own curricular targets such as ‘add and subtract two-digit numbers mentally’ 

and ‘solve 1 and 2 step problems’. Subsequent work provided opportunities for 

pupils to work on these aspects. 

The best use of RAG123 asks students to self assess with a reason for their rating. Teachers can review/respond and shape these self assessments in a very dynamic way due to the speed of turnaround. It also gives a direct chance to follow up by linking to DIRT

98. An unhelpful reliance on self-assessment of learning by pupils was prevalent in 

some of the schools. In plenary sessions at the end of lessons, teachers 

typically revisited the learning objectives, and asked pupils to assess their own 

understanding, often through ‘thumbs’, ‘smiley faces’ or traffic lights. However, 

such assessment was often superficial and may be unreliable.

Assessment of EFFORT as well as understanding in RAG123 is very different to these single dimension assessments. I agree that sometimes the understanding bit is unreliable. However with RAG123 the teacher reviews and changes the pupil's RAG123 rating based on the work done/seen in class. As such it becomes more accurate once reviewed. Also the reliability is often improved by by asking students to explain why they deserve that rating. The effort bit is vital though... If a student is trying as hard as they can (G) then it is the teacher's responsibility to ensure that they gain understanding. If a student is only partially trying (A) then the teacher's impact will be limited. If a student is not trying at all (R) then even the most awesome teacher will not be able to improve their understanding. By highlighting and taking action on the effort side it emphasises the student's key input to the learning process. While traffic lights may very well be ineffective as a single shot self assessment of understanding, when used as a metaphor for likely progress given RAG effort levels then Green certainly is Go, and Red certainly is stop.

99. Rather than asking pupils at the end of the lesson to indicate how well they had 

met learning objectives, some effective teachers set a problem which would 

confirm pupils’ learning if solved correctly or pick up any remaining lack of 

understanding. One teacher, having discussed briefly what had been learnt with 

the class, gave each pupil a couple of questions on pre-prepared cards. She 

took the cards in as the pupils left the room and used their answers to inform 

the next day’s lesson planning. Very occasionally, a teacher used the plenary 

imaginatively to set a challenging problem with the intention that pupils should 

think about it ready for the start of new learning in the next lesson. 

This is an aspect of good practice that can be applied completely alongside RAG123, in fact the "use to inform the next day's lesson planning" is something that is baked in with daily RAG123 - by knowing exactly the written output from one lesson you are MUCH more likely to take account of it in the next one.

So there you have it - I see RAG123 as entirely in line with all the aspects of best practice identified here. Don't let the traffic light wording confuse you - RAG123 as deployed properly isn't anything like a single dimension traffic light self assessment - it just might share the colours. If you don't like the colours and can't get past that bit then define it as ABC123 instead - it'll still be just as effective and it'll still be the best thing you've done in teaching!

All comments welcome as ever!

SOLO to open up closed questions

I've been dabbling with SOLO for a while now, it's been part of bits of my practice (see here, here and here) but I've yet to really embed it in all lessons as fully as I would have liked. I have used it as a problem solving tool, or to help structure revision, but not really deployed SOLO on a more day to day basis, and I want to change that.

I recently completed an interview for an Assistant Head position and as part of that was asked to teach a PSE lesson. This took me well out of my Maths comfort zone, so I had to give the planning deeper consideration than a maths lesson might have. After some thought I decided to introduce SOLO as part of the lesson, and it worked really well...

SOLO as a structure for discussion
I was teaching this PSE lesson to a group of year 7 students that I had never taught before and I knew that they had never seen SOLO before. As such a bit of my lesson needed to become an intro to SOLO. Fortunately the symbols are so intuitive that once I'd suggested that a single dot (Prestructural in SOLO terminology) meant you basically knew nothing about a topic, and a single bar (Unistructural) meant you knew something about it, the students were able to develop their own really good working definitions for Multistructural, Relational and Extended Abstract:

Once they had defined this hierarchy I could refer back to it at any point in the lesson and they knew what I was talking about. As such when I asked a question and the student responded with an answer I could categorise their response using the SOLO icons, such as "one bar," "three bar," "linked bar." If the student gave a "one bar" response I then asked them, or asked another student what was needed to make it a "three bar" response, and so on.

I was really pleased with how natural the discussion became, escalating up to really high level answers in a structured way. Similarly the students could use the same method with each other to improve their written answers through peer and self assessment. It even gives an easy way to open up a closed question question... For example:

T: "Name a famous leader"

P: "Nelson Mandela"

T: "What type of answer is that?"

P: "It's just a fact so it's got to be One bar"

T: "How could the answer be improved?"

P: "Give more facts about him, like that he led South Africa, or say why he was famous"

T: "Can you improve that further?"

P: "Maybe make links to other countries or compare him to other leaders"

T: "Fantastic, work on that with your partner..."

Rightly or wrongly I have a feeling that the opportunity for this type of discussion is much more common in a subject like PSE, and the SOLO linkage is much clearer as a result, however it got me thinking about how this approach could be used in the same way for Maths...

SOLO vs closed questions
A constant battle for maths teachers is the old "there is only one right answer in maths." Now of course that may be true in terms of a numerical value, but that ignores the process followed to achieve that answer, and often there are many mathematically correct processes that lead to the same final answer. In more open ended activities there may also be multiple numerical answers that are "right."

In maths we constantly battle to get students to write down more than their final answer and to show their full method. Following my experience of using SOLO for PSE I started thinking about how to use it to break down the closed answers we encounter in maths. As such I've put this together as a starting point...

The pupil response could be something that is seen written down in their working, or something that they say verbally during discussion. The possible teacher response gives a suggestion of how to encourage a higher quality of response to this and future answers. This could be part of a RAG123 type marking (see here for more info on RAG123), verbal feedback, or any other feedback process.

An alternative is to use it for peer/self assessment, again to encourage progress from closed, factual answers, to fuller, clearer answers:

 I realise I may be diluting or slightly misappropriating the SOLO symbols a little, e.g. is the top description above truly Extended Abstract or is it actually only Relational? In truth I don't think that distinction matters in this application - it's about enabling students to improve rather than assigning strict categories.

Proof in the pudding
The assessment ladder is part of a lesson plan for Tuesday, and I am going to try and use the pupil response grid throughout the week to help open up questions and encourage students to think more deeply about the answers - watch this space for updates.

As always - all thoughts & comments welcome.

Policies not straightjackets

I'm starting to lose track of the number of times I've heard or seen people say that they can't do or try something because it's out of line with their school or department policy. It really worries me when I hear that - it means they feel unable to innovate or experiment with something that could be an improvement.

Most often for me it's linked with RAG123, but I've seen it at other times in school, and all over the place on twitter too. It normally goes something like this:

• Person A: "Why not try this (insert suggested alternative pedagogical approach here)?"
• Person B: "That sounds great and I'd love to, but our policy for (same general area of pedagogy) means I can't try it."

Frustratingly this is usually where the discussion ends - the opportunity for person B to try something new that might improve their practice and improve outcomes for their students is squashed.

More specific examples I've actually seen/heard over the years include:
A: "For that lesson why not try using a big open ended question as your learning objective that all students work towards answering?"
B: "I can't because we're required to have 'must, should, could' learning objectives for all lessons"

A: "Could you re-arrange the tables in you room to help establish control with that difficult group? Perhaps break up the desks to break up the talking groups?"
B: "No because our department policy says we have to have the tables in groups to encourage group work."

A: "Why not try RAG123 marking?"
B: "I can't because our marking policy requires written formative comments only."

What are policies for anyway?
Policies should be there to provide a framework of good basic practice that all in a given organisation can use as a bare minimum to baseline their practice. However there is a difference between a framework to guide and a set of rules to be applied rigidly.

For example a policy that says that learning objectives must include suitable differentiation for the class being taught is substantially different to saying that all lessons are required to have Must, Should, Could learning objectives. One is  the essence of what we really want, the other is a single, rigid example of how this might be achieved. One allows the teacher to use their professional judgement to set objectives in a way that is appropriate for their relationship with that class and the material being taught; the other applies a blanket approach that assumes that every lesson by every teacher with every class is best set up in an identical fashion.

For me policies should set out a standard that is the bare minimum to ensure that the students get a good deal in that aspect. For example if a teacher is unsure of how often to mark their books the policy should clarify the minimum requirement, it should also detail what minimum information is needed in order for it to count as good marking.

However policies should never stifle innovation. Should never prevent the trial of something that could be even better. They also shouldn't dictate set structures that can't be deviated from under any circumstances - it should always be allowed to do it better than laid down in the policy!

Teachers as professionals should always have the option to deviate from the policy if it will produce better outcomes for their students in that particular situation (and if this becomes a consistent improvement then perhaps the policy should change to incorporate the deviation so that everyone benefits). However as professionals they should be both able and willing to justify a decision like this if questioned. Similarly if they have deviated from policy to try something that turns out to have not been so good then as professionals they should acknowledge this and return to the policy.

Consistency not uniformity
The bottom line is that policies should ensure a consistency in quality of experience, which mustn't be confused with a uniformity of experience. Quality in education is about high standards, high expectations and about professionals making informed decisions about how to get the best from the students in front of them. Quality is not about every teacher doing exactly the same thing in exactly the same way, if it was we could record model lessons and just play them to students, or just learn scripts to follow.

Uniformity and rigidity isn't the answer to the multi-faceted challenge that teaching presents; we can't always assume that one size fits all. Therefore policies should never be straightjackets. Policies should be guidelines and bare minimums, with innovation and improvement specifically allowed and encouraged.

Comments always welcome - I'd be interested to know your thoughts. :-)