The article below began life as three separate articles, partially in the form of a diary. The “Levels” it refers to have long since been consigned to the dustbin of history, and the agencies mentioned joined the choir celestial a long time ago. But still, I think the thought processes involved are still apposite, even if the contexts have changed.

For example, the ability of AI to “write” perfectly acceptable code means that some of the old assignments set for students simply won’t do any more. The issue of whether the means of assessment measures what it purports to measure is still very relevant — perhaps even more so. After all, there’s a danger that instead of testing a student’s ability to write a computer program, or to see what’s wrong with one that has been presented to them, you migth be assessing how good their AI prompt was.

Trying Times Part 1

From June 2008

I've been involved in two very different forms of assessment this week, and only today I read about some new research going on. So over the next day or two I thought I would just talk about those things and, in particular:

·     The two key difficulties of assessment;

·     My difficulties with rubrics; and

·     My problem with some newly-published research.

The two key difficulties of assessment

I had an extremely enjoyable day today. You will quite possibly think I am a masochist, because my day consisted of being locked in a room with half a dozen other people, looking at items for an on-screen test for educational ICT.

This is the test whose development I was very closely involved in whilst working at the Qualifications and Curriculum Authority (QCA) a few years ago. The idea of the test is that it tells you, or gives a very strong indication of, the level at which the student is working, or has attained at that point in time, in information and communications technology (ICT).

This test was different from all its predecessors, not because it was taken at a computer screen -- many tests do that -- but because it attempted to assess the student's ICT level by their performance in problem-solving tasks. So, think about that for a moment: we are not talking about a multiple choice test in which, if it has been written properly, there is only one correct answer per question. This is a situation in which there is not necessarily any correct answer, but one or more correct ways of approaching the problem.

For a test like this to work, there has to be a set of rules underpinning its judgements, and that's what I came up with: the rules base (see below). This is, in effect, a vast grid of possibilities, but one which at its heart is very simple Boolean logic. Basically, the test comprises a set of rules which, if written in ordinary English, would read something like this:

“If the student does A and then does B, but does not do C”, she is probably at a Level 5 in this area of the curriculum. But let’s test this assumption by seeing what she does on this next task. If she does X...”

Originally, the test was intended to be a summative test, ie one that measured the student's grade at the end of what is known in England and Wales as "Key Stage 3", ie age 16. It was, therefore, to be a high-stakes assessment. And it was, perhaps inevitably, removed from the "mandatory" column to the "optional" column of the government's plans, to become, instead, a tool for formative assessment.

I think that was the right decision, because the test is so good at testing understanding in real-world problem simulations, and one that can therefore form a great basis for class discussion.

Over the past few years I have been involved in the discussions of the test items. What happens is that people write the test items, they are then translated into interactive questions, and we come along and pull them apart for hours on end.

The "we" are people from the key organisations involved, ie the QCA, the National Assessment Agency (NAA), and the National Strategies, and me. To put that another way, the participants in these discussions have been the organisation that has written the ICT curriculum for England and Wales (QCA), the organisation whose remit it is, amongst other things, to oversee the assessment of the National Curriculum (NAA), and the organisation which has taken the ICT Programme of Study and made it more concrete, in the form of a Framework and lesson plans and other resources.

The two key difficulties of assessment that become obvious in this sort of activity are:

1.    Does this test item measure what it purports to measure? In other words, is it valid?

2.   If so, does it succeed?

Put like that, it may sound like those are more or less the same question, so let me give an example to make my meaning clear:

The first question is about looking at what the test item requires, and seeing if that needs knowledge or understanding in a particular area in order to answer it correctly. Today, for example, we were looking at sequencing (programming) questions. If they had included examples of things like boiling water for a cup of tea, we would have said that you don't need knowledge of ICT to answer such a question. All you need is knowledge of the world and maybe some common sense.

Where a question did pass the validity test, we then had to say,

“OK, you would have to know about ICT to be able to answer this question, but are the options provided correct, does the flowchart sequence shown actually work, do you need to be highly literate in order to understand the question, would you be disadvantaged if you were a particular gender or of a particular ethnic group?”

Clearly, what is enjoyable in such discussions is that you grow in your own understanding of the subject. Also, it is very interesting to hear other people's views on particular issues. It's the discussion, the interaction, that moves things on. And sometimes there are moments of humour, like today when I realised, halfway through an impassioned  plea against a particular question, that I had successfully demolished my own argument!

One of the things I have noticed that distinguishes good ICT/Computing departments in schools from the others is that time is made available for discussion about assessment. Teachers need to have a clear and shared understanding about what, say, a Level 5 student is capable of, and they need to be able to assess their students' performance with a reasonable degree of accuracy.

They also need to be able to articulate to their students what these levels mean, and what a student must do to progress from one level to the next. They must also go beyond even this, and help students become, to use Maslow's term, self-actualised in the realm of assessment. In other words, they must be able to evaluate, accurately, their own performance.

Unfortunately, time to do all this is the one thing that teachers no longer seem to have.

Sadly, today was the last of these periodic meetings, because we have now gone through all the main areas of the ICT curriculum. In an ideal world, everyone involved would now start back at the beginning, in order to ensure a constant supply of fresh material, but, for the time being at least, that is not to be.

In my next article on this subject I will describe a rather different form of assessment I have been involved in this week, that of using a rubric as a "meta-judge" for the Horizon 2008 project.

Trying times part 2: aye, there’s the rubric

From 2008

I was recently approached by Julie Lindsay and Vicki Davis with an invitation to be a "meta-judge" on the Horizon 2008 Project. It was a great honour to be asked, and I hope my judgements are received in a positive way.

But, as usually the case with this sort of thing, it did raise doubts in my mind about the value of rubrics for this type of activity. They are useful, but they are also limited, and not nearly as objective as one might think.

In a nutshell, the project involved students from several countries collaborating with each other to do research into how modern technology is affecting various aspects of modern life (government, education, health and others). The end product, besides the wiki itself, was a video submitted by each student. These have been judged by a number of educationalists, who decided on the winner in each of the 13 categories. My role as "meta-judge" was to decide which of these 13 finalists was the ultimate winner.

I have to say that this was not an easy task despite having the rubric to guide me. It isn't easy on a human level, if I can put it that way. The trouble with identifying one winner is that by doing so you automatically identify 12 "losers"! I would hope that those 12 don't see it that way. The quality of all the videos was extremely high, and there are even one or two that didn't come out on top that I will have no hesitation in using in my own work (with full credit and citation given, of course). To end up as one of just 13 finalists is good going, and all of the students should feel proud of themselves.

Indeed, even those students not in the final line-up did a fantastic job. If you’d looked at the wiki you would have discovered a cornucopia of ideas and resources, almost all of which were put together by the students.

The rubric I used is called Rubric 1, Multimedia Artifact. As rubrics go, it isn't bad at all. It's shorter than many, which is good, because the longer, ie more detailed, they are, the more easy they are to apply, but the less meaningful they become. The reason is that once you start breaking things down into their component parts, you end up with a tick list of competencies which, taken together, may not mean very much at all. That is because the whole is nearly always greater than the sum of its parts, so even if someone has all of the individual skills required or, as in this case, has carried out all of the tasks required, the end result may still not be very good. So you end up having to use your own judgement about how to grade something, which is exactly what a rubric is meant to avoid in the first place. Let me give you a concrete example.

One of the sentences in the rubric reads:

“Content is constructed from a superficial synthesis of information on the wiki.”

That seems straightforward enough, until you come across a case where the content on the wiki page is itself superficial -- in which case the right thing for the student to have done would have been to ignore the wiki page all together and put in some fresh insights. But if they had done that, they wouldn't get credit for using the information on the wiki page. In other words, it's a no-win situation which actually penalises the student who exercises her own judgement.

I think the main problems with rubrics in general can be summarised as follows:

1. Do the individual criteria reflect what it is we are trying to measure? This is the problem of validity which I discussed in my first post in this mini-series (see above).

2. Are the criteria "locked down" sufficiently to ensure that the rubric yields consistent results between different students and between different assessors (judges)? This is known as the problem of reliability.

3. Are the criteria too "locked down", which could lead to an incorrect overall assessment being made (the validity problem) or assessors introducing their own interpretations to aid the process of coming to a "correct" conclusion (the reliability problem)?

4. Does the rubric emphasise process at the expense of product? It is often said that in educational ICT, it's the process that's important. Well actually, that is not entirely true, and we do young people a grave disservice if we fail to tell them so. If you don't agree with me, that's fine, but I invite you to consider two scenarios, and reflect which one is the most likely to happen in real life:

Imagine: Your Headteacher or Principal asks you to write a report on whether there is a gender bias in the examination results for your subject, in time for a review meeting next Wednesday. You can't find the information you need, so you write a report on the benefits of blogging instead. You desktop publish it so it looks great, and even burn it onto a CD for good measure. To add the icing on the cake, you even make a 5 minute video introducing the topic in order to get the meeting off to a flying start.

Scenario 1:

The boss says:

“Wow, that is fantastic. It’s not what I asked for at all, but let’s face it, it’s the process that’s important. Let me raise your salary.”

Scenario 2:

The boss says:

“What is this? I asked you to produce a report on gender issues. If you can’t follow a simple instruction like that, do you really think you’re cut out for this job?”

OK, I know that both responses are slightly far-fetched, but hopefully I've made my point.

Which also leads me on to another thing. I think some of my judgements may have come across as a bit uncompromising. But I really do not see the point of saying something like "Great video", or even "Poor video", without adding enough information for the student to get a good idea of why it was good or poor, and how to improve their work and take it to the next level in the rubric.

Getting back to the issue of interpretation, I am afraid that, in the interests of better accuracy and of giving the students useful feedback, I introduced some of my own criteria. Well, I was the sole meta-judge, a title so grand that I felt it gave me carte blanche to interpret the rubric as I saw fit. Lord Acton was right: absolute power really does corrupt absolutely .

The extra criteria I applied were as follows:

1. Did the medium reflect the message?

To explain what I mean by this, let me give you an example of where it didn't. In one of the videos, the viewer was shown some text which said that businesses can now make predictions. This was then followed by a photograph of chips used in casinos. So, unless the video was intended to convey the idea that predictions can now be made which are subject to pure chance, which I somehow doubt, that was a completely inappropriate message.

2. Could I learn what I needed to know about the topic without having to read the wiki? If not, then I would be at a loss to explain the point of having the video, unless question #3 applied. This includes the question: is the information given actually meaningful? Look at that point about businesses can now make predictions. Businesses have always made predictions, so that statement tells me nothing. What I want to know is, how does communications technology aid forecasting, and does it make the process more accurate?

3. Did the video inspire me to want to find out more, or to do something, even though there wasn't much substance to it? If so, and if that was at least partly the aim, maybe that would be perfectly OK. I'd take some convincing though.

4. Did the video only synthesise the information on the wiki, or did it do more? The word "synthesise" implies adding value in some way: it's more than merely "summarise". But if if the information was of a poor quality, did the student deal with the matter effectively or merely accept the situation?

5. In every case I watched the video first, and then read the wiki, because I wanted to come to it with as few preconceived ideas as possible, to see if the video was able to stand on its own. I then read the wiki and then re-watched the video (sometimes more than once), looking for specific things.

If you have any views on using rubrics, I'd love to hear them -- especially if you completely disagree with anything I've said in this post!

The title of this piece is a horrible play on words, although I have to say I'm quite proud of it! It is, of course, taken from Hamlet's "To be or not to be" soliloquy, in which he says, "Ay, there's the rub".

 Some notes on the Rules Base

This was worked out on paper first, and then created in a spreadsheet. The spreadsheet, based on some amazing formulae and quite a few macros, was interactive. Bear in mind that this was constructed back in the days when we talked about “Levels”. Here’s the front page:

As you can see, this was intended to be use-friendly, by which I mean that the user didn’t have to know anyhting about Excel in order to make use of it.

The coloured rectangles were buttons. Clicking on one took you to the relevant section of the spreadsheet.

The skills & techniques calibration tool attempted to say which skills could indicate a particular Level in different applications. For example, configuring AutoCorrect in Word is an example of efficiency; using an absolute cell reference in Excel, or named ranges, are also examples of efficiency. Therefore there is a sense in which AutoCorrect and named cell ranges are the same, even though they are very different.

Please note what this was not:

• It was not promoting the teaching of Microsoft Office, which was one of the complaints people started to make about the ICT Programme of Study. It was simply acknowledging that some ways of using the Office applications were more efficient than others.

• It was not suggesting that using an absolute cell reference in a spreadsheet meant you were on Level 7, the “key characteristic” of which was efficiency. It was simply suggesting that if a student had a well-designed spreadseet that addressed a particular problem, and they had used devices like absolute cell references, then perhaps they had the knowledge, skills and understanding one would associate with Level 7.

Actually, the Rules Base was never intended to be an oracle that would tell you definitively what Level a student was on. It was, rather, intended to be used as a stimulus to discussion. Further screenshots are shown below.

I always think it’s important to think about an issue before reading a book or an article about it. That way you can be a more engaged reader, more critical (in the intellectual sense), more active, rather than as a blank canvas on which to soak up someone else’s ideas.

I have somewhat dichotomous views of this question of whether leaders make a difference, or much of a difference. I think my views can be classified as macro and micro. Let me explain.

I have always been somewhat dubious of the so-called “great man” theory of history. For instance — and this is an example given in this book — if Hitler had been bumped off earlier in his life, would the Holocaust and a world war have been averted? It’s tempting to say, “Of course”, but as I pointed out in my review of Weimar, Germany was already a hot bed of virulent antisemitism and other far right views long before Hitler was even born. Hitler, in my view, was in the right place at the right time — and when I say “right”, I mean by his standards not mine.

That’s what I call a macro perspective, but turning to the micro one, leadership at a school level or even a departmental level can make an enormous amount of difference. For example, the technical suppoirt set-up was second to none in one school I worked with — until a new deputy head was appointed. He preferred his own ideas to the ones that actually worked, and the whole thing went down the pan in virtually no time at all.

Profits… looks at leadership in several different fields, including politics, sports, and religion, and I am very much looking forward to reading it. It will be published in September by Allen Lane, who very kindly sent me the ARC, followed by the published book itself.

An article in New scientist caught my eye recently. Entitled “The sneaky trick for solving problems without answering them”, Jacob Aron describes a mathematical approach to solving a problem without actually doing anything about it. All you have to do is see that it can be solved.

I have to say, I could have done with knowing about this when I was at school, where we had to answer questions like, Sam is five years older than Freda who is twice as old as Jan. How old is John? Or something like that. Anyway, the not-actually-doing-anything approach is known as “non-constructive proof”, and is best exemplified by a joke.

A person walks into a room and sees that it is on fire. She looks around and spots a fire extinguisher in the corner. Realising that the fire is able to be tackled, she goes out of the room and carries on with her day.

I have to say, it reminds me of the old joke about economists and their unrealistic assumptions. An economist, a physicist and a chemist are stranded on a desert island. They have plenty of tinned food but no tin opener. The physicist cuts down a few branches of a palm tree and constructs a makeshift catapult, which he uses to hurl the tins against some rocks. The ruse fails. Then the chemist tries to dissolve the metal by soaking the tins in salt water taken from the sea. To no avail. All this time, the economist has been sitting there with a smug look on his face. “You’re looking quite pleased with yourself” says the physicist. “what’s your solution then?”

“Let’s assume we have a tin opener” he replies.

I realised that I’d been using this approach for years. I’ve always loved working out how I could solve a particular problem using either a ready-made program, like MS Access, say, computer programing, or an amalgamation of the two, such as using VBA in a Word document. Once I’ve worked out how it could be done, I haven’t seen the need — or had the desire — to actually do it. This non-constructive business sounds like the same approach to me.

“Hard hats not just graduation caps”

Britain’s new Prime Minister, Andy Burnham, has announced that schools are to offer technical education from the age of 14. One of the problems of being as old as I am is that I remember the last time this sort of thing was done. And the time before that. And….

Don’t get me wrong, I’m all in favour of it. Not everyone is academically inclined, and neither is everyone technically inclined. I was marvelling today at a CD cabinet (see photo) that my father-in-law made. It’s perfect, and it was made without any written specifications.

Compare that with me. At the age of 15 I was banned from the woodwork room in school by the head of department, Mr Cleaver (how’s THAT for an example of nominative determinism?). Why? Because while one of the achievements of another (admittedly older) lad had been to make a guitar, my crowning achievement was to cut a piece of wood, shape into a wedge, and glue it to a block of wood on which I had drawn two eyes and a month.

No, my frustration is that we seem to go on and on reinventing the same wheel, abandoning it, and then inventing it again, ad nauseam. During my teaching career I’ve seen these initiatives come and go several times. This version sounds very like the Diploma, a mixture of theory and practical study that involved visits to organisations and working in or with local businesses. That was a great scheme. Of course, there are a few minor details to be sorted out, such as who is going to teach it, and when? Will music and languages go by the board, for example? Will the school day be doubled in length? Anyway, I shall reserve my cynicism for now, and wish this new scheme good luck.

This article about technical education first appeared in my free newsletter, Digital Education. Also in that edition: news about an AI petition, and Andy Burnham’s AI and technology-related reorganisations.

When I taught Economics I was always on the look-out for games and simulations that would help me get across certain difficult economic concepts. The ones I employed were Stockpiler and Running The Economy, which were, of course, intended to be used for such a purpose. But I also used Sim City and even Monopoly.

This book looks at the maths concepts — and, to some extent, the physics concepts — hidden in popular video games. It’s a great idea, and while I can’t say I understand all the mathematical explanations, I am sure a qualified maths teacher would have no such difficulty.

At the time of writing the book is nine years old, so perhaps the games used as examples have moved on or been discontinued, but it is the principle that counts.

In 2012 I wrote an article called The Stimulating Classroom, updated in 2016. Much of it still relevant, so I've republished it here, with a few minor amendments and an updated link.

It seems paradoxical, but the most boring classrooms tend to be the ones that are full of technology – and little else. The worst ones I’ve been into are those in which 30 or more computers are crammed into rows, allowing no room for note-taking, let alone collaboration -- despite plenty of evidence indicating that kids learn better by working with others, at least some of the time anyway.

But even the ones with wall-to-wall interactive display screens, visualisers, graphic tablets etc etc are often, to be frank, Tedium City. How come?

I think the reason, in a nutshell, is that in the sorts of classrooms I’ve just described, the emphasis is on technology rather than learning. What a shame. What a missed opportunity. Technology can enable great and transformative things to happen in the classroom and beyond, but I think it's easy to lose sight of that if we're not careful.

Primary (elementary) classrooms tend to have this more right, or right more often, than secondary (high) schools in my experience. Secondary schools are serious, with a timetable to stick to come what may, whereas primary schools have still not entirely lost that ethos of exploration they are so good at, despite numerous “initiatives” over the years to ensure that anything that moves, or even anything that doesn’t move, is measured. The best schools still have that despite the constant pressure by Governments to test the kids and micromanage the teachers.

So what sort of things should a classroom have? I’m basing this list partly on my own experience of having taught in secondary schools over many years, and my visits to what I considered to be the schools where the education technology classrooms were the most vibrant.

Different areas

Different sorts of activity require different kinds of spaces. Within the classroom there should be an area where students can use the school’s computers, and places they can sit and plug their own device in if they need to. There should be places where students can collaborate, using pencil and paper if necessary, at tables. There should also be, if space allows, some comfy chairs too. If the school wireless network, layout and rules allow, the classroom can be extended beyond its walls by allowing students to work outside.

In my experience, even if the room you have is incredibly small, you can still arrange furniture and equipment in such a way that you can accommodate different sorts of spaces. And if you can’t then I would suggest, at the risk of being designated a pariah, that you might consider getting rid of some of the equipment. Nobody needs one-to-one computing all the time, and for learning purposes it’s not usually the most effective strategy anyway unless you build in opportunities for collaborating with others.

Books

Not e-books, but books. Not because I’m a Luddite (I’m not), but for four reasons:

First, the very practical one that if you have a few manuals and other books around, it is easy to pick them up and look things up while your partner is working at the screen.

Second, the nature of the books don’t have to be restricted to manuals. There are science fiction novels and short stories related to technology, books about the development of cyber warfare, books about the rise of Google and the fall of Boo. All sorts of technology-related books to stimulate thinking and broaden the mind. I dare you to start a class library.

Third, there’s a hidden message that old technology is still important and useful. Ebooks are great, no doubt about it, but there’s no reason to ditch paper ones, especially as you can’t display ebooks on a bookshelf as far as I know.

Fourth, I think it designates the classroom as a place of learning, not a hub of technology. I think that’s important.

Magazines and other periodicals

I always had newspapers, magazines and even comics in my classroom. I used to buy a lot of computer mags, and I’d bring them in when I was finished with them. If you want a student to write a software review, show them a variety of types of review in different magazines; they’ll soon get the idea. Even the Sunday newspaper supplements sometimes have interesting articles, such as on cyberbullying.

And the comics? Well, comics like 2000 AD or the Marvel comics, though somewhat outlandish in many respects, have interesting ideas and a great appearance from a design point of view. One of the requirements Computing curriculum worthy of the name should be the ability to present information in a variety of formats according to its purpose and intended audience. Comics or, if you prefer the grown-up term, graphic novels, can be pretty effective. There are plenty of comic-creation resources available: see the article referenced in A cornucopia of comic-making applications (updated).

Tablets on demand

I like the idea of students and teachers being able to use technology where and when they need it. So that means there should be good enough connectivity to make that feasible, and that banks of laptops or tablets are available quickly and easily. Having to book the laptop trolley a day in advance doesn't cut it.

A stand-alone computer

I always liked to have a computer that wasn’t connected to the rest of the network. Yes, I know it’s a bit retro, but it meant that students could check out DVDs without having to go through a whole rigmarole to gain access to the DVD drive. It also meant that they could use programs that were not available on the network, or in the cloud but not available from the network. This sounds all very old-fashioned, but if you have some useful DVDs or CDs from days of old, why abandon them just because technology has moved on?

I also used a stand-alone computer to play music while we worked – Vivaldi, which tended to keep the kids both calm and productive at the same time!

These are just a few ideas, and I don’t even think I’ve scratched the surface. In my opinion, the classroom should be a powerhouse of learning and exploration. If that’s the starting point, then in my experience learning will follow much more than if the starting point is technology.

Permalink