WebinarOctober 14, 2026

A layered approach to academic integrity: moving past the proctoring debate in Europe

The conversation around proctoring, digital learning, and AI in Europe has split into two camps: one calling for a return to the traditional classroom, the other pushing an all-digital future. Both miss the point. Robert Burns of Cirrus Assessment and Proctorio's Justin Depuydt make the case for a third path, a layered approach where institutions choose the right model for each circumstance instead of committing to one extreme.

TranscriptDescriptionResources

Daley Craft (0:17)

Good morning, everybody. Thank you for joining. We'll get started in just a minute to give everyone time to get into the meeting. Justin and Robbie, whenever you're ready, you can take it away.

Justin Depuydt (2:57)

Hey everybody, thanks for coming, and welcome. This is one of our Proctorio webinar series, and this one fits into the thought leadership category. My name is Justin Depuydt, and I'm Head of Product Marketing at Proctorio. I typically do two kinds of webinars. One is about our products, especially the ones that are coming out soon, and that isn't this session. This one is less about proctoring specifically and more about the bigger picture: the challenges we all face together in the industry, and how we can solve them. We really believe proctoring is one part of the solution, but there's a bigger picture out there, so the guests we bring into this series are people we'd love for you all to learn from, and maybe even follow.

Our guest today is Robbie Burns. We met on a panel some time ago, and since then I've followed his work and his posts, and found he had a lot of really valuable things to say, and a lot of alignment with us. By way of background, Robbie is currently Head of Business Consulting at Cirrus Assessment, and he's been working in and around assessment delivery for nearly 30 years, which is amazing for someone who's only 35. That's a joke. He's overseen large-scale assessment programs and, really relevant to today's conversation, he's supported a number of organizations through the transition to digital delivery, as well as advising for many years on assessment quality, integrity, and operational resilience.

You can tell we've got people from all over the world here. I'm calling in from the state of Arizona, Robbie is calling in from Scotland. The subject here is layers, and we've already made the Shrek jokes, so if onions come up, apologies in advance. I'm a dad, so I make bad jokes.

Here's roughly the agenda. We're going to talk about the pendulum and which way it's swinging, specifically around whether to go in person or fully remote. Maybe some of you have had the same experiences Robbie and I have had, seeing it go back and forth. Full disclosure, both of us are working on the more remote side right now, so we'll make the case against our own models to be fair, and then we'll make the case against going fully in person. A lot of this isn't unique to us, we hear it from lots of people. Then we'll talk about the middle approach Robbie and I have been discussing, Robbie will share a framework he's developed, and then we'll have time for Q&A.

So here's the pendulum. Robbie, do you want to start and talk about where you see the pendulum now, and maybe some of the experiences you've seen with the shift?

Robert Burns (6:34)

Thanks, Justin. At the moment, what I'm seeing and hearing when I speak to people are two extremes, both of which are probably getting louder as the weeks go by. One is very much the keep proctoring, keep digital camp. The other is pushing back towards in-person exam halls. For me, what we need to do is move beyond that.

Maybe it's just my age, but the older I get, the more I seem to think that in every industry, to use your pendulum analogy, fashions change. We often see things that were out of fashion, considered old and outdated a few years ago, get brought back, usually dressed in strategic camouflage gear and given a trending new name. So what we're seeing in the sector isn't unique. What concerns me a little is that we're not trying to market an exercise regime or a leisure range. What we do has a really serious impact, for both good and bad, on people's lives, on their careers, on their futures. In my opinion that means we need to make every decision only once we've given it serious consideration, detailed planning, and robust testing. We shouldn't be jumping on the back of whatever's fashionable at a point in time, or following what we see someone else doing.

In this particular debate, I'd be the first to admit that there are genuine benefits and genuine risks on both sides, which is why I don't think we should be trying to find a compromise between remote and in-person assessment. I don't think one is right and one is wrong. We've seen the pendulum swing in both directions over time and, for another dad joke, like Austin Powers I bet it's going to keep swinging well into the future. But for me, the whole binary nature of the debate risks taking us in the wrong direction.

Remote versus in person is a delivery debate. Where the assessment takes place, how it's invigilated or proctored, isn't the same as what it measures. Neither automatically establishes whether an assessment has validity or integrity. So I don't think either model is inherently right or wrong. What's important is that we choose the method that best suits the purpose.

Justin Depuydt (10:00)

I've been encouraged by the way you sometimes talk about this, the idea that we don't have to accept the false binary. I wrote that down at one point. I've been in the business much less time than you, but I've already seen it. Five years ago it was "in person is done, the future is only remote," and this past year I've seen more of the opposite. But that's a false binary we don't actually have to accept. Is that fair?

Robert Burns (10:35)

Yeah, I think it's false. And just now, not only in this argument but perhaps with social media and the world we live in, things tend to be much more polarized than they have to be. In this case, I don't think it's a binary choice. What worries me is that when you look at the debate, or listen to it, or watch what others are doing, we may end up being sucked into one side or the other. And I don't think we have to be.

Justin Depuydt (11:16)

You said we were going to be honest about the pros and cons on both sides, so let's jump into that. Do you want to start? I have some ideas, but I'd love to hear your thoughts on what the detractors, or the problems people are running into, are with the fully remote model.

Robert Burns (11:38)

In my opinion, remote proctoring addresses some very real challenges that we face, but if we're honest, it introduces some others. That said, I think that's true of every assessment delivery model. Where we can sometimes go wrong is assuming that the strengths or weaknesses of a delivery method tell us everything we have to know about the assessment itself.

On the proctoring side, over the years I've dealt with candidates who have concerns about the software being too intrusive, or monitoring their living spaces. I've dealt with organizations that have concerns about data and regulatory risk, perhaps bias in the algorithms, false positives, the technical side of it. I've spoken to candidates who have argued passionately that the very presence of a camera in the room puts extra psychological pressure on them, that being recorded increases their stress levels and makes them perform less well in the exam environment. That said, I've spoken to just as many candidates who say it absolutely works for them, and who hate the thought of going back to an exam hall.

So to be completely honest, there are definitely challenges that come along with remote proctoring. I think there's a strong argument that not every type of assessment is suited to remote proctoring, but equally there are some assessment types that are perfectly suited to it.

Justin Depuydt (13:44)

I'll share some of the ideas I put down, and you can tell me if you think they're accurate. That last point you made is a really important one, that there are some assessments that aren't suited to it. I was at a conference where the whole point of someone's talk was that they weren't doing remote anymore, and all the challenges they'd had, but her assessment was one that was suited to in person. They didn't have some of the populations that others do.

Here's what I've seen and heard. When you introduce more devices, because part of remote proctoring is often the bring your own device problem, which I'd say is frequently the source of a lot of the other troubles. If you're allowing people to use their own devices, that adds variables, and that adds the necessity of more security solutions and more technology, and then they may have technical issues that can be challenging. I'm in this industry, I see the data, and statistically the numbers are really low, but as an administrator you may not experience that. Every one person who has a really hard experience with the technology can feel like a mountain. And then you can get more of a combative attitude. It's been 10 years since I did in-person testing, so it's been a while, but I remember people being combative there too, so maybe that last one isn't fair. Any thoughts on my list?

Robert Burns (15:33)

There's definitely a concern and a challenge around the digital equality of candidates. You spoke about the tech side of it and I think that's true. I could have a really top of the range, brand new device, and the person sitting closest to me in the exam is on a 15-year-old laptop that isn't really suitable.

I also think the equality part extends beyond the device argument, because as individuals we differ in how comfortable we are with technology. As everyone can see, I'm no longer a spring chicken, and I assumed wrongly, when we started moving towards a much more digital world, that the generation of students coming through now would all be very technically literate, would all be really comfortable finding their way around technology, and that all of this would be second nature to them. I was really surprised that that isn't the case. I don't know if it's because everything is an app now and it's all done for them, so even something as simple as clearing their cache isn't easy for a lot of people.

Although that's a challenge, it's also a challenge that's quite easy to deal with, because you just have to educate, train, and support candidates through that process before they get into the exam environment. So, sorry, I'm rambling, as you know I do. I think there are challenges, but I also think that as assessment organizations we have a greater responsibility to deal with those challenges in advance, and to support students before they hit the point where it's nine o'clock on a Monday morning and they're about to sit the exam, because by that point they're stressed, they're worried, they're panicking, and everything feels like much more of a drama than it actually is. So, clearly challenges, but challenges I think are surmountable if we plan properly.

Justin Depuydt (18:17)

Let's look at the other side. Part of the reason this is relevant right now is that, transparently, it would be easy for me to say "here are all the problems with in person." But it's important to talk about because a lot of people are talking about it as a panacea. You bring up a problem from a remote exam and people say, just go back in person, just go back in person. Anyone who's been in the industry can say, well, wait a minute, did we resolve all of the problems we always had with in person? For most of them the answer is no. So we don't have a panacea either way. That's my setup, so I'll let you go ahead on where you see some of the challenges there, and then I'll add mine.

Robert Burns (19:10)

There's a lot of truth in what you said there. I spoke to a really large university about three weeks ago, and they're quite seriously considering moving back towards an exam hall environment. When I was talking to them, two or three of the people at the table were perhaps as old as me, or close to it, and they were looking back very fondly through rose-colored glasses at how great life was when everything was done in an exam hall. Whether they were forgetting, or whether they were deliberately avoiding, all of the issues that come along with an in-person assessment, whether that's accessibility or travel or cost or capacity, just trying to find a venue that will let you plug in 300 untested laptops is a challenge before you even begin to factor in all the human and operational challenges that come with a physical exam.

The other thing I don't think they were taking into account is that the world today isn't quite the same as the one we look back on when we delivered in-person exams. For example, I've spoken to another couple of organizations recently, and to your combative point, they're really struggling to get invigilators to participate in exams. The reason they're struggling is that individuals have either experienced or witnessed threats of violence, and in a couple of cases actual physical abuse, from candidates who weren't happy that their ID was being challenged, or weren't happy that they weren't being given ten minutes extra time for a perceived delay in the exam. These are things I certainly wouldn't recall being issues 10 or 15 years ago. So we do have a tendency to look back at things with rose-colored glasses.

Even the whole control of the location piece I would challenge. We seem to forget that it's really difficult to control that kind of exam environment, and I don't just mean the noise of 100 people writing, or 100 people typing, or someone sneezing all the way through the exam. Everybody watching this will have experience of roadworks outside the exam venue, or cherry pickers outside the windows. I had an experience several years ago running an assessment in Scotland where a religious band came parading past the exam hall, and they literally stopped about 150 yards away to play their music and shout their chants and sing their songs. I vividly remember standing in front of three pretty scary march organizers, pleading with them to take pity on the 100 students we had down the road trying to sit their exam. I wasn't very successful, if I remember correctly.

The point I'm trying to make is that an examination hall is a controlled environment, but it's not a perfectly controlled environment. I think we sometimes forget those issues, the logistics, the missing scripts, the couriers that don't turn up when they're supposed to. I'm not sure we remember just how painful they were, or how many grey hairs they gave us over time.

Justin Depuydt (23:42)

You mentioned this thing about different delivery models being appropriate for different exams. The one I attended and referenced was a higher-level exam that only established professionals would take, and it wasn't particularly high stakes.

There are a couple of things there. One of the problems that existed before, and continues to exist as things change, is the technology factor even in an exam center. I don't know how many people saw that there was an exam center where they had put servers underneath the floor, connected into all of the computers. And then somebody at the conference I was at asked the question: five years ago we were worried about professional contract cheaters, and to your point, if we can't find enough invigilators, we lower the standard of who we're willing to hire, and we end up with invigilators, and this has happened, who are actually on the take with contract cheating companies. Now your entire test center is exposed. So those challenges still exist.

The other one, which you sort of indicated here, is the accessibility concern. The person I went and listened to had the luxury of being able to say, well, if somebody isn't able to get to me, that's their problem. But to your point that things have changed, I don't think most people can still make that case. Because we have the ability, that carries with it an obligation, and on the part of a lot of universities, the desire to reach people who in the past, for whatever reason, whether it was mobility or cost or lifestyle, a working parent or something, couldn't be reached. If you go back fully in person, you're kind of saying "too bad" to all of those people.

Robert Burns (25:56)

I think that's true, and it's probably also important that we're in a place today, more than at any point I can remember in the past, where candidates aren't afraid to say what they think. They're not afraid to comment if they think things are going well or not so well. We have to recognize that the way our candidates engage with us, the way they interact, and what makes them feel confident, is important. We have to accept that the examination environment can influence how they feel and can influence their performance. It's a massive part of what we're asking these candidates to do.

If that's the case, then I think we really need to be asking those questions about which pressures are relevant and which pressures are appropriate, in order to give us confidence in what we're trying to assess. And it's back to that: it's not a binary decision. I don't think it's a black or white, proctoring or exam hall decision.

Justin Depuydt (27:18)

So for the next section I'm going to refer back to a comment you made earlier. You said the strength or weakness of the delivery method isn't necessarily the cause of all of the problems. I think that's going to lead us into some of the solutions you have, so I'll try to go quickly here. You and I have talked about these two examples, and I'll share mine first.

These are examples of exams where a lot of the articles and a lot of the reading about them really pinned the fault on the delivery method, and in particular on AI. And I get it, it's clickbait, it's very sensational, so you saw a lot of headlines that basically said AI messed up this exam. Here's my example. I'm not going to name them, but this was a university entrance exam. It was remotely proctored, and they used two different services, neither of which was Proctorio. What was interesting for me is that at first, transparently, I was looking for what the proctoring services did wrong, because those are competitors of ours and it's good for us to know what mistakes to avoid. At least from the articles, I couldn't find anything that either of the proctoring services had done incorrectly.

This exam had over 100,000 concurrent examinees, so over 100,000 people taking the exam at the same time, and they ended up having to ask about half of them to retake because of integrity suspicions. I'd add a little color here: if you caught 50,000 people with suspicious activity, that suggests to me that your proctoring worked. I get that it's a number far higher than what you want, but you did actually catch the infractions.

The good news about this, in terms of getting to a solution, is that we want to see what the problem really is. When people look at it and say the problem is AI, or the problem is proctoring, or the problem is the delivery method, they try to solve for that, and that doesn't actually resolve most of the challenges. From my observation, and I wrote a blog about this if people are interested, they took a university entrance exam they'd been running for a long time, didn't redesign it in any way, just digitized it and put it out. That's a missed opportunity. I think I learned this partly from a webinar you did with a psychometrician, Robbie, where she was talking about how much you can do in the assessment design to mitigate a lot of the vulnerabilities of remote delivery. The only real change they made was to make it remote.

The other big thing I noticed is that they had a one-time, all-or-nothing exam. Especially if you're part of a university, there's a lot of internal knowledge about what causes people to choose to violate integrity policies, and this is one of the biggest. If you give people a very high-stakes exam where everything is riding on that one thing, especially if your examinees are in potentially desperate circumstances, where it literally makes the difference between a quality life and a really difficult life, you should expect them to cheat. That's going to happen. The good news is you can mitigate it by changing something, so that everything doesn't ride on this one exam. That was my example. Do you want to talk about yours?

Robert Burns (31:30)

The example I was talking about was a European organization that delivered assessments across multiple different continents simultaneously. It's not a dissimilar scenario. They were really keen to make sure there was consistency across all the continents they were delivering to: same start time, same invigilation process, same technology. They realized further down the line that, as noble as the desire for complete consistency was, the infrastructure, the technology, and so many other factors varied so much from continent to continent that they couldn't deliver consistently by using the same mechanism across all those different places.

I learned a really similar lesson many, many years ago, when I still had no grey hairs and no wrinkles. I was told that one of the departments I managed, plus all the assessments run within it, was going to be moved from the UK offshore to another continent. So I travelled out there, really naive, really fresh-faced, thinking all I had to do was lift what I'd done in location A and drop it into location B, check that it worked the way it had at home, and that was it. I had absolutely no idea I was about to walk into an assessment environment unlike anything I'd ever seen before. Everything was completely different: the infrastructure, the expectations, the security requirements. At the same time I had a regulator who wasn't at all happy that all this activity was being moved offshore, and who took great pleasure in scrutinizing every single move we made as we were building the process out. What we had to do in the end was completely recreate the entire assessment process to meet those different circumstances.

That taught me a lesson years ago that I've never forgotten: consistency of standards doesn't necessarily mean identical delivery arrangements. If we look at the two examples we've touched on, I think we can see that the challenges those organizations faced are what happens when we assume that a single delivery model can work successfully everywhere, or that because it works successfully in one environment it's going to work equally well anywhere else. Whether it's a technical issue, connectivity, or an international delivery issue, we can't assume that because something works in place A it's going to work just as well in place B.

So if these grey hairs and the wrinkles on my face have taught me anything at all, it's that rather than starting with the assumption that we're going to deliver every assessment in exactly the same way, we need to ask ourselves what we actually need in order to be confident in what comes out of that environment.

Justin Depuydt (35:47)

My initial reaction to that, and this is something I've thought before and you've kind of talked me off the ledge about, is: that sounds great, but that sounds like a lot. You've indicated, I don't know if you'd say it's easy so much as that it's accessible. So I'm going to lead us into the framework you've proposed, and I'd be curious as you go through it to help us understand how this doesn't have to be a huge, monumental undertaking.

By the way, I haven't told you this, but I set these up to go one at a time on the framework, so just let me know when you're ready to go to the next one. I'd like you to talk about the framework, and how you think we can get to this middle way without it being a hugely expensive and time-consuming undertaking.

Robert Burns (36:38)

I've touched on this before, so I'll try not to repeat myself too much. For me, one of the problems we've got in the assessment sector is that we often start the process by choosing the delivery model or the technology first, and then we try to make everything else fit around that. What I'm suggesting in this framework is that we reverse that thinking, and start with what we're actually trying to establish about the candidate. We work out what evidence we need, and then we decide how best to protect that evidence.

To be fair, "framework" is a much fancier word than this deserves. It's five straightforward questions to help us make decisions in a more structured way. Nothing complicated, just five questions I think we should always be asking when we're designing or reviewing an assessment.

The first is: what is it we're trying to establish? Are we testing knowledge, practical ability, professional judgement, something else? And what are the consequences if we get that decision wrong?

The second is: what evidence would demonstrate that? What would we actually need to see a candidate do, if we're going to have genuine confidence that they possess the knowledge or the competence we want to evidence?

What follows from that is: what would undermine our confidence? Perhaps that we can't accurately establish who completed the work. To go back to the point Justin made earlier, perhaps the candidate received inappropriate assistance. Maybe the marking itself isn't as reliable as we'd like. We have to understand what within the process could undermine the confidence we've got in the outcomes.

Then we move on to: what combination of measures addresses those risks? This takes us right back to the beginning, to Justin's onion layers. This is where the layers come in. There's the assessment design, which we've touched on already, then identity, supervision, technology, marking, security, QA, and support. Every one of those layers has a different role to play, and we need to choose the combination that's appropriate for the assessment we're building, rather than assuming one size fits all.

And then finally we need to establish how resilient the arrangement is, which speaks to Justin's example a bit earlier. If we had to change the exam venue, or the platform we're delivering on, or the method of supervision, could we still make a credible decision about the candidate?

That should be it. The aim isn't to add more and more controls, it's to ensure that every measure we do have in place contributes something useful to the confidence we've got in the result. The approach also encourages us to identify where our arrangements depend on assumptions that might not hold true in every circumstance. Conscious of time, as an example, we might assume that because remote supervision has been effective for cohort A it's going to be equally effective for cohort B, or that because a platform has performed really well under normal conditions it's going to perform just as well when we put it under more pressure, or that because we've given candidates a copy of our rules and regulations we all understand those rules in exactly the same way.

There's nothing at all revolutionary in this framework, because quite truthfully I'm not that clever. It's just built on making good decisions more consistently. As you said when you introduced this, Justin, it's not about reinventing the assessment wheel. It's about stopping expecting one part of the assessment process to solve all of our problems, and about asking ourselves those questions any time we build or review an assessment, and building out from a different starting point. The technology shouldn't be the starting point.

Justin Depuydt (42:42)

What you just said is, for me, the biggest takeaway: not letting one part be the be-all and end-all that determines whether or not you have a successful assessment. I asked you to give us a really concrete example, so here's the page where we go through each of those five steps with one. Could you take us through that?

Robert Burns (43:14)

Absolutely. You'll see the text on the screen, so I'm not going to read through it, but let me try to paint the picture, as I have a tendency to do.

Let's imagine we're assessing a trainee accountant. The first thing we want to do is think about the decision. What we want to establish is whether they can make and justify a sound professional recommendation. So rather than immediately deciding that the way to do that is a closed-book written examination, we ask what evidence would actually demonstrate that ability most effectively. Perhaps that's a client case study with a recommendation and an explanation of the reasoning behind it. Whatever the appropriate vehicle is, we need to sit down and map out what gives us the most confidence that we can reach that place.

Then we consider the risks. Could someone else have completed the work for the candidate? Did the candidate receive inappropriate assistance, whether from AI or another person or Google or whatever? Could different markers interpret the response in different ways? We need to establish, as we think through all the different points in the process, where the risks could potentially lie.

Then we select the appropriate layers, so we identify what the appropriate checks are. Are our rules clear? Have we got a strong marking rubric? Is the moderation in place suitable? Is the security suitable? Perhaps in this particular scenario we might want to introduce a short follow-up discussion, or a viva, where the candidate can explain how they reached their conclusion. That's where we start to build up the various layers that give us confidence in the outcome without being overly burdensome on the candidates. We're not just throwing in layers of security or challenge for the sake of it. If we're going to add a layer, that layer has to contribute something.

And then finally we ask whether that model would still be credible if we had to change how or where it was delivered.

Perhaps it's worth emphasizing that we haven't started that process by deciding whether there should be an exam hall or a proctored exam. We started with the competence we wanted to assess, and then we worked towards an appropriate assessment model. The end result might well include proctoring, or an exam hall, or something completely different. But whatever that decision is, if we follow that framework, what we've given ourselves is the ability to explain why those measures are necessary and what they contribute to the process.

Because of time I'd just add one more element, which is the point you touched on at the beginning, Justin: at every stage, for all five steps, we should always be asking whether the approach we're taking is fair, accessible, and proportionate to the decision we're making.

Justin Depuydt (47:24)

As you've been talking I've been applying your framework, in my mind, to the example I gave earlier, the North American school. I haven't done this before, so tell me if this sounds right. It might be appropriate in some cases to have more than one assessment. We start with the decision: this was an entrance exam, so what are we trying to evaluate? Well, it's a few things. So maybe that leads me to say I can have my multiple choice remote exam, but maybe I should have some other things at different times that evaluate, in a different way, the different things I'm trying to assess. Is that a good way to apply it?

Robert Burns (48:05)

Absolutely. And I actually think that kind of model is becoming more and more relevant as AI gets better and as candidates are able to do things they couldn't do before. We touched on this in a conversation we had previously: a multiple choice assessment doesn't have to be recall and regurgitation. It doesn't have to be just explaining the knowledge. You can build multiple choice questions in a way that tests application and understanding, not just knowledge.

And then, to your point, that may not be enough. You might want to build on top of that with a viva, or a take-home exam, or a different kind of assessment, and then pull together two or three different strands that help you have real confidence in the candidate's ability.

One of my passions, hobby horses, rants, call it what you want, is that for me there's a real difference between assessing that someone can pass a test because they can remember something, and assessing that they're actually competent to do whatever it is the test is set up to do. That also builds into this weighting concept: if we really want evidence that the candidate can prove competence, it's about making sure we're using the right techniques. Increasingly that's more than simply one assessment. It ends up being a variety of different things pulled together over the length of the training.

Justin Depuydt (50:06)

Folks, we'll move on to Q&A in just a sec, but I want to say that the pendulum swing I'm actually excited about is the one where the challenges AI brings, and the need for remote delivery, force us into a place where we really need to start looking at whether this actually addresses the competency, or whatever it is we're trying to do.

I'm a former educator, and that was always an old thing, forever. For the longest time we've said, oh, did you memorize the things, and are you able to fill out the things you memorized? Great, then you get the credential. All that did was tell you that somebody has a good memory or good study habits, which might indicate other things. But I like this trend where I hear people saying, let's not think first about the delivery model, but about what we're actually trying to evaluate, and whether we're doing that in the right way. I think that's an exciting step forward.

Robert Burns (51:13)

I agree. When you talk to people in the sector there's an awful lot of doom, gloom, panic, and angst around AI and the risks it brings. I'm not trying to suggest it doesn't bring risks, but I think you're right, I think it's going to force us to do something that in my opinion we should have done 10, 15, 20 years ago. We're still assessing people the way we did decades ago, and people no longer work in a world where they have to remember things, where they have to recall. They've got the internet, AI, so many digital tools. We all work in a different way, and in my opinion assessment hasn't kept pace with that.

So I actually see the AI challenges as something that might finally force us as a sector to move away from what we've been doing for the past 50 or 60 years, and move into something more relevant to how people work today, something that can better prepare them for the workplace when they move into it. I don't think recall and regurgitation bears any relevance to anyone in the workplace anymore. To very few people.

Justin Depuydt (52:44)

This is so exciting. We only have a couple of minutes for Q&A. For all of our guests, you can find a lot of this material from Robbie on LinkedIn. He's also doing his own series where they talk about a lot of these things, and I recommend finding him there and joining those, it's really great material. If we can get everybody moving in this direction, we think there's actually a brighter future.

For Q&A, we've probably got time for two questions. Somebody asked: do you have any advice for convincing leadership to adopt a layered model? I presume the thought is that for people in a leadership position it might seem like too big a sandwich to bite into.

Robert Burns (54:05)

It's a really fair question, and I had this debate with a client about three or four weeks ago. It comes back to the piece you touched on before, Justin: it's possible to look at this and think it's a massive exercise and that you're reinventing the wheel, and we're really not doing that. Those five questions I spoke about, I'd suspect there's not a single person watching or listening to this who hasn't thought of them, who doesn't apply them at some stage to what we're doing. All I'm really trying to encourage people to do is to make those slightly less random, which is a poor word, but to make it much more of a structure. Whenever you're reviewing or creating an assessment, let's start with that framework.

It's not even about saying "I've got 25 exams and I'm going to sit and rebuild them all." Don't do that. Pick one assessment, one particular assessment you think you'd like to apply this model to, and work your way through it. I think you'd be really surprised at how little extra time it takes.

Now, I'm not going to lie: if your outcome is that you have to completely change the way you write the questions and build the assessments, that becomes a bigger piece of work. But even if that's where you end up, if you've got a bank of multiple choice questions at the moment where it is just recall and regurgitation, but the content is good and the material is sound, it's really not a massive leap to take that content and reframe the questions to the point where you're testing application or understanding or whatever it might be. And as you're going through question reviews anyway, you can build it into that process.

So for me, none of this is about reinventing the wheel. I've said that several times. It's just about having a different lens, applying the lens you probably always apply at a different stage, and doing it consistently. It doesn't cause a lot more work, it doesn't cause a lot more pain, and I think the benefits you gain on the other side, by having the resilience and not having the complaints and appeals and challenges, will more than make up for it.

Justin Depuydt (56:58)

I'm glad I chose that question, because I think that really is a good summary. We're almost out of time, so I'm going to leave the rest of the questions. If you still have questions you can put them in the Q&A, and if they're addressed to Robbie or myself we'll make sure we get an answer out to you. Robbie was kind enough that he and his team made a document showing the framework, and we'll send that out to all of you who were here as guests. Here's our contact information if you'd like to reach out to us.

With that, thanks to everybody for coming, and thanks so much, Robbie. This has been great, as usual, and I really appreciate your time.

Robert Burns (57:35)

An absolute pleasure. Thanks very much.