Tag: assessment

14 reasons why your multiple-choice quiz sucks

Unlike some of my peers, I’m not an opponent of the multiple-choice quiz. It’s a convenient and efficient means of assessing e-learners, if it’s done well.

The problem, unfortunately, is that it’s often done poorly. So poorly, in fact, it’s embarrassing.

Lego businessman, upset

At my workplace, I am regularly subjected to the multiple-choice quiz.

In fact, over the years in a range of jobs across several industries, not to mention my time in high school and university, I have been subjected to countless multiple-choice quizzes.

So I feel eminently qualified to tell you why yours sucks…

  1. The questions don’t represent the learning objectives, so why did I waste my time doing the course?
  2. There aren’t many questions, so I should be able to bluff my way through them.
  3. The number of attempts is unlimited, so I’ll just keep guessing until I pass.
  4. The pass mark is low, but my customers aren’t so forgiving.
  5. The questions and answers aren’t randomised at each attempt, so I’ll jot down my responses to prepare for next time: 1-A, 2-D, 3-B…
  6. I’ll also ask my mate what he got for Questions 3, 6 & 8.
  7. The questions follow the order of the topics in the course, but I’m unlikely to encounter them in that order on the job.
  8. You use “All of the above” only when it’s correct.
  9. The ridiculous answer is obviously incorrect.
  10. The longest answer is probably correct, otherwise you wouldn’t have bothered writing it.
  11. More than one of the answers is arguably correct, and I’m shocked you didn’t know that.
  12. Your use of the double negative can only mean one thing: you can’t write no good.
  13. You try to trick me rather than confirm my understanding of the central concepts, so remind me: what’s the purpose of your role in this organisation?
  14. Your questions test my recall of obscure facts rather than my behaviour in authentic situations, so this exercise has no bearing on my performance.

Clearly I’m being deliberately pointed, but in all honesty, have you ever been guilty of one of those clangers?

I know I have.

But that’s no reason to vilify the multiple-choice quiz. When combined with other forms of assessment, it’s a valuable piece of the pedagogical puzzle.

Let’s just design it better.

Chicken takes a multiple-choice quiz

Vive la evolution

Last week, Laura Layton-James stumbled upon my post Online courses must die! and she left a wonderfully detailed comment.

I was so enamoured with what she wrote that I feel compelled to repeat it here…

An excellent post Ryan. When running courses on how to create engaging eLearning (concentrating on the self-study module) we concentrate on designing, as Cathy Moore puts it, experiences. To me it’s a total waste of time reposting the information that already exists somewhere on the intranet in a pdf. As you say, a waste of resources when L&D’s expertise as learning consultants (which is what we are) can be put to better use creating those skills based activities that will test application rather than regurgitate facts and figures.

Unfortunately, the blame (if we have to lay blame) lies with the increasing need to tick boxes. It seems that if we just point people in the right direction for the information we can’t be sure they’ve read it. So we think the solution is to type it all up again in shorter chunks and ask them a load of questions which only tests immediate recall.

Bored at the computer

If it’s eInformation / eReference that’s needed, that’s fine. We can make that more visually appealing and easier to read on screen. We can even make it more enjoyable to view in the form of videos or podcasts.

Where the real learning takes place is in the analysis of the material in relation to a specific work-based problem. A problem that the learner is likely to face in the workplace.

An example I use is the mandatory fire safety course. It tends to be boring when done in the classroom where facts upon facts about the fire triangle are poured into learners’ heads. If they’re listening carefully enough they might be able to answer some questions on the fire triangle and what constitutes fuel, heat and oxygen (I’m still not sure if I’ve got it right). Tell me, in a fire how many people will be standing there pondering on the fire triangle. Really what would do us more good is to either assess realistic risks, or evacuate safely in the event of a fire.

The fire drill

Encouraging more of a user-generated and peer-to-peer learning environment may not be to everyone’s taste so a VLE such as Moodle will give more control. But L&D’s real and untapped value will be in the nurturing of learners, working with SMEs to provide digestible chunks of information, designing bite-sized resources and providing study guides and recommended personal learning plans so learning becomes more individual and task based.

Definitely, why force individuals to go through the same mandatory content year after year when all they may need is a yearly, skills based assessment. If that assessment highlights skills gaps then a more flexible learning programme will make sure individuals learn only what they need not what they don’t.

It’s no longer about what we know but more about where to find the information and apply it to tasks.

I couldn’t agree more.

Thanks Laura!

How to revamp your learning model

In my articles Online courses must die! and The ILE and the FLE in harmony, I advocate the development of a virtual Informal Learning Environment (ILE) to work in tandem with the Formal Learning Environment (FLE) to support both the learning process and its administration.

Heeding the advice of Bill Brandon, I will now flesh out that idea with an illustration of how it might be implemented in a real organisation.

Informal learning

I believe in the power of informal learning. In fact, I go so far as to say it should be the central philosophy of the organisation’s learning model.

In a practical sense, that means we need to provide our learners with tools and resources that they can use to drive their own development.

This is where the ILE fits in: It’s a space (like a website or intranet site) that centralises those tools and resources.

The ILE illustrated

There are a thousand and one possible combinations and permutations of an ILE.

However, if I were to consider (read “fantasise”) a greenfield opportunity (read “pipedream”), what would I design?

Essentially I would base my design on three core components, as illustrated in Figure 1.

Informal Learning Environment, consisting of a wiki, a discussion forum and personal profiles

Figure 1. Informal Learning Environment

Core component #1: Wiki

The primary component of my ILE is a comprehensive wiki.

In a big corporation like the one I work for, knowledge is distributed everywhere – on obscure intranet pages, in random folders, in people’s heads – which makes it really hard to find.

A wiki enables the organisation to centralise that collateral, whether directly (by inputting it) or indirectly (by linking to where it exists elsewhere), thereby functioning as the first port of call.

A wiki can contain – or point to – all manner of media, such as text, graphics, documents and multimedia. The learner can search and explore the content that’s relevant to them, just-in-time if need be.

The flexibility of a wiki also allows anyone to contribute content. This empowers the learner to share their knowledge with their colleagues, build on the knowledge that has already been contributed by others, and communally keep it up to date.

Core component #2: Discussion forum

The secondary component of my ILE is an open discussion forum. I say “secondary” because my rationale is that, if the learner can’t find the knowledge they need in the wiki, they can crowdsource it via the forum.

A discussion forum enables the learner to post a question to their peers, thereby leveraging the collective intelligence of the organisation. Of course the learner can also share their knowledge by answering someone else’s question, and they can learn incidentally by reading the questions and answers of others.

The questions posted to the forum may also serve to expose knowledge deficiencies in the organisation, which can be remedied by updating the wiki!

Core component #3: Personal profiles

The tertiary component of my ILE is a bank of personal profiles. I say “tertiary” because my rationale is that, if the learner can’t find the knowledge they need in the wiki nor via the discussion forum, they can target an SME directly.

For example, if the learner is struggling with a Java programming problem, they can look up a Java expert in the system and send them a direct message. The SME may be recognised as a “Java” SME because they have said so in their profile, or – if the technology is sophisticated enough – their contributions of Java-related content in the wiki and participation in Java-related conversations on the discussion forum flag them as such.

I’m in two minds as to whether a full-blown social network is useful for internal learning purposes. Apart from profiling, I’m not convinced that friending, status updating and other Facebook-like activities add much value – especially when a discussion forum that accommodates groups is already in place.

Formal learning

Self-directed, informal learning is great. However, there are some things your employer must know that you know.

The most obvious example is compliance, eg privacy, trade practices and OH&S. If you breach the regulations, the company will be in hot water, so they’re not just going to take your word for it.

There are plenty of other examples, such as a certain level of product knowledge, that may be critical to the role.

In a practical sense, this means we should map required competencies to each role and assess the employee’s proficiency against each one. That probably leads to a development plan, which in turn forms a subset of the performance agreement and is subject to regular appraisals.

Then there are formal training events like courses and workshops that are important and require documentation, and some people want their informal learning (eg reading a book) recorded too.

The FLE is a space (like a database or platform) in which all this administration is done.

The FLE illustrated

Again, there are a thousand and one possible combinations and permutations of an FLE.

However I base my design on two core components, as illustrated in Figure 2.

Formal Learning Environment, consisting of an LMS and reports

Figure 2. Formal Learning Environment

Core component #1: Learning Management System

The primary component of my FLE is a Learning Management System (LMS).

The LMS is an oft-derided yet invaluable educational technology. I suspect the typical organisation under appreciates it because it uses it illogically.

My advice is to use the LMS for what it’s designed for: managing learning. Competency maps, auto-marked assessments, registrations, completion statuses, grades, transcripts, performance agreements and performance appraisals are what the LMS does well. Some even extend into talent management and other HR domains.

Conversely, my advice is to avoid using the LMS for what it is not really designed for: managing content. Leave that to the ILE, which is a much more open and flexible environment, and is purpose built to support “learning”.

Core component #2: Reports

The complementary component of my FLE is the range of reports that can be generated from various systems to provide useful data. Such data may include productivity statistics, quality scores, complaint volumes, engagement indices… whatever can be analysed to identify training needs and/or evaluate learning outcomes.

At the end of the day, learning must support performance.

Putting it all together

My revamped learning model, then, comprises two discrete but related virtual environments:

1. An ILE, and
2. An FLE.

The former supports the process of learning; the latter supports its management.

A revamped learning model, consisting of an ILE and an FLE

Figure 3. A revamped learning model

Separating the two environments like this aids in segregating them in the human mind.

Why bother?

Because learning should be a joy.

By definition, an ILE should be unforced, unscored, unthreatening.
It should be a safe, open space where people are excited to go because they want to learn, without the burden of forced navigation and pass marks.

Simultaneously, an FLE should focus on what really matters. Too often when formal and informal learning are mixed, goals blur and we run the risk of formalising for formalising’s sake. We don’t need to monitor our colleagues like Big Brother; we just need to assess them when necessary.

How long is a piece of string?

Of course, many more components may be reasonably argued for inclusion in the learning model.

An onsite classroom, for example, is obviously a part of the formal learning environment. So too is a university campus on the other side of town.

In terms of informal learning, the water cooler, a cabinet of books – and even the pages in a book – may be considered components of the ILE.

How about a library of online courses? That might be considered a component of the ILE if the learner is free to explore it at their convenience, but it will suddenly revert to the FLE if the learner is instructed to complete a particular course.

Clearly then, the ILE and the FLE are elastic concepts, highly dependent on perspective and context. That’s why I have focused on the core components that I think can provide a universal framework for a revamped learning model.

The two virtual environments are constant; everything else around them is variable.

Online courses must die!

A touch dramatic, isn’t it?

Now that I have your attention, please bear with me. There’s method in my madness…

The myth of rapid authoring

The proliferation of so-called rapid authoring tools over the last few years has coincided with an explosion in the number of online courses developed in-house.

In the bad old days, technically challenged L&D professionals had to pay exorbitant fees to development houses to produce simple modules. These days, however, everyone seems to be creating their own online courses and distributing them via an LMS.

In tandem with this trend, though, has been the increasingly familiar cry of “It’s not interactive!”. Critics rail against boring page turners – and rightly so.

Woman bored at the computer

But you know what? Even when L&D professionals consciously integrate interactivity into their online courseware, I usually don’t think it’s all that engaging anyway. Increasing the number of clicks required to view the content does not make it more interactive. It just makes it annoying, especially for time-poor employees in the corporate sector.

Yes, I know you can embed real interactivity into courseware via games, branched simulations, virtual worlds etc, but hardly anyone does that. It requires time – which you don’t have because you’re too busy building the online course – or dollars – which defeats the purpose of developing it in-house!

So what’s the alternative?

Frankly, there’s nothing most online courses do that a PDF can’t. Think about it: PDFs display structured text and pretty pictures. Just like a typical online course, without the fancy software or specialist skills.

Anyone (and I mean just about anyone) can create and update a PDF. Suddenly SMEs are back in the game… Write up a Word doc and convert it? Easy. Update the Word doc and re-convert it? Easy.

Now that’s what I call rapid.

The best of both worlds

If we dispense with online courses in favour of PDFs, how can we incorporate interactivity into the learning experience?

Enter the Informal Learning Environment (ILE).

Occupying a place on the continuum somewhere between a VLE and a PLE, an ILE is an informal learning environment that a facilitator manages on behalf of a group of learners.

Essentially, an ILE is a space (like a website or intranet site) that centralises relevant learning resources in a particular domain. The site may host some of those resources and point to others that exist elsewhere.

So your PDFs can go in there, but so too can your audio clips, videos, puzzles, games, quizzes and simulations. Don’t forget podcasts, RSS feeds, slideshows, infographics, animations, articles and real-life case studies. Not to mention blogs, wikis, discussion forums and social bookmarks.

Unlike a VLE, an ILE is strictly informal. The learners can explore its resources at their own pace and at their own discretion. No forced navigation, no completion status. In this sense, the pedagogy is constructivist.

Unlike a PLE, an ILE is communal. It exists to support a community of practice, whose members can (or more accurately, should) incorporate it into their own respective PLEs. In this sense, the pedagogy is connectivist.

But that’s not to say that the pedagogy of an ILE can’t be instructivist either. The facilitator should provide a learning plan for novice learners which defines a sequence of study, identifying specific resources among the potentially overwhelming array of options.

The sky’s the limit

An ILE is a scalable and flexible learning environment. If we view each resource within that environment as a learning object, we can appreciate how easy it is to add new content, update old content, and remove obsolete content.

Four marbles

It’s incredibly inefficient to use up the precious time of an L&D professional to build, publish, test and upload an online course, only to edit, re-publish, re-test and re-upload it later, just because a few words need to be changed and a graph replaced. Instead, the SME can create and update the object via Word.

If you are keen on creating interactive tutorials, games or virtual worlds, now you can go for it! You have more time, and new tools are coming out that are making these kinds of thing easier to do. The finished product can be added to the ILE as another learning object. Again, if it needs to be updated later, there’s no need to edit, re-publish, re-test and re-upload a whole course – just that object.

If you commission an external developer to build a smokin’ hot immersive scenario, guess what: you add it to the ILE as another learning object. When it needs to be updated, you pay the developer to work on that object and that object only.

In this age of iPhones and Flip cameras, why not encourage your learners to generate their own content too? It’s another rich source of objects to add to the mix.

All these examples illustrate my central premise: when content is managed in the form of independent learning objects, it remains open and flexible, which means you can keep it current, relevant and organic.

Rockin’ role

Under the ILE model, the role of the L&D professional finally evolves.

The SME is empowered to produce content, which frees you up to apply your own expertise: instructional design. This may involve a greater focus on engagement and interactivity.

The responsibility of learning is assigned back to the learners, which frees you up to guide, scaffold, encourage, discuss, prompt, probe, challenge and clarify. In other words, facilitate learning.

Your value in the organisation goes through the roof!

Take the ass out of assessment

I claimed earlier that there’s nothing most online courses do that a PDF can’t. I glaringly omitted assessment. Please note I left it out on purpose.

There are just some things that the company must know that you know. You get no argument from me on that.

However, how we assess that knowledge is bizarrely old fashioned.

A donkey

While it’s convenient to wrap up some content and a quiz into a single package, I just don’t see the point from an instructional design perspective. Forcing someone to register into a course, just to pass a dinky quiz at the end, doesn’t make a lot of sense to me.

It is widely acknowledged that the vast majority of learning in the workplace is informal. From exploring an ILE to chatting around the water cooler, there is a myriad of ways that people learn stuff. Assessment should represent the sum of that learning.

This is where the LMS comes in. In my view it should manage assessment, not content. More specifically, it should deliver, track and record standalone tests that are linked to particular competencies.

When the LMS is used in this way, the L&D model aligns more closely with the learning process. The employees learn informally all over the place, using an ILE as their central support resource, then (if necessary) they record their competence. The focus of measurement shifts from activity to outcome.

This unorthodox approach makes many people nervous. Their primary concern is that someone can jump straight onto the test and pass it immediately, without ever “doing the course”. In response, I make these three points:

  1. You can jump straight to the assessment in most online courses anyway.
  2. If someone bluffs their way through the assessment and passes, clearly it wasn’t robust enough. That’s your fault.
  3. Conversely, if someone passes the assessment because they already have the knowledge, what’s the problem? You are recording competence, not making people’s lives difficult.

Of course, this kind of nervousness isn’t confined to the corporate sector nor to e-learning. For example, many universities have a minimum 80% attendance policy for face-to-face lectures. I don’t see the point of turning up just to fall asleep with my eyes open, but that’s another story!

The method in my madness

Online courses must die because they are unsustainable in the modern workplace. They aren’t rapid, flexible or scalable, and they usually don’t take full advantage of their medium anyway.

So unlock your content and manage it in the form of individual learning objects in an ILE.

Shift the bulk of the content to PDF. In the age of e-readers, no one will notice much difference.

By all means invest in authoring tools, but only in ones that will help you create interactive and engaging objects – easily.

Exploit Web 2.0.

Use standalone tests to record competence on your LMS. They cover all sources of knowledge.

Informalise learning. Formalise assessment.