Choose A) Performance-Based Test, or B) Multiple Choice Test

Performance-based testing isn't a new idea but it certainly seems to be one whose time has come to the realm of IT certifications. While many certifications are still based on traditional "paper and pencil" formats, there appears to be a move afoot in IT to embrace performance-based testing, or at least some combination of performance-based and traditional testing, as a measure of competency to achieve certifications.

The Cisco Certified Internetwork Expert (CCIE) has long required candidates to pass both a written and performance-based exam to achieve the credential. CompTIA also requires both performance and traditional written exams for several of their credentials such as the CompTIA A+ and Network+ certifications. Cisco and CompTIA aren't alone in their use of both performance and written exams.

Why are we experiencing a move to performance-based testing? Is performance-based testing better than traditional paper and pencil (or computer-based) testing formats? Good question! The answer is not as simple and straight-forward as you might think. Believe it or not, there is a fine art to developing an effective assessment and drafting effective exam questions. I should know — I spent a long painful summer in a Testing and Measurements course learning all the ins-and-outs of testing methods and writing effective test questions.

To understand how we test, perhaps we first need to understand why we test. According to the Center for Public Education, "Assessment is the essential ingredient of accountability and accountability is the key word in education today." In plain English, we test to ensure than the skills and knowledge gained are what we think they should be. A good assessment should validate that skills and knowledge meet a certain quality and level of competency.

How we choose to measure that information, skill and competency (traditional versus performance-based testing) is an entirely different question. The best testing format depends in large part on what you're trying to assess and what goal you're seeking to achieve as a result of the assessment. Both traditional and performance-based testing have unique functions and differ both in their purpose and in what they successfully measure.

Traditional "Paper and Pencil" Testing

Most of us are familiar with traditional, standardized testing. While traditional testing may include many formats — multiple choice, fill-in-the-blank, matching, short answer, and true/false, for example — we'll concentrate on standardized multiple-choice exams for purposes of this article, as this type of assessment is the one most frequently encountered in certification exams.

Multiple choice exams are a popular exam format. They're great if you need to test facts or large amounts of data sets.  A few (but by no means all) of the reasons multiple-choice exams are so popular include:

� Easy to grade — With the advent of computer-based testing and scoring keys, exam results can be obtained almost instantaneously.

� Versatile — Easily adaptable to assess multiple skills from problem solving, reasoning, information analysis, drawing inferences and conclusions, and knowledge of data sets.

� Reliable — Provides scoring consistency; exam results can easily be tracked to specific knowledge domains.

� Efficient — Provide ability to test a sampling of multiple knowledge domains in a single exam.

� Diagnostic — Easy to identify patterns where knowledge gaps occur.

Multiple choice exams are not without their disadvantages — or their critics. Common complaints include the fact that all possible answers are provided to students. As a result, students do not have to initiate or provide any independent responses, or engage higher-level thinking and problem solving skills.

(As I make this observation, it should be noted that writing an effective test question is an art. An experienced and careful test writer can craft multiple choice questions which engage some measure of higher-level thinking skills. The quality of the question depends on the skill of the creator of the question!)

The outcome of multiple choice exams can be affected by external factors. For example, some students may possess better reasoning and deductive skills enabling them to deduce answers which they may not otherwise know. By their nature, multiple choice exams are limited in the types of information which they can measure and tend to focus on independent applications and memorization of facts and data. While multiple choice exams can be used to test your knowledge of electrical standards, they cannot be used to test your ability to implement those standards and perform tasks that conform to those specifications. 

Performance-Based Exams

Unlike traditional multiple choice exams, performance-based assessments require the student to independently generate a response or answer. Sometimes referred to as an authentic assessment, performance-based exams are historically used when the goal of the assessment is to evaluate the ability of an individual to perform a particular task, demonstrate a certain skill set, or confront and solve a problem in a real world, real-time setting. Performance-based exams are focused on what you can do with the knowledge you possess.   

Typically, you'll see performance-based exams approached through two different formats: a single, individual performance test or a performance portfolio. In the first scenario, candidates are presented with a single hands-on performance-based exam which is taken at a specific time and location, with a specified duration of time allowed to complete. Completion times may range anywhere from a few hours to several days. Depending on the certification provider, performance-based exams may be administered at a certified testing center or in a virtual lab environment.

Rather than a single exam, some credentials may require submission of a performance portfolio demonstrating the candidate's ability to successfully complete a variety of specific skill-based projects or tasks (such tasks may be referred to as "challenges" by some credentialing entities). These tasks may be conducted independently or under the supervision and direction of a project monitor. Candidates may be allowed anywhere from a few weeks to a few months (or longer) to complete the required tasks and prepare their final performance portfolio.

Performance-based exams require candidates to use complex thinking and reasoning skills, problem-solving abilities, and higher-level thinking skills. These types of exams provide for evaluation of multiple competency levels.

While perfect for evaluating practical skills and abilities, or mapping skills gained against a specific curriculum, performance-based exams have a few drawbacks. First, they are expensive for both the credentialing agency and the candidate. Credentialing agencies must set up an adequate virtual or physical environment in which to conduct the assessment and then recruit qualified individuals to develop, administer and evaluate the exam results.

Candidates may be faced with purchasing costly equipment to set up a virtual environment or subscribe to virtual services to practice required skills. In addition, performance-based testing locations are often limited so candidates may incur the additional cost of lost time from work, as well as travel and lodging expenses during testing.  

Which assessment is better?    

The million dollar question is which type of test is better: performance-based, or a more traditional, standardized style exam? Both exams are high-stakes with consequences attached to the outcome.

Again, the answer depends on what you want to assess and measure. If you're looking to assess knowledge of facts, such as standards or general knowledge, then a multiple choice exam is a good approach. If you want to know what a candidate can actually do, however, then a performance-based exam is a better approach. 

From a purely personal perspective, my preference is a combination of both written and performance-based exams. Based on past experience, the best results in understanding a student's true knowledge, skill, and competency were achieved when using a combination of both approaches. For example, if I were teaching history and wanted to know if students knew the timeline of when certain events occurred leading up to the Civil War, I'd administer a multiple choice test because I'm interested in knowing if students understood relationally when certain event occurred. If, on the other hand, my goal were to assess a Civil War reenactor's ability to handle a Model 1860 Light Cavalry Saber while leading a charge on horseback, then I'd select a performance-based assessment.

If the goal of certifications is to assess both general knowledge, as well as the ability to actually perform certain tasks in a real-world environment, then I think we'll see more credentialing authorities expanding certifications to a more robust assessment which incorporates both approaches.