When constructing a test, for whatever purpose, the question of test validity needs to be addressed. Test validity asks to what extent does the assessment tool being used truly measure what it purports to measure (known as construct validity in testing jargon). Take for example the classic I.Q test in the 1960s. These tests often found to be “invalid”, because they weren’t really measuring intelligence as they claimed to do, but rather, they were testing reading skills under time constraints.The test often failed to spot intelligent children who may have simply possessed poor reading skills due to poor training, lack of practice, motivation, dyslexia, etc. In time, the development of I.Q. tests changed and included, as well as reading, oral questions and responses, drawings, visual recognition, tactile exercises, and discussions. Train the Trainer Program
Criterion Validity
But the story continues. Researchers found that even though tests could be constructed to more accurately measure a given criterion, there was still the question of knowing whether the criterion itself was also meaningful. What exactly did the variable(s) being measured mean? Did that which was being measured have a meaningful connection to what was truly happening in reality (i.e. criterion validity)?
The key was to understand the big picture. For example, for years language classes were taught with a heavy emphasis on learning grammar, the so-called rules of a language. The assumption was based on the surface observation that articulate native-speakers of a language knew and could use their own grammar very proficiently. Therefore, it was assumed that to learn the grammatical rules of a language (along with vocabulary) would transfer into acquiring the language for the second-language learner. Teachers essentially then taught grammar lessons - with a corresponding method for teaching it (e.g. usually academically). Students struggled to learn it, tests were designed to test their knowledge of it, and grades were assigned, based on the test scores, to determine their level of proficiency. The result? Once out of the classroom, “high-proficiency” second-language learners were barely able to formulate even the simplest sentences in a foreign language they studied for years, let alone comprehend that language from native speakers who spoke it at a normal tempo.
So what went wrong?The tests were indeed accurately measuring grammatical comprehension of the second-language learner (construct validity). But the bigger picture of what really happens in language learning was misunderstood. The question of whether conscious knowledge of grammar is a true correlate of language proficiency turned out to be a false assumption based on a partial and misunderstood observation of how language learning works. The tests failed the criterion validity. Indeed, a closer look revealed that many native-speakers of a language could also not consciously explain their own grammar. Therefore, “knowing” grammar explicitly was not a prerequisite for using it proficiently. Something else was at play here which the original observations had also missed. Train the Trainer online
To design a criterion-valid test requires a subtler and deeper level of questioning about both our observations and assumptions (and how they inform each other). We need to ask if what we are measuring is clearly related to some concrete criteria in the real world. Getting it right (or wrong) has great implications for the way we design our curriculum and develop our training methods.
Asking the Right Questions about Technical Product Training
In the business world, just consider how many marketing departments have designed product training seminars (usually of more sophisticated high-tech products) around simply knowing the features of that product (similar to knowing the rules of a language above). When asked, “Why is this important?” the response is typically, “Because sales engineers have to know about the features of the product in order to offer our customers the right solution for their needs.” As we have seen, on the surface a test can be designed to assess if a salesperson has learned the features of a certain product (construct validity). But at a deeper level it may be missing the bigger picture, but still deliver “satisfactory” results of a reality that does not exist in the concrete world. Again, to achieve criterion validity, one must ask if “product knowledge”, is the key criteria for a technical salesperson’s ability to offer a customer the right solution? The word “key” is central here simply because the overwhelming type of technical training offered in high-tech companies consists in having an engineer “describe” the so-called “rules” of the product (i.e. its features and how it works). The content and training method betray the underlying assumption about our observations of what makes for a successful encounter between a sales rep and a customer (i.e. product knowledge). Train the Trainer Courses– online
But our assumptions are rarely questioned, and in many cases, remain hidden from conscious awareness. We’re usually content to make surface observations and base our needs-analysis (and by extension, our training) on that. To get a bigger and better picture of what’s really happening is not as difficult as it may seem. It simply requires asking questions, at a few layers, about what we initially consider “the source” of a competency under analysis – and then test to see if it is true in “real life”.
The answer we arrive at will also influence the content of our training – and the approach we use to carry out that training. For example, if we assume that product knowledge is the key to speaking to customers, then we will design our courses around training sales representatives to learn as much knowledge about a product’s features as possible. It will be an information rich approach given either by a speaker (with detailed PowerPoint slides) or by having the rep essentially read through dense product manuals. But if, in reality, something else is occurring when offering a customer a solution, then the content, and probably the training methods used, will have to be radically reconsidered.
In reality, when analyzing many successful sales reps, one finds that it is the ability to listen attentively to the customer’s situation, understand the problem in a big-picture context, ask meaningful questions, handle doubts, build rapport with the customer, and cooperate on a solution that are only some of the key criteria necessary for a successful sales encounter.
Summary
Significant resources in time, money and energy are often lost on training every year, whichis based on an incomplete understanding of what makes for genuine competency in a specific area. That is why valid testing methods and training go hand in hand. Testing our observations and assumptions – with a more expansive and accurate analysis of what is really happening when defining proficiency for any competency is crucial to seminar design and training methodology. But it will require the skill of asking good questions at multiple layers – and being prepared to make some major changes, based on our expanded findings, in the content and the approach we take to train our staff.
International Presentation Academy
At the International Presentation Academy, we will help you with designing interactive training modules (also online) that includes designing interactive valid tests for the material you present. Our Train the Trainer Courses – online will guide you through a step-by-step process in developing your own engaging and highly didactic training module. This Train the Trainer in Muenchen (also available online)seminar will show you how to structure you seminar for maximum effectiveness, develop easy-to use activities, as well as guide you through a hands-on approach at creating games and simulations for learning purposes. And finally, you will create an audio PowerPoint presentation (during the workshop) that you can use as classroom input or as pre-seminar preparation for your own training.
Our Train the Trainer Courses areappropriate for those who train soft skills of every kind as well as technical skills. We will engage you with the topic from beginning to end – using a completely hands-on approach. Indeed, we employ the very methods we advocate. We invite you to join us at one of our open seminars, or contact us for an in-house quote at: [email protected]
Comments