Instructor Course Application
A formal 10-Step Evaluation identifies critical dimensions and areas for improvement of an application process, designed to biannually identify the top 5 of 300 qualified candidates, for admission to an exclusive tactical training program.
Note: This is information from an actual evaluation project. The Corporation, OP, OPIC and related terms are pseudonyms used to protect the privacy of the client and to safeguard sensitive information. No associations are claimed, nor should be assumed, from any images, data or information herein.
The Corporation (a pseudonym) fills a vital role in national and global physical security, providing tactical and strategic weapons support. The OnPoint Officer (OP) is a key leadership role within the Corporation. OPs require high-level specialized training, delivered by OnPoint Instructors (OPIs).
The five and one-half month OnPoint Instructor Course (OPIC) is offered only twice per year and accepts only 4 – 6 out of approximately 300 eligible OPs each class. In 2020, the OPIC developed a scoresheet to screen candidates’ applications prior to selecting a maximum of 12 for in-person interviews. The introduction of the scoresheet effectively created a two-phase OPIC application process:
The Corporation OPIC leadership (client) developed Phase-I, and they were the main reviewers and graders for the Phase-I assessment. As upstream stakeholders, their main concern was determining the effectiveness of Phase-I in selecting the best qualified candidates for OPIC.
Future OPIC applicants are direct stakeholders as users of Phase-I, as are the OPIC staff who depend upon the utility of Phase-I and the quality of the selected candidates it produces for OPIC training.
The U.S. and foreign military partners are downstream stakeholders. The critical function performed by OPIC graduates and the OPs they train will significantly contribute to their success in ways that can neither be quantified or disclosed in this report and are beyond the scope of this evaluation.
Following the 2020 Phase-I rollout, the client requested a program evaluation to determine its effectiveness in consistently identifying the top 2% of 300 qualified OPIC candidates.
The program evaluation project team consisted of three master’s degree students enrolled in the Evaluation course offered by the Organizational and Workplace Learning (OPWL) department at Boise State University. The team conducted this program evaluation as our OPWL 530 Evaluation course project (Robertson et al., 2021).
The client intended to use the evaluation findings to identify areas for improvement in Phase-I of the OPIC application process. Accordingly, the evaluation team conducted a formative goal-based evaluation with a goal-free approach to determine how well the Phase-I scoresheet system selected candidates with the highest potential to succeed in OPIC. The evaluation team also incorporated a goal-free evaluation approach to investigate if there had been any unexpected results from the recent addition of the Phase-I scoresheet.
The evaluation team employed an evidence-based 10-step evaluation process (Chyung, 2019) to conduct the identification, planning, and implementation phases of the project. (See Figure 1)
Figure 1. 10-Step Evaluation Process
Having identified the stakeholders of the Phase-I evaluand, and the formative focus of the evaluation, the team developed a Program Logic Model (PLM) (Appendix A). Using the PLM, the team selected the three most pertinent program dimensions for investigation and established the relative importance weight (IW) of each, as listed in Figure 2.
Figure 2. Three Evaluation Dimensions
Table 1. Summary of Data Collection Instruments
The three dimensions were measured individually using multiple data sources and measurement methods. The evaluation team applied “critical multiplism” in the selection of methods, and triangulated the resulting data sets, thus verifying what any one source indicated, with information from the other sources (Chyung, 2019, p. 118). A customized two-stage (instrument/dimension) rubric was used to score each dimension of Phase-I’s performance as an OPIC candidate selection tool, based on the desired (expected) performance relative to each dimension. (See Instruments and Rubrics Development Worksheet, Appendix B)
Table 2. Evaluation of OPIC Phase-I Performance Relative to Each Dimension
Specifically, regarding the primary and most important dimension (Dimension #1), and the ability of Phase-I to assess applicants’ five macro-characteristics:
To better assess relevant characteristics of applicants during Phase-I, use additional sources of data to gain more in-depth information (e.g., personality and aptitude tests, and direct supervisor surveys).
To better interpret Phase-I for objectivity and consistency, the scoresheet should mirror the application format, and it should include a clear scoring guide for each area of the application.
To better prepare applicants for Phase-I, market the OPIC program and its application process to leadership and potential applicants using a comprehensive multimedia approach within the Corporation.
American Evaluation Association. (2018). American Evaluation Association guiding principles for evaluators. Retrieved March 14, 2021, from https://www.eval.org/About/Guiding-Principles
Chyung, S. Y. (2019). 10-Step evaluation for training and performance improvement. Sage.
Joint Committee on Standards for Educational Evaluation. (n.d.). Program evaluation standards. https://evaluationstandards.org/program/
Robertson, J., Blanchard, M., & McCraw, V. (2021). ‘OnPoint’ instructor course (OPWL 530 Evaluation Project Spring 2021) [MS Word].