COMPANY STANDARDS: AN ALTERNATIVE TO PHYSICAL PERFORMANCE TESTING
BY JOHN LECUYER
Physical performance assessment (PPA/task-based testing) has been a high-profile issue over the past five years, particularly with incumbents. Attempts to resolve this problem have been unsuccessful, as evidenced by the demise of the National Fire Protection Association (NFPA) 1583, Recommended Practice For Fire Fighter Physical Performance Assessment, the controversial draft document that addressed fitness and physical performance. The 1583 document was never issued and was returned to committee in 1996 following an appeal to the Standards Council.
Associated with that decision was a legislative directive in the 1996 amendments to the Age Discrimination in Employment Act (ADEA) for the National Institute for Occupational Safety and Health (NIOSH) to conduct a study and issue advisory guidelines on fitness and job performance requirements for police and fire departments by September 30, 1999. Though directed toward agencies that enforce mandatory-age retirements, the regulations will affect all departments. Although Congress authorized the funding, no money was appropriated in the fiscal budget. To help resolve this issue, I propose restricting the use of PPA to entry-level candidates, using the company standard for incumbents.
COMPANY STANDARDS VS. CTTESTING
A company standard is a minimum task performance requirement that a company working as a team must meet. Company performance standards for incumbent evaluations can be used in place of a physical performance assessment. The more appropriate term for PPA is the criterion task test (CTTest). Substituting company performance standards for a CTTest (for incumbents only) is based on the need to evaluate members in a manner that more accurately reflects the actual job, is commensurate with individual assigned positions, and retains the essential functions included in the CTTest.
The CTTest evaluates candidates on the essential job functions. Since restrictions on preemployment tests necessarily exclude skill-dependent tasks and full use of personal protective equipment (PPE) (e.g., self-contained breathing apparatus) and incorporate a sample of tasks that are typically out of sequence or overlap standard company assignments, the value of using a CTTest protocol for incumbents is questionable. Though task restrictions are necessary for entry-level testing, they are irrelevant for incumbent evaluation. Incumbents are expected to possess the requisite skills, and through several company evolutions all tasks–including those deemed essential–can be assessed in a format that more accurately reflects the work environment. Substituting standard company evolutions that include the essential tasks is the most logical and appropriate method for evaluating incumbents.
In terms of test development, the CTTest is best designed by extracting essential tasks from department standard evolutions. Theoretically, the process is reversed when viewing company standards, in which the essentials are put back into the evolutions. The concept has advantages and disadvantages.
Advantages
–Matches company standards to actual job tasks. This helps resolve the complaint, “We don`t do all those tasks at one time or in that sequence.”
–Serves as a physical performance assessment and as a training exercise. It meets training hour requirements and eliminates the need for two separate scheduled sessions.
–Supports entry-level CTTest validity. Includes essential tasks specific to each department with written documentation appearing in standard operating procedures (SOPs).
–Evaluates teamwork and individual task performance. The team as a whole and the individual in the team are assessed.
–Evaluates specific job tasks based on individual assignment. A truck officer who would never be required to handle a hoseline or be evaluated on a hoseline operation may be evaluated on another task that is commensurate with assigned duties.
–Skill proficiency is evaluated in addition to physical work capacity. The skill in doing the task and the physical ability to perform it are both examined.
–All the tasks required for each evolution are evaluated, not just the essential tasks. Marginal and essential tasks are as-sessed.
–Task assignments match job descriptions. Assigned tasks reflects normal job duties.
–Meets legal requirement to evaluate incumbents on job performance essentials. Assessment covers minimum essential skills needed to perform task in accordance with applicable laws (e.g., Americans with Disabilities Act).
Disadvantages
–Requires performance of multiple evolutions to encompass all the essential tasks. This takes more time to fully evaluate an individual.
–Some members may not perform all the tasks. However, members could be rotated to ensure everyone completes most of the tasks.
–Requires multiple evaluators to observe all company members performing simultaneously. Staff availability may present a problem.
–Requires careful design, planning, and the development of company time-based standards. The process must be carefully set up to ensure accuracy and relevance.
GENERAL GUIDELINES
Company standards should reflect actual job requirements such as staffing, specific task assignment, and the appropriate PPE use. If firefighters are routinely assigned to different companies (truck vs. engine), they should be evaluated in both capacities. If firefighters are assigned one permanent position, the evaluation should be restricted to that specific as-signment. Special assignments related to department operations (e.g., heavy rescue, haz mat, confined space) should also be evaluated and include essential tasks specific to those assignments that ordinarily would not be required of entry-level candidates or other members. If members are occasionally as-signed alternate duties or re-quired to perform in a different capacity based on operational procedures, they should be evaluated in the alternative assignments, too.
RELATIONSHIP BETWEEN COMPANY STANDARDS AND CTTEST
Professional test developers should be contracted to develop and validate a CTTest. Company standards are the basis of the validation and don`t require professional assistance. During the Job Task Analysis (JTA), department SOPs, which include company standards, should be reviewed to identify essential tasks. Figure 1 illustrates how essential tasks can be extracted and added.
Figure 1 presents four sample company evolutions in a simple format with potential tasks identified to include in the CTTest. The essential tasks can be pulled from any number of evolutions, and more than one task can be identified in any given evolution. In Ladder Company Standard 3, two tasks were extracted; conceivably at least four could be selected. The second task in that evolution consists of a ladder carry, raise, and extension. Since the ladder raise qualifies as a skill-dependent task, it should not be selected. However, other portions of the task can be used (e.g., the ladder carry or ladder extension).
If the CTTest is assembled from the essential tasks identified and pulled from the company standards as recommended, the tasks selected must meet the criteria for a CTTest battery:
Critical and essential.
Nonskill-dependent (minimal skill).
One firefighter can perform them.
The most arduous tasks.
Reflect actual job performance requirements.
When the CTTest is assembled from the company performance standards, the company standards can be used as a substitute for evaluating incumbents as long as the evolutions to be performed include all the essential tasks. It is thus unnecessary to administer the CTTest to incumbents on any given schedule, since the same CTTest elements are evaluated within the company standards format.
EVALUATION
Individuals can be evaluated effectively using the standard concept in lieu of the CTTest. Using the subject matter expert (SME) concept, raters would employ a Likert-type scale to grade performance (Figure 2). Though the SME method may appear highly subjective to the novice, it is an accepted valid procedure.
The evaluation process requires multiple graders so that at least two (preferably three) raters grade each individual. The resultant scores are averaged to arrive at a final score. The team effort is graded based on a time standard. Individual effort is evaluated by SME observation using the scale above. I have employed this evaluation method with recruit firefighters and have found it very effective and surprisingly accurate. In any organization, if members were to rank other members on job performance, there would be top performers, the mid-group and average, and the low performers. We can all easily identify the top performers and the poor performers and do so informally every day. Most members who fall between the two extremes are more difficult to pigeonhole and exhibit greater variance in their SME ratings. However, the variance in the mid-group is practically inconsequential, since it is the grouping that is most important. Distinguishing between acceptable and unacceptable performance and particularly identifying the low performers are the priorities.
When this method is applied in a training environment with recruits, officers with no prior contact with the recruits or personal knowledge of their performance ability easily identified the groups, with the top and bottom performers in some cases appearing in an almost perfect rank order. The key to the evaluation is using experienced raters, preferably training officers. However, any competent officer with adequate experience will serve well if properly instructed on the process.
THE PROCEDURE
A company is assigned an evolution and given the time standard for completion. The time clock starts as soon as the evolution begins and the raters observe both individual group performance for proper procedures and the overall operational smoothness. Individual raters are assigned one or two members to observe specifically. The individuals and the company are then graded using the scale, and the performance time is recorded on completion. A second evolution is assigned, and the raters observe and evaluate a different member. The process continues for all subsequent evolutions until the predetermined number of evolutions is completed and each member has been evaluated by at least two raters. The procedure should be carefully designed to ensure all members are evaluated in their primary and alternate positions. Although the evaluation specifics are beyond the scope of this article, the process is far less tedious and time-consuming than it sounds. I have found it less cumbersome than administering a CTTest.
Although we use the company standards concept principally to evaluate member and team performance, the most important aspect of substituting the process for the CTTest is to identify the low or unacceptable performer. Once identified, proper action can be taken to modify performance. In Figure 3, the process is outlined with the three principal categories identified.
The company performances are graded on a pass/fail system based on the time standard determined for each evolution. If the team does not meet the time standard, training is indicated. If unacceptable performance is attributed to one or more members, the individual problem(s) should be identified. Be aware that an individual could score poorly even if the company met the time standard, so do not overlook the individual ratings. Both the company score and each individual`s score must receive an acceptable rating. Categorically, all potential problems fall under one or more of the following:
1. A physical fitness or medical problem. A physical fitness deficiency is identified by inspecting an individual`s fitness profile, then prescribing an exercise program to correct the deficiency. (Note: The fitness program and fitness test are different programs from the performance program under discussion.) If the problem appears to be medical, the appropriate physician referral is indicated.
2. Skill proficiency. The remedy could involve training, equipment modifications, procedural adjustments, or reassignment to another position. Individual training is the most common adjustment, since skills degrade rapidly with slower companies and/or reduced training hours.
3. Behavioral. An employee assistance program (EAP) may be the most appropriate referral for behavioral issues. However, disciplinary action may also be appropriate if a member simply refuses to perform.
The company standards serve two purposes when substituted for the CTTest: (1) to assess team and individual proficiency levels and (2) to evaluate individual physical performance. This dual-purpose procedure is a viable and effective solution to performance testing. Combining both assessments conserves training time, meets the criteria for incumbent physical performance and company standards testing, lends support to CTTest validity, and is palatable to both labor and management without compromising standards. n
 |
 |
 |
 |
n JOHN LECUYER is a 19-year veteran of the Aurora (CO) Fire Department and currently serves as the department`s health and safety officer. He has degrees in several fields, including a master`s degree in kinesiology. He is also a member of the NFPA 1500 Occupational Safety and Health Committee.
