BY MICHAEL A. TERPAK
Fire service leaders throughout the country are continually challenging the promotional testing process in an attempt to produce the most qualified candidates to fill their ranks. Because of the increased responsibilities placed on the fire service, test development agencies are being forced to pursue new and innovative methods for evaluating a potential company or chief officer. For many years, the only method used to evaluate a candidate’s ability to become a fire officer consisted of reading a series of books and answering a lengthy written exam. As time, technology, and the demands of the fire officer increased, it became evident that we could no longer base a promotional testing process solely on a person’s ability to comprehend written material; there needed to be more.
Over the past 20 years, oral assessment centers have become one of the most creditable methods for truly evaluating a candidate’s knowledge, skills, and abilities. Formatted and delivered in various ways, the assessment center’s ability to efficiently and effectively measure a candidate’s readiness as well as a potential fire officer’s need to be better prepared have been the justifications for its increased use.
So how do potential fire officers prepare for this type of promotional testing? They prepare best by understanding the process and its design.
An assessment center is an examination process designed to simulate situations common to the rank being tested. Each exercise is designed to measure the knowledge, skills, abilities, and personal characteristics of the candidate in a given situation. It is the belief of many within the promotional testing field that the oral assessment center is the most efficient and effective means of testing the candidate’s knowledge. The thought behind this is that it is simply a true evaluation of the candidate’s abilities-specifically, either you know it or you don’t.
As indicated earlier, many testing agencies in the past relied on the multiple-choice method as one of the common and often-used forms of testing. Often perceived as one of the least measurable forms of testing, it is probably the easiest to develop and defend should candidates appeal. With the answer already in front of the candidate, it eliminates the student’s need to prepare and retain information for future use. With a basic knowledge base of the subject area and the student’s ability to perform a “process of elimination,” the student has the increased ability to choose the correct response. When measuring whether a candidate possesses the qualities or capability to be a fire officer, the fire department should expect and require the most efficient and effective form of testing. Assessment centers are designed to meet these challenges.
ORAL ASSESSMENT TYPES
Exercises in an oral assessment can be designed in a structured, evolving, or interactive format.
Structured oral assessments present a situation or an incident with a series of questions a candidate will need to answer in an allotted time frame. The questions are designed to elicit courses of action specific to the situation presented. There is generally little to no interaction between the candidate and the assessor during the exercise, thus the name “structured.” In a structured oral assessment, candidates are required to answer the questions based on the available resources presented in the exercise, their responsibilities within the position/rank, and their own technical knowledge of the subject.
An evolving exercise presents a changing environment within an exercise by introducing specific questions the candidate will need to recognize and answer. The evolving exercise is a more commonly used design, since it reflects challenges similar to those with which a fire officer will be confronted. One of the more dynamic methods used in an evolving exercise is a deteriorating condition designed from a computer-based fire simulation. In an attempt to introduce as much realism as possible, the exercises are designed to present visual challenges to which the candidate must respond. In some of the more advanced computer-based exercises, fire conditions can improve or deteriorate, depending on the candidate’s responses.
In an interactive exercise, candidates may have to exchange thoughts, ideas, and concerns with role players to achieve the desired outcome. In this type of exercise, role players are scripted to respond to the candidate’s comments and actions in an attempt to elicit measurable responses.
In another type of interactive exercise, candidates may have to make an oral presentation describing a program or a training exercise and will be asked questions by a group or role player. The interactive exercise is generally reserved for measuring the candidate’s supervisory and administrative skills.
WHERE DOES IT ALL START?
The design, implementation, and evaluation of an assessment center starts with the gathering of information from the specific areas of responsibility for the rank being tested. For the test development agency, this starts with a job analysis for the rank/position being evaluated. Through a careful and often tedious collection of information specifically designed to identify the roles and responsibilities of the company or chief officer, exercises and measurable courses of action are designed and developed to simulate an actual environment a candidate may face as a fire officer. This is a great place for candidates to start their research. As a candidate, it is important that you understand the areas outlined in a job analysis and the importance of the individual subject areas that may be evaluated. Knowing the specific responsibilities of the position is the intent of a job analysis and where you should begin your research.
Exercise Design and Format
The exercise-or, as it often is referred to, the scenario-is the forum used for eliciting certain candidate behaviors needed to perform the job. A candidate’s performance during the exercise will determine if the candidate possesses the level of knowledge, skills, and abilities to perform the job of a fire officer. Each exercise is designed to measure a number of behaviors. The behaviors or responses of the candidate are classified into categories referred to as “performance dimensions.” Dimensions are the tools assessors use to evaluate your performance.
Individual exercises vary according to the assessment center design and the rank being tested. As an example, some of the commonly used situations or exercises often focus around the responsibilities of the company and chief officer responding to and operating at an emergency incident. In others, supervision principles and practices of a fire officer in a specific situation are assessed. In still yet another, administrative responsibilities associated with officer’s rank are evaluated, depending on the assignment and the task.
How Performance Is Measured and Evaluated
Having an understanding of what is being measured and evaluated by the assessors is the next step in your preparation efforts for an oral exam. The anxiety of being in a testing environment that requires you to present your thoughts, concerns, and actions verbally is enough to unnerve even the most well-read student. Unlike a multiple-choice exam where the correct answer is in front of the test taker, the oral assessment requires that you verbally demonstrate how you would handle a given situation or exercise. For the vast majority of people, public speaking or speaking in front of a group is literally terrifying, to say nothing of the fact that your career is riding on your presentation. So how do you attempt to tackle this personal challenge? There are a number of ways; one is understanding what is being measured and how.
ORAL ASSESSMENT PERFORMANCE DIMENSIONS
Listed below are the performance dimensions that can be measured during an oral presentation. It is critical that you understand the meaning of these dimensions and not attempt to memorize them verbatim. What is important is that you understand the behaviors associated with dimensions and that you are able to exhibit these behaviors during the assessment center exercises.
• Technical knowledge: In the technical knowledge dimension, assessors measure if the candidate’s responses represent an efficient, effective, and safe course of action to the challenges presented. Undoubtedly, to do well with the technical knowledge questions, you need to be well-versed in the tested subject areas. As many have said before, “Knowledge is power.” As simple in thought as this quote may seem, it is an individual and essential trait that must develop from two resources: the candidate’s education and understanding of the subject matter, and practical experience with the material. For many, this will come in the form of one’s education through referencing and reading. Reading, studying, and reviewing documented principles and practices provide the candidate with a knowledge base to which he can refer.
Personally speaking, experience is an extremely valuable resource that I continue to foster and develop, but relying on your experiences alone is not enough when preparing for a promotional exam. As valuable as being “street smart” is on the fireground, street smarts must be supplemented with academics. It is common practice for test developers to give the candidate a list of potential resources to study for the exam. Resources often will include specific textbooks, standard operating guidelines, training bulletins, and departmental policies and procedures, to name the most common. What should be gathered from this is that without a significant knowledge base of the tested subject matter, you will score poorly in this area.
• Safety: This is the extent to which you address any safety concerns related to your firefighters, incident scene occupants, other emergency service personnel, and the general public. Safety is always an overriding concern in all strategic and tactical exercises.
• Incident scene management: This is the extent to which the candidate shows the ability to assign, manage, and account for all individuals operating within or near the emergency incident. If any freelancing is detected anytime throughout the presentation, the score will be adversely affected. You must be prepared to identify resource allocation and assignment through the common use of Divisions, Sectors, and Groups.
• Delegation: This is the extent to which you assign tasks to subordinates to effectively; efficiently; and, depending on the exercise, safely accomplish individual and organizational goals.
• Problem-solving/decision-making skills: Within this scoring dimension, assessors measure the candidate’s ability to exhibit problem-solving and decision-making skills within their scope of responsibility as a company or chief officer. The one thing you will notice in this performance dimension, as well as in a few others, is the multitude of behaviors that can be measured, which will depend on the exercise. Measurable behaviors within this dimension can include
- The candidate’s ability to recognize problems or concerns within the scope of responsibility.
- The candidate’s ability to identify the source of problems or concerns.
- The candidate’s ability to develop logical solutions for eliminating and dealing with problems and concerns.
- The candidate’s ability to take organized action within the scope of responsibility.
• Supervisory skills: Within this scoring dimension, assessors measure the candidate’s ability to take command of a situation or assignment to achieve departmental goals. Depending on the exercise, this area can also be measured from a number of different responsibilities. Measurable behaviors within this dimension can include
- The candidate’s ability to identify tasks/assignments.
- The candidate’s ability to delegate responsibilities and assignments.
- The candidate’s ability to provide structure and guidance for carrying out those responsibilities.
- The candidate’s ability to follow up to ensure completion of the task/assignment.
- The candidate’s ability to recognize acceptable and unacceptable behavior and performance.
- The candidate’s ability to interview, counsel, and motivate subordinates.
- The candidate’s ability to take disciplinary actions within departmental guidelines.
• Administrative skills: Within this scoring dimension, assessors will be measuring the candidate’s ability to effectively plan, organize, implement, manage, and evaluate. This measurable skill is often reserved for candidates seeking a higher rank. Job analysis often shows that these skills fall within the responsibility of a chief officer. This is not to say that candidates studying for the rank of captain will not be assessed in this area. Because of rank structure and the responsibilities of those ranks within a given fire department, many departments around the country will require that a company commander possess significant administrative responsibilities to perform the job. Again, this goes back to knowing the test subject areas for the rank being tested.
• Attention to detail and directions: This is the extent to which your oral presentation provides an adequate amount of information concerning the topic/subject area being presented. Your responses and objectives must be specific to the subject and its questions and must follow outlined rules, regulations, and guidelines. This dimension can be measured in any exercise.
• Standards and initiative: This is the extent to which you possess an internal drive to excel or achieve personal and professional goals without being ordered, coerced, or motivated by others. When an assessment center exercise is designed to measure this particular dimension, I firmly believe it starts to paint a vivid picture of the candidate’s qualities and capabilities.
• Interpersonal skills: This is the extent to which you are able to relate with a wide variety of individuals in a positive and effective manner. We find this dimension most notably measured within a supervisory or an administrative exercise.
• Adaptability: This is the extent to which you are able to effectively adapt to new situations with which you are confronted.
• Aggressiveness: This is the extent to which you come across as overbearing or abrasive to a role player or assessor asking the questions.
• Attitude toward subordinates, superiors, and the public: This is the extent to which your manner toward any of the above positively or negatively affects your ability to effectively accomplish individual or organizational goals.
• Political sensitivity: Since the fire department is an agency that deals with the public on a daily basis, it should be obvious that we need to possess this quality. In this dimension, assessors determine the extent to which you are aware of the political ramifications of the content of your presentation. Specifically, does it negatively affect anyone?
• Creativity, energy, and enthusiasm: This is the extent to which your presentation is stimulating, positive, and motivating. This is an excellent and necessary quality to possess for every exercise. What is important to remember here is that you will do poorly within this dimension if you know little about the roles and responsibilities of the position being tested. Having a strong knowledge base of the material/subject matter being presented is the key to performing well within this dimension.
• Composure: This is the degree to which you appear calm and under control during your presentation. This dimension can be measured in a number of exercises. Examples may include your reaction and response to questions from the assessor or role player; time pressure situations in a fire, collapse, or hazardous materials scenario; or other pressured stimuli displayed during an oral exercise. It is a necessary quality of a fire officer; be prepared for it!
• Decisiveness: Just as composure is a necessary quality of a fire officer, so, too, is your ability to initiate action. Decisiveness is the extent to which you are able to initiate action when action is necessary.
• Organizing and planning: This is the extent to which your oral presentation is well thought out and delivered in an organized manner. This dimension is often reserved for a formal presentation that may involve a simulated training exercise, the possible development and presentation of a program or a procedure, or the delivery of a speech to a civic group on a specific subject.
• Oral communications skills: The ability to clearly and persuasively present information to a group or an individual can be a difficult area to master. We often see the most difficulty with candidates studying for their first promotional exam, because the firefighter position and the responsibilities associated with that position do not require many of the skills outlined in the oral communications dimension. They need to be developed for the company officer position. This lack of experience in oral communication, accompanied by the anxiety of taking their first promotional exam, will often interfere with the candidates’ ability to perform well in this area.
Regardless of the rank being evaluated, the candidate’s ability to organize and express thoughts, ideas, and concerns is a measurable skill and a requirement for a company and chief officer. If you can express yourself well during the exercise, you will not only score well within the oral communications dimension but also make it easier for evaluators to determine whether you have the knowledge, skills, and abilities to carry out the responsibilities the position requires. This is a critical component of your promotional exam.
• Time management: Using your time wisely and efficiently is a measurable skill within an assessment center exercise. Your ability to organize and present your thoughts/responses in the allotted time frame of the exercise is a skill that will be cross-referenced and scored within your planning and oral communication dimensions.
To give you an example of what is meant by this, typically you are given a varied period of time to prepare your presentation, based on the design of the assessment center and the exercises for which you are preparing. In the case of a fire, structural collapse, and hazardous material scenario, the candidate is given a minimal amount of time to prepare, generally five to 10 minutes. The objective behind this approach is to force the candidate to act/respond quickly to the scenario presented, in an attempt to simulate the rapid decision making required in an actual emergency.
For the administrative and supervisory assessment exercises, candidates are typically given more preparation time, approximately 30 to 45 minutes. The added preparation time is given to simulate the additional time a fire officer would have in preparing for an administrative or supervisory incident.
Presentation times for the exercises described above generally range from 10 to 30 minutes, depending on design, dimensions measured, and the level/rank being tested. What is critical to note here is how the candidate manages and organizes time.
• Using performance dimensions to rate your presentation: Each exercise is capable of measuring a number of individual performance dimensions. The design and objective of an assessment center allow assessors an opportunity to measure a variety of job- and rank-related performance dimensions. The number and type of dimensions to be measured are based on the assessment design, the individual exercise, and the position’s responsibilities. Don’t forget, it all starts with the job analysis.
On average, most assessment centers use a rating scale that assigns a numerical reference to each behavior. Rating scales vary, but the most common we have found is a simple rating of 1 to 5 or 1 to 7, with the number 1 the lowest score. Additionally, assessors use a check-off sheet to indicate if you exhibited behaviors specific to the exercise and its dimension. Some of the behavioral quotes are generic to the dimension being evaluated; others contain required behaviors based on the individual exercise. As an added note, in many assessment designs, assessor scoring sheets also allow the evaluator to make written comments about the candidate’s performance, in an attempt to eliminate any potential subjectivity in the scoring process as well as to add clarity to individual scoring. (See the sample scoring sheet on this page.)
 |
At this point, it should be obvious that oral assessments require a significant amount of preparation time. You will have to understand the responsibilities for the rank being tested and the potential exercises with which you could be involved, how an assessment team will measure your performance, and how to prepare for the oral assessment exam. All this requires dedication, time, and direction. Hopefully, the information presented here will help you begin that process.
MICHAEL A. TERPAK has been in the fire service for 30 years, spending the past 25 years with the Jersey City (NJ) Fire Department, where he is a deputy chief in charge of the Training Division. Throughout his career, he has worked in the city’s busier Lafayette and Greenville areas with Engines 10 and 17, Ladder 12, and Rescue 1 and as chief of the 2nd Battalion. He lectures nationally on fire/rescue topics and is the founder of Promotional Prep, a New Jersey-based consulting firm designed to prepare firefighters and fire officers for promotional exams. He has a bachelor’s degree in fire safety administration from New Jersey City University and is the author of Fireground Size-Up (Fire Engineering, 2002).
