FIRE DEPARTMENT PROMOTION TESTS: A NEW DIRECTION
Analysis of the most recent exams given for fire department promotion tests confirms a new but very clear trend. In national education circles, it is called “alternative assessments.” This trend, which has been developing in national education testing for some time, dramatically affects the way a candidate prepares for a promotion exam. Instead of only one form of testing, candidates are tested through several test formats on more than one test component. For example, a test component for skills and abilities has been added to the traditional technical knowledge component. The premise here is that a person’s capacity is proved by the ability to perform or produce, not by the ability to answer on cue. Because of the unusual performance requirements for fire officers, civil service has adopted the alternative-assessment concept for fire department promotion tests. This action was taken primarily to ensure test validity, the federal government’s requirement that fire department promotion tests relate to the job for which candidates are being tested.
Since the position of fire officer requires a good deal of technical knowledge and performance as a supervisor/manager, the alternative-assessment concept for promotional exams is appropriate and probably will remain as part of the testing program.
TECHNICAL KNOWLEDGE FORMATS
Civil service has taken the concept of alternative assessments one step further, however. Not only have tests been developed for both the technical knowledge and skills-and-abilities components, but the standard multiple-choice question test now is supplemented with alternative test formats as well. Standardized multiple-choice tests—which require that the candidate select one correct answer from among four choices—may be fairest in equitable testing, but they also have a flagrant weakness: They do not show whether the candidate can actually apply what he has learned. When it comes to testing for the level of knowledge the candidate possesses, however, the multiple-choice question fares as well as other types of testing modes. The multiple-choice question test has always been popular with students and teachers. Students and test makers generally prefer multiple-choice tests because they are easier to prepare, to prepare for, to take, and to score. The multiple-choice question test has been the traditional form of promotion testing for many large city fire departments. It probably will be used to some degree or another for some time to come. Candidates, however, can expect that future tests will be composed of several testing formats.
CIVIL SERVICE RATIONALE
For fire department promotion tests, the multiple-choice testing format, which rates general knowledge well, never seemed to measure job performance in any convincing way. Given the nature of firefighting, candidates for fire department promotional ranks must possess qualities that are difficult to test using traditional testing methods. In fact, there appears to be little connection between the traditional multiplechoice questions and the real-life situations that may be found on the fireground, in the field, or in the fire station. Multiple-choice questions just do not accurately reflect the conditions or circumstances candidates will have to face when promoted to the new rank. This is the reason civil service gradually has upgraded these promotion tests. They are more related to performance and less related to establishing knowledge levels, a conceptual change directed away from testing only technical knowledge to testing both technical knowledge and ability. According to a recent U.S. Department of Education study, the move to alternative assessments places greater emphasis on the ability to solve complex problems through critical thinking and communications skills than on the ability to memorize decontextualized facts. For testing technical knowledge, questions may be of the kind in which the candidate selects the answer or of the constructed-response type in which the candidate constructs the answer. For performance-based testing, the ideal, of course, would be to see and hear the candidate perform in a simulated job situation. This kind of test has been developed, but the multiple-choice question is being used as well.
TECHNICAL KNOWLEDGE: SELECTION-TYPE QUESTIONS
To make up for claimed deficiencies found in the traditional multiple-choice testing format, new forms of multiple-choice questions are being used to supplement the traditional questions. Although it is unlikely that the traditional multiple-choice question will disappear from the testing scene, it probably will be used less frequently as the only question format. Civil service is moving toward multiple-choice questions that rely on visual representations to test technical knowledge. It used to be that the traditional multiplechoice test would include an occasional visual-type question. On a recent test for company officer, however, as many as half the questions were of this type. Visual representations can better depict circumstances related to the functions of the rank to which the candidate is aspiring than a question based strictly on the text can. If the visuals accurately simulate an event, the candidate often feels close to the situation and is more likely to propose a real-life solution. It may be too soon to say whether this question form is here to stay. The simulation-question format has yet to stand up to test-question litigation students usually bring after promotion tests.
TECHNICAL KNOWLEDGE: MULTIPLE ANSWERS
Civil service uses another new form of multiple-choice question, one that has more than one correct answer. As many as 15 answer choices may accompany a question. The candidate must choose all correct choices. This question format is very difficult and favors the candidate who has thoroughly prepared for the exam. The candidate who ordinarily might guess at an answer choice would be at a disadvantage in this testing situation. The probability that the candidate will eliminate all the incorrect choices by guessing is just about nonexistent. In addition, a penalty is assessed for all incorrect choices. Generally, not all the questions would be of the multiplechoice type. Time management is an important factor when answering this type of question. Candidates need a good working strategy to be successful with this test format.
For More Facts Circle 122 on Reply Card
TECHNICAL KNOWLEDGE: ALTERNATE-RESPONSE QUESTIONS
The “true-or-false” or “agree-or-disagree” question was used recently to complement multiple-choice questions on an examination for the position of lieutenant in the City of New York (NY) Fire Department. When used with a single isolated fact or statement, the alternate-response question is not as difficult as other forms of questions. When the true-or-false question is used with a visual representation, however, a great deal of graphic detail is often included. The candidate usually has to figure out if every detail in the visual is represented correctly. The candidate then responds to the question. A substantially detailed visual takes time to examine. When used with a visual representation, the alternate-response question can he difficult to answer correctly and can considerably tax the available test time. It has not been used extensively on civil service tests and represented only about 10 percent of the questions on the FDNY lieutenant’s test. This question form favors the candidate who is thoroughly prepared for the test.
TECHNICAL KNOWLEDGE: THE IN-BASKET MULTIPLE-CHOICE TEST
The in-basket test has been used several limes as part of a technical knowledge test component. The candidate is given a package of documents that are representations of reports, memos, orders, etc. pertinent to the administrative functions of the job for which the test is given. The candidate reviews the documents, puts himself into the scenario or situation described in the exercise, and then answers the questions. The answers will depend on the candidate’s knowledge of the situations referred to in the documents. Paper or document management and time management can be critical factors in dealing with these types of questions. The candidate must have a preplanned method for attacking the questions.
TECHNICAL KNOWLEDGE: REFERENCE TEST
A reference-test format frequently is used to test knowledge. As in the case of the in-basket format, the candidate is given a booklet of materials containing complete procedural circulars, bulletins, etc., or just extracts from these or other administrative materials. The candidate usually must answer multiplechoice questions, the answers to which can be found in the distributed reference materials. A candidate who has prepared for the test usually needs only to verify the answers; the unprepared candidate probably would run out of time trying to find the right answers. Regardless of what the candidate knows, a strategy for answering these questions must be carefully developed and carried out on the day of the test.
TECHNICAL KNOWLEDGE: CONSTRUCTED RESPONSE QUESTIONS
Constructed-response questions require that candidates produce an answer to a question rather than select front an array of possible answers (as is done in multiple-choice items). In constructed-response items, questions may have just one correct answer or they may be more open-ended, allowing for a range of responses. The form may vary and might include supplying answers by filling in a blank; solving a hydraulics problem; writing short essays; or writing out all the steps in a fire control or emergency problem. The traditional technical knowledge essay question is a constructed-response-type question. This form of question has been used widely on chief officer promotion examinations. Traditionally, the essay question has been a misnomer. The word “essay” was unsuitably applied to a question that more correctly should have been called a list question. Generally, the question asked the candidate to list certain facts or items of knowledge pertinent to the promotion rank sought. The sources used for the essay question could be technical-, management-, or supervision-related. For these tests, time management plays a very important part in the candidate’s testmanagement strategy. An essay format sometimes can be used with a writing sample test and is a performance-based test, which will be discussed later.
Still another form of constructed-response question used to test specific knowledge is the fire problem essay, a format that has been used mainly on chief officer promotion tests. A fire or emergency is described in the question and the candidate is given instructions about what he is expected to do. Preparing for this questioning format often requires that the candidate develop a “canned answer”-—one that will fit almost any fire situation. Since candidates must expend a considerable amount of practice and preparation to be able to answer this kind of question, many fire professionals believe this question form is a fair gauge of the candidate’s ability to perform as a command chief on the fireground. Civil service likes the constructed-response question; the difficulty with it is the litigation involving question formats that followed the tests on which it had been used.
SKILLS AND ABILITIES TESTING
Ability tests are shaped so that a degree of performance ability can be accurately measured. Performance assessment is a broad category of testing methods that requires candidates not only to select answers but also to be able to create answers that demonstrate practical knowledge. One way to simulate actual conditions encountered by persons in the rank is first developed. The candidate then is required to perform a task that requires a certain skill and is assessed on how effectively the skill is performed or the candidate’s understanding of the skill is communicated to the test assessors. Performance-based assessment in conjunction with the standard multiple-choice question test is becoming the new format for testing candidates for fire department promotions. Recent FDNY promotion examinations for captain, battalion chief, deputy chief, and chief of department contained performance-based assessment components. One type of performance-assessment test incorporates interviewing as a “constructed-response” format. Students produce their own answers rather than select from a set of choices. This test format attempts to assess a candidate’s ability to use knowledge intelligently. The interview gets them to use their knowledge to produce or perform.
ABILITY TESTING: JOB ANALYSIS
For the new assessment methods to be valid for fire department promotion examinations, a process for determining which skills need to be tested must be developed. The most common way to identify the skills would be through a carefully assembled and implemented job analysis. The analysis would begin with interviews with administrators, superior officers, and officers of the same rank for which the test is given. A questionnaire based on the interviews then would be developed and distributed to about 20 to 30 percent of the total number currently in the rank for which the analysis is being prepared. This percentage is necessary to provide validity. The questionnaire would list knowledge items, skills, and abilities relevant to the tasks and duties of the rank. Participants would rate the relevance of each of the items listed on the questionnaire. After scoring the ratings given on the questionnaires, civil-service would establish a final list of knowledge items, skills, and abilities to be tested and publish it in an examination notice.
ABILITY TESTING: INTERACTIVE INTERVIEW
Using assessment-center methodologies, civil service has employed assessment-center-type exercises for performance-based tests. Performance assessment is a broad term. It covers many types of testing methods that require candidates to display their competencies or knowledge by creating or constructing an answer. The most notable of the tests is the interactive interview, which requires the candidate to interact with a role player. A scenario that at best is described as fluid is played out by the candidate and the role player. Depending on the candidates’ responses and questions, the role player adds information to the scenario to which the candidate also must respond. Through this verbal exchange between role player and candidate, the candidate shows his level of skill in each ability being tested. In effect, the candidate constructs answers or solutions to the situation described in the exercise. The interview may be witnessed by assessors or videotaped for later review and marking by assessors. Part of the candidate’s success with this form of test is knowing how the interactive interview test format works and what is being looked for
ABILITY TESTING: THE WRITING SAMPLE TEST
A writing sample test is another type of constructed-response exercise. The candidate is required to write clear, complete. concise, relevant, and well-organized summaries concerning descriptions of incidents, unusual occurrences, etc. The essays, memos, or reports produced must often express the candidate’s viewpoint, analysis, thoughts, or recommendations. Writing skills, including those of writing basic reports, as well as knowledge of the tested abilities are essential to success with this answer format. Writing sample tests measure a candidate’s knowledge of facts, concepts, and procedures; ability to solve problems; and ability to comprehend text. They also record how candidates compose and critique various kinds of texts and reports and evaluate performance. The writing sample test measures the extent to which the candidate has grasped not only the obvious information but also the relationships, subtleties, and implications of the topic. Essays used in writing sample tests are the most common form of performance assessment. They assess content mastery as well as performance. The candidate must construct responses that demonstrate comprehension. A candidate’s understanding of a subject has long been assessed through a written essay that includes the facts in context. Answering essay questions effectively requires analysis, synthesis, and critical thinking. The writing sample test can be used to test for a broad range of skills and abilities.
ABILITY TESTING: THE INCIDENT MANAGEMENT TEST
The incident-management exercise tests the candidate’s skill in dealing with or managing incidents that occur in the make-believe world civil service creates by using documents, related materials, and a descriptive scenario. Many times, a lengthy document is included with the related materials. It might describe a procedure, an evolution, or a particular program. Other documents in the package include work charts, personnel profiles, calendars, and so on. The candidate must resolve the situations described in the multiple-choice questions while relying only on the materials distributed as part of the exercise. The candidate draws from his experience and knowledge of the specific skills being tested in each question. Since the candidate need not construct an answer, this form of test is considered an answerselection type and usually is not as difficult as other test forms.
A NEW TEST EXPERIENCE
For most candidates, an assessment exercise is a new experience. The candidate’s unfamiliarity with this form of test creates a good deal of anxiety. To overcome this anxiety, the candidate must practice these test forms before the actual test and must try to understand the requirements of the promotion rank for which the test is given, including knowing which abilities (demonstrated by superiors) are important to successfully performing at the level of the new rank.
On the day of the test, it is very important that candidates read and follow all instructions and that they manage test time effectively. It is also necessary to clearly understand form, any one or several different test formats, and substance, the elements of performance deemed necessary to succeed in the rank.
THE FUTURE
Performance-based testing and technology. During the past decade, civil service has been introducing new test formats that measure not only technical knowledge but also performance. The trend toward these alternative assessments will continue. The bias clearly seems to be toward performance-based testing. One of the most obvious changes in the future will be assessing performance through the use of audio or videotaping technologies. The candidate will be videotaped while responding to open-ended questions developed to allow the candidate to show an understanding of the skills being tested, enabling an assessor to rate the interview fairly and more equitably. As video and audio make it possible to record performance, the use of oral-test formats is likely to expand. It also is probable that we will see the use of interactive video and computer programs cued into the civil service testing process as well. Technical knowledge will not be enough.
For More Facts Circle 122 on Reply Card
THE CHALLENGE FOR CANDIDATES
The new challenge for test takers is twofold: to become more aware of the abilities and skills needed to perform the duties of the rank being sought and to understand and master the variety of test formats being used by civil service. The candidate’s preparation must encompass a broad variety of study topics. Study groups and coaching schools can help considerably. Many coaching schools, such as New York City’s “Fire Tech,” already have begun to include these new ideas in their curriculums. In addition, the candidate must understand the test’s structure and objectives and practice working within the format of each test mode.
