Written Fireground Simulation— New Tool for Teaching and Testing

Written Fireground Simulation— New Tool for Teaching and Testing

features

Think back to the last fire service test you took. Did it ask questions relevant to your performance as a fire fighter or officer? Were multiple-choice and fill-in questions used when what was really expected of you was to make decisions or solve problems? Maybe it asked you to write an essay on “what you would do if . . .” If these situations sound familiar, then there is a new idea for fire service teaching and testing that you should know about entitled written fireground simulation (WFS).

There are few professions today in which the margin for error in decision-making and problem-solving is so small. Both officers and fire fighters alike must make decisions daily which may save or lose both lives and property. Individuals must be able to combine knowledge about fireground strategy and tactics with an ability to make critical decisions at the right time. Fireground situations can change rapidly, demanding that strategies change to meet these situations.

Training officers ask, “How can we best teach these kinds of skills? How can we determine if fire fighters and officers have learned these skills so that they can apply them?” The individual fire fighter or officer, on the other hand, looks for relevance and direct applicability in his training. Written fireground simulation may provide some answers to these demands and dilemmas.

Realistic problems

These simulations are a written exercise which present the participant with a realistic problem to be solved. The situation requires that the participant display decision-making and problem-solving skills in attempting to resolve the problem via a branching logic format. While written simulations cannot duplicate the actual fireground situation, the process of responding to options presented in the exercise closely approximates the process of thinking that fire fighters and officers must go through in making decisions about fireground operations. The requirement that a test “closely approximate an observable work behavior” is found in Equal Employment Opportunity Commission “Uniform Guideline” 14 c(4).

Simulations represent some aspects of reality in fireground situations which can provide methods for understanding, manipulating and predicting behavior in complex fireground situations. The information that WFS can provide about an officer’s or a fire fighter’s performance may be either qualitative or quantitative, i.e., WFS may be used to improve skills and to facilitate learning or to evaluating competence. Simulations are lifelike in that the problem evolves and the situation changes in response to specific decisions made by the participant. A “real-life” situation unfolds, based upon the consequences of the type of information gathered and the appropriateness of decisions made in response to that information.

To begin a WFS, a short scenario de scribing the situation is presented to set the stage for the problem. It is accompanied by a brief description revealing only those aspects of the situation that the participant would have available to him from a dispatcher or upon arrival on the fireground.

Two types of sections

Each exercise consists of two types of sections, arranged in scrambled order, corresponding to various stages in the resolution of a fireground problem. One type of section is called “information gathering,” the other, “decision-making.” Each exercise contains several of both types of sections, lettered alphabetically.

Participants are directed to proceed through the exercise by uncovering pertinent information by using what is termed the latent image technique. This technique involves developing designated spaces on the answer sheet by using a special latent image developer pen similar to a magic marker. The participant would indicate his decisions by rubbing the marker on the empty space adjacent to the number corresponding to the item chosen. Printed material instantly appears, which provides additional information that may be useful in making subsequent decisions or may direct the participant to the next appropriate section of the exercise.

As in reality, the participant cannot retract decisions once they are acted upon, even if they are revealed to be ineffective or harmful. Again consistent with reality, he can often (but not always) modify the consequences of an improper decision, but it cannot be ignored, especially on the fireground.

Optional routes

Each exercise has more than one possible decision-making route or “map” through which the participant may resolve the situation presented. This unique feature allows for individual choice in approaches and enables participants who may make inappropriate decisions to recover by selecting more appropriate responses later in the problem. Participants who neglect indispensable measures or who choose potentially damaging or life-threatening procedures may be directed to a section in which they make take steps to rectify the previous error.

Few fire service veterans would deny that a competent fire fighter or officer must be able to deal effectively with fireground problems and to make sound decisions. If we agree that practice makes perfect, then WFS can afford fire fighters and officers the opportunity to perfect decision-making in a realistic, systematic fashion. McGuire and others have written that to cope effectively with a problem, individuals must be able to:

1. Use a variety of resources.
2. Order priorities of data-seeking and decision-making.
3. Manipulate the situation to alter it.
4. Monitor the effects of decisions and manipulations.
5. Readjust decisions.^2,3

Written simulations are designed to sample a broad range of competencies, not simply the recall of isolated facts. A WFS exercise requires the participant to apply general principles to the solution of a particular problem.

Purpose of simulations

The purpose of written fireground simulations is to facilitate teaching or assessment of a participant’s ability to perform the following kinds of tasks:

1. To gather appropriate data in a timely fashion.
2. To interpret and/or synthesize
3. data to establish criteria for decisionmaking.
4. To solve common fireground problems with special emphasis upon identification and analysis of a problem and taking appropriate steps toward its resolution.
5. To make judgments and/or decisions relative to appropriate management of situations with regard for timing and ultimate effects.

Generalized simulation techniques have been used with varying degrees of success in the fire service for some time. Written situations, audiovisual simulations using hypothetical situations, mechanical simulators, scaled mockups, and time-consuming oral review boards are all variations which have been used. The use of latent image written simulations may prove to be a successful adjunct to existing methodologies.

Construction of WFS

Simulations are generally patterned after three major designs. The first is known as “free-branching,” which is the most complex, but it provides the participant complete freedom of choice in choosing options. The second and third designs are known as “forced-choice” and “linear,” which provides progressively less choice in options. Branching is essential if high-level skills of judgment and problem-solving are to be taught or assessed.

The opening scene is a short paragraph which provides the following information:

1. Participant’s role, e.g. “You are the lieutenant in charge of a first-due truck company in a city of 40,000 population…”
2. Physical setting of the fireground, e.g. “An alarm is received reporting a smell of smoke in a large department store warehouse within the city limits…”
3. Imposed limitations, e.g. “Your normal duty crew is down one member, leaving only yourself and two fire fighters on the truck …”
4. Additional information, e.g. “Water supply is readily available …”
5. Participant’s task, e.g. “Your job is to command the fireground until the arrival of a chief officer.”

The opening scene is meant to supply the participant with just enough information to make the initial set of decisions. Notice that the nature of the information given is what would ordinarily be available through, for instance, a typical dispatcher or communication’s center.

Bridge segments

The subsequent choice of action upon arrival at the scene, whether it be further data gathering or immediate attack, is provided by bridge segments. A typical bridge segment from our opening scene is as follows:

Upon arrival at the scene you would (choose only one):

1. Immediately sound a second alarm.
2. Immediately institute aggressive attack with large exterior hose lines.
3. Institute cautious defensive tactics, including immediate protection of exposures.
4. Quickly try to gather more pertinent information.

Choices 1, 2 and 3 represent bridge segments dealing with direct intervention without data (see fig. 1). Such intervention is likely to result in complications, require reassessment, and require further management of the complications created. Each complication would have to be dealt with in subsequent option sections in which the participant is provided with further information or opportunity for action. Note that choice number 4 (above) leads the participant to a data gathering section in which access is provided to pertinent (and nonpertinent) information required for proper management of the original scenario stated in the beginning of the exercise.

In keeping with the flow of the problem, let us look at a typical “option” or “data gathering” section which would result from choosing option number 4 (above).

Data gathering sections are followed by bridge sections to appropriate or inappropriate management sections until the fireground situation is resolved successfully or further complications develop and result in serious property damage or loss of life.

Validity of technique

The single most important issue regarding the use of any evaluation technique is validity. Does it measure what it is supposed to be measuring? Does the content of such a test or exercise constitute a representative example of the subject matter, the types of situations, the settings, and the types of decisions required by fire fighters or officers? Do scores on such an exercise relate to observations of actual performances of the same skill as a fire fighter or officer? Do scores relate to performance of the individual at some later date? These questions, and others, regarding validity and reliability of written simulations in fire service training await answers. The approach which must be taken to arrive at satisfactory answers has been well documented in studies done in other professions.

Simulations are generally regarded as valid assessment tools when they represent an actual picture of the roles performed on the job. Thus, they are “content valid.” Particularly if simulations are used to test performance, then they must demonstrate content validity. Content validity represents the degree to which the competency areas sampled by simulation represent the knowledge and skills required in actual performance.

For use of written simulation in large-scale testing and evaluation, it would be absolutely necessary for content experts to agree that the knowledge and skills sampled by appropriately designed simulations would approximate those employed in real fireground encounters. Simulations based on the accurate delineation of officers’ or fire fighters’ roles can be expected to exhibit a high degree of content validity.

A more potent indicator of the validity of simulations would be the direct measurement of actual performance as an indicator of the degree to which simulations measure what they were intended to measure. This type of validity is known as “concurrent validity” and would have to be given consideration if simulations were to be used in large testing programs, i.e., for promotion purposes.

A recent case filed in 1980 in the Eighth Circuit Court of Appeals (Firefighters Institute for Racial Equality vs. City of St. Louis) recently struck down the content validity of a fire scene simulation because the responses to slides depicting fire scenes were made in writing. The same case clearly struck down the use of multiple-choice questions as a valid method to be employed in promotional exams. Obviously, validation is an important consideration.

Scoring simulations

All options in a fireground simulation exercise are scored on a scale of +3 to —3, according to the appropriateness of a particular option to the conditions existing in the problem at that time.

The scoring and rationale for each is as follows:

SCORE RATIONALE

+3 Of central importance for effective operations. Omission would result in serious property damage or loss of life.

+2 Strongly facilitative for effective operations.

+ 1 Mildly facilitative for effective operations.

0 Does not contribute to operations, but does not cause harm in terms of significantly greater damage or risk of life; and/or may be a controversial option due to regional differences.

— 1 Mildy detrimental to effective operations.

—2 Seriously detrimental to effective operations.

—3 Gravely damaging to operations in terms of signifiant property damage or probable loss of life.

A helpful way to establish weights for each exercise is for the writer to assign weights to each response and to have five other experts independently assign a weight to each response. Then the group should discuss the weights and reach consensus on the final weight assigned to each response.

In setting passing scores, two methods have been used in other settings. Often a predetermined percentage of people will fail, such as the situation which exists when we curve grades on a bellshaped curve. During oral examinations, often used in the fire service for promotional purposes, examiner’s questions may be based on valid personal experiences, but the judgments made by oral examiners may be inconsistent and capricious, thus resulting in an arbitrary pass level.

Pass mark format

The format for determining the pass mark goes like this. Look at the “choose only one” (decision-making) sections first. Decide, for each section, the lowest acceptable score for a passing performance. In most cases, —1 choices, if they exist, are the lowest acceptable since a —2 cannot be acceptable by definition. After the lowest acceptable choice is established for each of these sections, the path that the examinee could follow by selecting these choices is identified. The minimum score is based on these sections.

The procedure for “select-as-many” (information-gathering) sections is a bit more complex but easy to understand. This format is derived from that used by the National Board for Respiratory Therapy and the Educational Testing Service. First, identify all the required positives (RP), these are the +2s and +3s by definition. They may also include some +ls because while some choices by themselves are only mildly facilitative of effective operations, a group of them may be important in a given situation. Now all of the forgivable negatives (FN) must be identified. These are usually the — 1 options. Then the nonrequired positives (NRP or +1s) are also counted. Add the forgivable negatives (—1s) to the nonrequired positives ( +1s). If there is an excess negative score, subtract it from the sum of the required positive score. The reason for this is that careful or experienced students select only the required positives.

The most common error is to select many more choices than is absolutely necessary. In simpler terms, we start out by assuming that the minimum pass level (MPL) for the “select-as-many” sections is the sum of the required positives. If there is an excess of forgivable negatives (NRP + FN), we reduce the MPL by that amount.

Figure 4 is a model which can be used for reporting scores. Notice that information-gathering and decision-making scores are reported separately. While the absolute percentage cutoff scores for passing marks may be set arbitrarily (or by complex validation procedures), it is generally accepted that in order to pass the exercise a student or candidate must pass both the decisionmaking section and the informationgathering section. Notice that feedback can be provided by simulation problem, by content category and by information-gathering and decision-making process. Thus, these exercises can pinpoint weaknesses with greater accuracy.

Pros and cons

The thoughtful developer of written fireground simulations should be aware of the potential advantages and disadvantages imposed by the characteristics of this methodology. The potential advantages include:

1. A methodology which can be developed locally without significant costs.
2. Participants perceive the problems as relevant to the realities of fireground operations.
3. Objective ratings of performance can be derived.
4. Realistic feedback can be provided immediately in response to judgments and decisions.
5. Participants can be held responsible for their actions without endangering lives and property or incurring large costs in terms of time or finances.
6. The most important competencies as well as a broad range of competencies can be sampled in a short time.
7. Evaluation of competence can be standardized.
8. Can be used as a teaching tool as well as a testing tool.
9. Easily adaptable to other instructional modes, i.e., computer-assisted instruction.
10. The “reality” created appears to have a lasting and potent impact upon participants.

The potential disadvantages include:

1. Can be time-consuming to construct intricate problems.
2. Large-scale testing can be expensive in terms of construction and administration.
3. Recognition rather than generation of choices is required.
4. May provide cues if used inappropriately in test situations.
5. Does not lend itself to standard psychometric analysis, if that is a priority.

Reproduction method

The latent image process can be easily and economically reproduced locally. The A. B. Dick Company markets an easy-to-use system for making latent image written simulations.

Here’s what you need:

1. A simple mimeograph machine or a hand-crank spirit duplicator (around $450).
2. A. B. Dick No. 97-1001-1 latent image transfer sheets, around $13.65 for a package of 10.
3. Latent image developer pens, No. 97-3011, cost $4.50 per dozen (long lasting).
4. Regular spirit masters, 8 1/2 X 11, No. 22-6011, cost about $10.50 per 100.

You just type the exercise (minus the responses) on a spirit master, take out the carbon paper and replace it with a latent image transfer sheet, type in the latent responses, and run it off. Spirit masters are good for about 150 copies.

Experience in other areas, notably in medicine and allied health, has demonstrated that learners find written, latent image simulations to be both interesting to take and relevant to their purposes. Persons in the fire service, when faced with tests or evaluations, are rightfully demanding to be evaluated according to specified criteria that represent the roles they will actually assume in practice.

The public, who represent our clients, trust in all of us to be equal to the demands for public safety. Thus, fire service training officers, whether in volunteer or paid departments, who are aware of all of these concerns and the inherent limitations of the traditional modes of teaching and evaluating performance, should consider the potential impact, of written simulations as a valuable tool. Written fireground simulations have the potential for considerable practical importance in teaching and testing in a profession which depends upon the decision-making and problem-solving skills of its officers and men.

References:

1. Scanlan, C. L.: Patient-Management Problems: Simulations in Clinical Evaluation. Respiratory Care 23,4: 397-406, 1978.
2. McGuire, C. H. and Babbott, D.: Simulation Technique in the Measurement of Problem-Solving Skills. J. Educ. Meas. 4: 1-10,1967.
3. McGuire, C. H. and Solomon, L. M. and Bashook, P. G.: Construction and Use of Written Simulations. New York, The Psychological Corporation, 1976.