BY CARR BOYD AND ROB CANNON
As a general rule, the fire service puts extraordinary effort into developing standards, testing equipment, and improving fireground strategy and tactics. We form committees, determine specifications, and expend considerable time and effort to ensure our equipment meets or exceeds specifications. We have draft tests, drop tests, shaft tests, and stop tests. We test in “low vis” and high heat in the station and on the street. However, few departments take the time to perform the same due diligence on the tests that will play a major role in determining who will be selected for positions within their organizations.
Too often, we “punt” this function to the Human Resources Department, assuming that human resources experts will make sure we get a high-quality test instrument to select individuals for hire or promotion. Smaller fire departments don’t have this luxury and must make up their own test or use an “off-the-shelf” test. Few departments take the time to research or even ask questions about the reliability and validity of the testing instruments used to select candidates. A testing process that is not reliable and valid will provide inconsistent results and could even lead to legal issues under Title VII of the Civil Rights Act, which requires testing instruments to be valid indicators of one’s ability to perform job-related functions.1
Fire executives who rely on faulty instruments to select persons for hire or promotion may select candidates who lack the characteristics necessary to be effective; conversely, they may fail to identify those candidates who are most qualified for the positions. The fire service is organizationally unique in that everyone from rookie firefighters to line officers to chief officers makes decisions on and off the fireground that will determine the success or failure of the department. When the right people are not selected for particular positions, organizations falter. Therefore, it is imperative that departments use the best selection tools available to hire and promote fire service employees.
General Colin Powell once said, “Ultimately, it is peoplenot plans, systems, structures, or budgetswho make the difference between organizational success and organizational failure.”2 Like all strategic leaders, fire service executives must concentrate on finding, developing, and promoting the best people. A fire chief might be an expert in operations; he might know every nut and bolt on the apparatus and every number in the budget. Even with all these qualifications, the fire chief who is unable to select and promote the best people will ultimately become ineffective, as will the department he leads. Conversely, the leader who selects, develops, and promotes the best people fosters the likelihood for current success while also creating a foundation for the future of the department.
RELIABILITY AND VALIDITY
The selection and promotion process in the fire service, for better or worse, has come to rely heavily on test scores. With these test results playing such a major role in selection and promotion, it is essential that the tests are reliable and valid indicators of ability. The concepts of reliability and validity are drawn from the art of scientific research. Researchers strive to ensure measures and instruments used when analyzing issues are both reliable and valid, because otherwise it would not be possible to determine whether findings are accurate. A reliable instrument is one that produces similar findings when used over and over again.3
For example, pretend you are interested in measuring the flow pressure of fire hydrants within your district. There are two methods you can use to determine flow pressure: (1) turn the hydrant on and have each member of the company guess the hydrant pressure, based on experience and appearance of the water flowing; and (2) use a pitot gauge to measure the flow rate. In the first scenario, the hydrant is turned on, and the four members of the company estimate the flow. One person estimates the flow pressure to be 100 psi, one estimates 130 psi, one estimates 90 psi, and one estimates 160 psi. The company officer quickly realizes that this method for determining fire hydrant flow pressure is not reliable. In fact, it can be assumed that if numerous fire companies used this method to estimate flow pressure for the same hydrant, each company would arrive at significantly different values. In the second scenario, a pitot gauge is used to measure flow pressure. Each member of the company takes a turn using the pitot gauge, which repeatedly indicates the flow pressure to be 110 psi. The pitot gauge proves to be a reliable instrument for measuring fire hydrant flow pressure, because the measurement outcome is consistently the same.4
Validity refers to the truthfulness of the findings. In other words, did you measure what you think you measured? Taking the example used above, the pitot gauge proved to be a reliable instrument for measuring flow pressures, but is it valid? For the pitot gauge to be valid, it must measure the actual flow pressure of the hydrant. If the gauge is reading high or low, then the measurements will not be valid, because the actual flow pressure of each hydrant is not being measured. If the pitot gauge is not accurate, it can still consistently provide the same readings; but if it is not indicating the actual pressure of the hydrant, it is not a valid instrument. If the hydrant has a flow pressure of 140 psi, then the pitot gauge must indicate 140 psi to be valid.
Let’s relate this back to fire department selection and promotion exams, where the goal is to predict those persons who will make the best firefighters and officers. Just as the pitot gauge was the instrument to measure flow pressure, the instrument to determine selection or promotion is often a written test, an oral board, or a combination of these methods. It is assumed that each organization has developed a clear description of the traits most desired in its firefighters and officers. A testing process is then used to take these indicators and predict who will be the best persons to hire or promote. Easy enough, right? Perhaps not.
How often do you hear questions such as how did so and so get promoted? Or, maybe it’s unclear why certain individuals who appear to be highly motivated to become firefighters or officers are unable to achieve high enough test scores even after years of trying, or difficult to explain why those who scored well on previous exams are unable to score as well this year. All of these are potential signs that a selection process has some reliability and validity issues. Using exams that are not reliable and valid does a disservice to the members of the department and to the public, because those most qualified may not be selected. This can also harm morale within the organization, create legal issues, adversely affect retention rates, and have an overall negative impact on departmental operations on and off the emergency scene.
Developing a reliable and valid testing instrument is challenging and requires a great deal of work and analysis. Some may scoff at this notion, but it has been found that effective executives spend as much as 75 percent of their time on recruitment, retention, and other “people”-related issues. (2) Remember, “Organization does not accomplish anything. Plans don’t accomplish anything, either. Theories of management don’t much matter. Endeavors succeed or fail because of the people involved.” (2, 125) Whether operating a hoseline during an interior attack, managing a fire station or battalion, or commanding a complex emergency incident, it is the people who will ultimately determine operational success or failure (photo 1). The time a chief officer devotes to identifying individuals who will best serve the fire service organization is time well spent.
null
Reliability and Exams
Let’s focus on the reliability of an exam. Remember, for an instrument to be reliable, it must provide similar outcomes each time it is used, which explains why the pitot gauge is a more reliable instrument for measuring flow pressure than having each member guess the pressure. A candidate taking an entrance or promotional exam that is reliable should receive a similar score if taking the exam repeatedly. A chief officer once complained to me that the scores on a written portion of a promotional exam were too high. He went on to state that in his opinion the high score on such an exam should not exceed 80 percent. This individual had little concern for reliability. In effect, an exam on which no person is capable of determining the correct answer for more than 80 percent of the questions is doing a better job of measuring the candidates’ ability to guess than it is testing actual knowledge of a subject. With any exercise that involves guessing, the outcomes are unreliable, as they were when the members of the fire company attempted to guess the flow pressure out of the hydrant. The often heard retort to this is that a difficult promotional exam is necessary, because we want to promote only the “best.” Well, if the “best” cannot score above 80 percent, it is unlikely that the exam is accurately predicting who is actually the “best” within your department. Hiring and promotional exams should be difficult, but they must also be doable for those who have prepared and have the qualifications to perform the job.
Validity is an even more pressing issue when using an instrument to predict the future success of applicants. A question that must be asked is, “Is the exam process identifying candidates who are best prepared to fulfill the job?” Exams that are not valid can mislead fire service executives into selecting individuals who cannot do the job or pass over those who could excel in the job. Either situation harms the organization. “At the end of the day, people are why the mission (and eventually the enterprise itself) rises or falls.” (2, 125) The first step in creating a valid instrument is to determine the characteristics that are most desired in your employees and officers. This is a difficult task that requires an honest and thorough analysis of the job and the traits held by members of your organization or others who excel at these jobs.
Developing testing instruments that can be used to effectively predict future success is far from obvious. The book Moneyball by Michael Lewis5 provides a poignant example from Major League baseball that emphasizes the importance of using valid instruments to predict future success of baseball prospects. The book focuses on the Oakland Athletics baseball team, which is led by a young executive named Billy Beane. He has the unenviable task of running a low-budget Major League club that often ranks in the bottom four or five clubs in player salary. How does a team that spends roughly $40 million on players compete with teams that are willing to spend four or five times this amount? The short answer is they develop an innovative, reliable, and valid instrument that affords them the ability to accurately predict which prospects would excel in the big leagues.
Beane, a fiercely competitive former Major League baseball player, focused his attention on how player prospects were selected during the draft. Major League baseball teams spend a fortune on scouts and other personnel to evaluate the potential talent of high school and college baseball players. Traditionally, the scouts focused on high school players who were athletes, hit for power, and “projected.” In other words, the scouts attempted to predict future success based on a player’s body makeup, which was hardly a reliable or valid measure when assessing a person who had yet to graduate from high school. However, this was the traditional method for evaluating talent used throughout the big leagues, especially by teams that could afford to pay large salaries.
Beane’s limited budget did not afford him the ability to be wrong too often. He hired a young Harvard University graduate who majored in economics to help him determine the characteristics most synonymous with players who are successful in the big leagues. Through careful analysis, Beane developed innovative measures to select young prospects. No longer would Beane select players based on athleticism. Nor would he select players who were still in high school. Instead, Beane focused on college players who had meaningful statistics. He selected players who earned walks and were able to get on base. He paid less attention to “power hitting” or the other traditional characteristics scouts use to select players. In the end, Beane routinely selected players from college that other ball clubs often never even scouted. Yet, the Oakland Athletics, using Beane’s reliable and valid instrument for selecting players, began consistently competing for the pennant against teams that spent significantly more on player salaries.
The lesson from Beane is that, to be successful, you must have an instrument that tests for the actual characteristics that are important to being successful in a particular job. Beane proved that being athletic or hitting for power is not necessarily the defining factor of a successful Major League baseball player. This is no different from the fire chief determined to promote individuals who can handle the awesome responsibility of leading firefighters on and off the fireground (photo 2). If the hiring or promotional process does not distinguish between candidates who have the necessary characteristics and those who do not have them, the process is not valid.
Why is this important to the discussion about selecting and promoting firefighters and officers? A hiring or promotional exam is an instrument, just like the methods developed by Beane to select baseball prospects and used to predict future success. Anytime we attempt to predict the future, there is a chance that we will be wrong. However, there are ways to create reliable and valid instruments that can significantly reduce the chances of mistakenly hiring or promoting someone who will not be able to handle the job. Each fire department has its own version of a promotional process to select officers, but are these processes effectively evaluating the characteristics that make up a productive fire officer?
A mistake often made when setting up promotional exams is to eliminate potential candidates based on criteria that are neither reliable nor valid. For example, some departments require a two- or four-year college degree to be eligible for promotion. In these cases, a person without the degree is not even granted the opportunity to compete. We should refrain from automatically eliminating persons based on criteria that may or may not be reliable and valid indicators of future success. Instead, an effective promotional process should test for the characteristics one would expect a college graduate to possess and that are important to being an effective officer or firefighter. Is it the piece of paper you are interested in or the characteristics associated with persons who are seeking degrees? Good organizations are focused on promoting persons with the characteristics that best predict future success, not those who have simply checked off a group of requirements that may or may not have anything to do with becoming a good officer.
Developing reliable and valid processes requires innovation that may not follow traditional methods. For example, National Football League (NFL) teams rely heavily on the Wonderlic test to measure prospective quarterbacks’ intelligence and decision-making abilities under pressure. The Wonderlic exam is a timed written test. This test has proven to be a reliable and valid predictor of future success for the quarterback position. Higher scores earned on the Wonderlic, with few exceptions, are correlated with future success in the NFL. One caution: The Wonderlic is obviously not the only evaluation required to accurately predict the future success of a quarterback. However, it has been shown that quarterbacks who struggle on the Wonderlic are also more likely to struggle in the NFL, regardless of their other strengths.6,7
Could such an instrument have the same impact on the fire service? Many of the same characteristics required to be a successful quarterback are also those needed to be an effective fire officer. Don’t we want persons who can think on their feet, are intelligent, and can make good decisions under pressure? The Wonderlic simply provides an example of an instrument that tests for specific human characteristics that are highly correlated with the ability to make sound decisions under pressure (photo 3). This and other similar instruments are used by a myriad of industries when attempting to hire or promote individuals to critical positions within an organization. Instruments such as the Wonderlic do not necessarily ask job-specific questions; they tend to ask general knowledge questions that require candidates to problem solve. This testing method and others have proven to be reliable and valid instruments for identifying certain human characteristics that are useful in predicting future job success. (6, 7)
Traditionally, fire service organizations have used written exams that ask questions related to fire, management, and leadership to narrow applicant pools for promotion. It goes without saying that having a basic understanding of these concepts is critical to the success of any fire service leader, but what about the intangibles? It has been shown that quarterbacks with all the basic skills and body makeup who do not have certain intangible characteristics (those tested by the Wonderlic) struggle in the NFL. Just as baseball players who hit for power and “project” but lack the ability to be patient and earn walks are less likely to succeed in the big leagues, firefighters who have a strong understanding of fire strategy and tactics but lack the intangibles to lead, teach, and make sound decisions likely will not prove to be good officers.
The point is that fire departments can save resources and more accurately predict future success of candidates if hiring and promotional exams are reliable and valid instruments for selecting candidates with the particular skill set desired. Shouldn’t we be able to assume that someone competing for a promotion has at least a basic understanding of fire strategy and tactics? Why waste resources retesting these skills? Why not focus resources on identifying the individuals who have the intangible characteristics that have been shown to strongly correlate with effective leaders and managers in other industries that may or may not be similar to the fire service?
There is no need to reinvent the wheel when developing a promotional or hiring process.
There is a wealth of literature about characteristics of successful officers in the fire service that can be used in conjunction with management and leadership research from other professional fields, such as the military, sports, or business. For example, the characteristics of a good leader described by Coach John Wooden in Wooden on Leadership,8 who coached the University of California, Los Angeles, men’s basketball team to 10 National Collegiate Athletic Association championships, are similar to those required to successfully lead a fire company, a battalion, or an entire fire department. If your promotional and hiring processes are not testing for the important characteristics that are synonymous with effective job performance, then it may be time to think outside of the proverbial “box” and develop new processes that are reliable and valid for selecting individuals who will serve in and lead your organizations. Remember, it is all about the people.
References
1. Varone, J.C. Legal Considerations for Fire and Emergency Services. New York: Thomson Delmar Learning, 2007.
2. Harari, O. The Leadership Secrets of Colin Powell. New York: McGraw-Hill, 2002:24.
3. Babbie, E. The Practice of Social Research(10th ed.). Belmont, CA: Wadsworth/Thompson Learning, 2004.
4. Singleton, R.C. & B.C. Straits.Approaches to Social Research (3rd ed.). New York: Oxford University Press, 1999.
5. Lewis, M. Moneyball: The Art of Winning an Unfair Game.New York: W.W. Norton & Company, 2004.
6. Kotala, C. Wonderlic Reaches Well Beyond the NFL. April 2006. Retrieved August 15, 2006 from http://www.usatoday.com/sports/football/draft/2006-04-16-wonderlic-feature_x.htm/.
7. Wonderlic Website. NFL Testing Provides Valuable Lessons for all Employers. March 2005. Retrieved August 15, 2006 from http://www.wonderlic.com/resources/press/2005-03-01.asp/.
8. Wooden, J. & S. Jamison, S. Wooden on Leadership. New York: McGraw-Hill, 2005.
CARR BOYD is a captain with the Charlotte (NC) Fire Department and an adjunct faculty member in the Fire Safety Engineering and Political Science departments at the University of North Carolina at Charlotte (UNCC). He is a nationally registered paramedic, has a master of public administration degree, and is a Ph.D. candidate in the public policy program at UNCC.
ROB CANNON is a captain with the Charlotte (NC) Fire Department and a lieutenant commander in the Navy Reserve. He is a graduate of the United States Naval Academy. He served for three years as an officer aboard the destroyer USS JOHN RODGERS (DD-983) and two years at the Pentagon on the chief of naval operations staff for manpower, personnel, and training. He is a graduate of the Department of Defense Emergency Planning Liaison Officer course; he has a bachelor of science degree from the United States Naval Academy, a master of public administration degree from the University of North Carolina at Charlotte, and a diploma in national security studies from the U.S. Naval War College.
