By Kirk McKinzie
Like it or not, the American and global fire and emergency response industry must continue to work very hard. The question at hand is, when will we start working SMART? This article presents some of the advances being made under the leadership of the Department of Homeland Security Science and Technology Directorate (DHS-S&T) Next Generation First Responder Apex Program, the National Fire Protection Association (NFPA), and a variety of international groups focused on saving lives through the advancement of SMART technologies. First responders are using their own phones to receive dispatch information, real-time traffic data analytics, and the locations of fellow responders. One day, SMART smoke, heat, and bio-telemetry sensors will allow first responders to know who and where a patient is and, if the patient allows it, the patient’s heart rate.
(1) This image, derived from a high-definition laser scan with a photographic overlay, is a digital twin of a five-story training tower. (Photo used with permission of Leica Geosystems.)
(2) Teams around the world, such as RiVR, are advancing photorealistic VR interactive room scale in the fields of fire and crime scene investigation for public and responder training. (Photos and screen captures from RiVR.)
Technologies developed for gaming and entertainment are being exploited to provide real-time three-dimensional (3-D) data sets (photo 1) and machine-to-machine (M2M) and machine-to-human (M2H) interfaces to provide responders with information they need. Auto-populated information can allow fire service, law enforcement, and medical personnel to visually understand their environment prior to and during response and in the future through predictive tools for structure and wildland fires, hazardous materials releases, active shooter incidents, and natural disasters.
RELATED
The Future of Artificial Intelligence in Firefighting
DARPA Demo Day: a Glimpse of the Expanding Technological Frontier
Robotics for Reducing Firefighter Injuries: Now and Potential
SMART solution suites are posed to provide citizens with lifesaving information through public education for emergency preparedness awareness or situational awareness in real time during emergencies. Akin to augmented reality gaming applications but aimed at helping victims escape harm instead of capturing digital characters, the “911-GO” application would direct citizens to exits during fires, active shooter incidents, earthquakes, floods, and other life-threatening events.
Virtual Reality and Augmented Reality (VR, AR), aka Mixed Reality (MxR), allow for the development of blended information from surroundings and output-mixed reality environments that can be viewed through 3-D displays in VR; AR; and, in the future, holographic light displays. Instead of an incident commander (IC) strategizing during a high-rise fire by calling for personnel accountability reports like LUNAR (Location, Unit, Name, Assignment/Air supply, Resources), what if the IC could see the whole event and know personnel’s positions, air supply, and heart rate and receive real-time imaging in 360o and 3-D?
Introduction
Development of applications and early funding of programs and processes directed at assisting first responders and the public have begun, but much more is needed. All humans, regardless of socioeconomic position, cultural heritage, educational status, age, and gender, will reap the rewards of the merging of MxR, “the Internet of things” (IoT), InfraRed (IR) technology, and remote sensing (RS) for first responders. It will save lives. The value of lucrative investments in 3-D games and action movies could be of enormous use when stakes are high, moments are fleeting, and actions are critical. What if the public safety industry leveraged available entertainment tech and added artificial intelligence? Technological components exist to provide dispatch centers, emergency responders, and their commanders and citizens with critical information in the time of need, in real time through MxR. It is possible to use a wristwatch to program a security alarm system, dictate a research paper, or provide live video conferencing with a loved one. Now, ICs and physicians are using the technology for saving lives. When buildings collapse or are alight with flames and during active shooter incidents, the world’s first responders and the global inhabitants deserve, as much as any superhero action figure, to know the emergency scene as a 3-D holodeck. Based on successful research, it has been proven to be possible. These 3-D technologies have become an integral part of the public’s perception of the world through film, video games, and home entertainment; yet the humanitarian value of this advancing technology lies in the use of 3-D by the first responder community to save lives. Government and private business venture capitalist funding for these programs and processes directed at assisting first responders and the public in life-saving instances must be increased; the fact is, lives depend on it.
(3) This screen snap of a 360° video, which looks like a black-and-white photo, allows citizens and elected officials to virtually witness structural fire operations. (Image courtesy of author.)
(4) Qwake Technologies is focused on simplifying complex sensor data like thermal images into intuitive visual cues. Its product C-THRU has been developed to enhance a firefighter’s vision and decision-making abilities through computer vision algorithms and augmented reality. (Photo courtesy of Qwake Technologies.)
(5) Global SMART city designers at The Digit Group suggest that homes be equipped with active listening detectors that tell where the fire in a house is located, how rapidly it is growing, where occupants are, and the victim’s heart rate. (Photo used with permission of The Digit Group.)
Review of Literature
Support for the research, development, and production of wearable technologies multilinked by wireless technology, cloud computing, and RS devices is critical. They can help prevent, mitigate, and analyze life-threatening events. There has been a variety of successes from groups working on technologies that previously were reserved for entertainment. Proof that AR, VR, 3-D, and IoT are valuable for keeping citizens safe instead of just entertained is obvious in “Data Driven in Pursuit of Smart.” The opening scenario has Chief Eugene Ditaranto of New York City “only a few keyboard clicks away from one of the most technologically sophisticated tools ever developed by the fire service.” The chief will ultimately be able to determine which of the 330,000 buildings in his district “are at highest risk of experiencing a fire that day.” After inspections, “firefighters can upload newly collected building information onto the Risked-Based Inspection System (RBIS) site. Information can include the locations of standpipes and alarm panels; building specifications; new violations; changes in occupancies; changes to fire suppression systems; unusual hazards or means of egress; and much more.”1
Further discussion comes from the National Institute of Standards and Technology’s (NIST’s) “Special Publication 1174 Smart Firefighting Workshop Summary Report.”2 The report indicates that new technologies, which include real-time firefighter location systems; reliable fire spread predictors; physiological monitoring of responders; VR and AR; evacuation route optimization; unmanned aircraft (UAV); weather, terrain, and traffic data; and enhanced incident command, could save lives.
The workshop provided an opportunity to identify and define the research and development needed for implementation of SMART firefighting. The goals included highlighting existing technologies and the deployment of emerging technologies, including cyber-physical systems (CPS). The team brainstormed the need for standards in data collection, exchange, storage, and situational awareness tools.
Additionally, the report cites recent advances at the Center for Firefighting Excellence/Fire Service CPS Integration R & D Support Center, which will further develop technologies for fire service and first responder needs ranging from communications and data sharing to real-time fireground and human and equipment resource management. (2)
Some view the concept as global. In “3-D Building Reconstruction and Thermal Mapping in Fire Brigade Operations,”3 researchers argue that firefighters’ performance can be augmented through reality applications. They also predict a direct increase in safety and efficiency during fireground operations. Researchers at the Department of Electrical Engineering and Computer Science at the Berkeley campus of the University of California propose a fully automatic system that would construct a 3-D thermal point cloud that would reflect in detail the building’s interior geometry.4 The 3-D model would include also walls; floors; ceilings; and objects such as furniture, lights, and windows. The approach is based on a wearable collection of sensors comprised of Light Detection And Ranging scanners and IR and optical cameras. (4)
While working on thermal dynamics for energy efficiency in commercial structures, researchers in Australia found their system was also able to produce an accurate and detailed 3-D model that could both provide RGB (red, green, and blue) and thermal and IR data. When first responders and ICs are visually aware of interior spaces, including their accurate temperature readings, safety is increased exponentially.5
Applying Technologies to Fire-Rescue
The special effects technologies being exploited in the film and gaming industries could be advanced in a way to keep those industries thriving by saving the lives of the gamers and movie-goers. If only a small fraction of the money spent on a major motion picture was dedicated for emergency responders’ use for more positive outcomes! Each of the aforementioned studies indicates that 3-D technologies could assist in a person’s time of need. It must first be established that the advancements in 3-D technologies can spare lives and reduce the negative financial impacts of emergency incidents before there is motivation to shift funding from entertainment to emergency response.
The NFPA is a respected name in the global fire service industry and the provider of and advocate for consensus codes and standards, research, training, and education. It is comprised of more than 65,000 members whose mission is to reduce the worldwide burden of fire and other hazards on the quality of life. It reported that fire departments in the United States responded in 2017 to an estimated 1,319,500 fires that resulted in 3,400 civilian fire fatalities; 14,670 civilian fire injuries; and an estimated $23 billion in direct property loss. There was a civilian fire death every two hours and 34 minutes and a civilian fire injury every 36 minutes.6
(6). The view from inside augmented reality glasses will provide incident commanders with enhanced situational awareness in 3-D and the name, location, bio-telemetry, and air supply of personnel. (Photo courtesy of Motorola Solutions Inc.)
The Centers for Disease Control and Prevention (CDC) is a natural place to turn to for estimates on the need for protecting humans. This is evident from its motto: “CDC 24/7; Saving Lives, Protecting People”; those efforts are visible through data collection on injuries that constitute an estimated 31 million emergency department visits annually.7 These statistics reflect the need to research and develop tools to mitigate the risk and manage the scenes when emergencies occur.
Cyber-physical, interconnected, 3-D SMART technologies for emergency response must be immediately effective to justify expending the time, effort, and funding for developing hardware and software applications for the next generation. 3-D scene documentation and AR/VR applications for fire department and law enforcement must provide responders and commanders with an enhanced understanding of their location and situation. At the 2013 Augmented Human International Conference, researchers from the Vienna University of Technology successfully reconstructed a 3-D model of an interior structure in real time and fused the visualization with the image of a thermal imaging camera (TIC). In addition, they noted that ICs would have a better overview of the situation.
Some Examples of Relevant Technologies
Opportunities exploited through the technology include “highlighting hot spots or human victims in need of rescue.”(3, 202) In the United States, the Fire Protection Research Foundation (FPRF), a research affiliate of the NFPA, acknowledged the immediate usefulness of these technologies. In “CPS-Smart Fire Fighting Creating the Research Roadmap for Smart Fire Fighting,” the FPRF cited a myriad of current and not-too-far-in-the-future values in cyber-physical systems related to wildland urban interface (WUI), real-time firefighter location, remote sensor information, current and expected weather forecasts, terrain, and traffic data.8 These systems in cyber tools will assist with situation awareness through heads-up technology and AR for firefighters. A variety of wearable virtually connected devices, including real-time situational sensors, could track people, equipment, and human assets during an emergency incident. Benefits would include everything from a first responder’s heart rate to the air supply of self-contained breathing apparatus; enhanced scene and building information; interconnectivity of communication devices and systems; and enhanced situational awareness provided by simple, intuitive advancements through computerized algorithms. (8) The research in Vienna and the United States suggests there will be an immediate value available to emergency responders using wearable, interconnect, and AR and VR technologies.
More than 100 fire and emergency response industry professionals, scientists, cyber-physical experts, and Silicon Valley icons attended NIST’s Smart Firefighting Workshop in early 2014. They predicted many devices, including I-sensors with the ability to document a scene without light. The use of IR TICs by response personnel has been in the mainstream for more than a decade. This same technology is proposed for use by cyber-physical systems to assist responders in low and zero light. (2)
In addition to the work done in the United States, researchers at the Queensland University of Technology, Brisbane, Australia, used an RGB-D sensor and a TIC to meet the needs of emergency responders when working in limited lighting. They demonstrated that the handheld system is capable of accurately mapping difficult environments, even in complete darkness. (5) Totally dark environments are an everyday experience in the fire service and a frequent experience for members of law enforcement, in particular those engaged in Special Weapons And Tactics. Augmenting their perception in real time, with no lighting, could help save their lives and those of citizens.
Researchers have repeatedly documented a need for CPS, AR/VR, IoT, and M2H interoperability in 3-D. That technology, as designed, is functioning for responders in trials, even in total darkness.
The value of these technologies for command staff is in defining the environment geographically and visually and combining the data with other sensors and transmissions. In addition to scanning in darkness, the Australian team noted its devices allow ICs to be visually aware of interior spaces, including accurate temperature readings, increasing knowledge for strategic and tactical decisions, communications, and safety exponentially. They discovered that wireless “output of our proposed system could see immediate applications in areas that would benefit from rapid 3-D visualization or analysis of surface temperature information. They include fire management and response.” (5)
(7) The Department of Homeland Security Science and Technology Directorate (DHS S&T) Next Generation First Responder Apex program is funding advanced research at the National Aeronautics and Space Administration Jet Propulsion Lab (NASA-JPL), developing computer vision and artificial intelligence solutions. The work of NASA-JPL will eventually be available as an open source set of codes. (Photo courtesy of DHS S&T.)
The types of incidents at which the IC will have CPS-enhanced knowledge offers varied benefits. M2M and M2H technologies will be used when managing incidents involving the WUI; commercial, high-rise, and residential fires; vehicle extrications; train derailments; hazmat incidents; natural disasters; active shooter incidents; mass-casualty incidents; and technical rescue incidents. (8)
The Future
With the advent and advance of SMART 3-D/360 technologies, the international first responder community will be prudent to collaborate with public and private entities to enable systems aimed at saving lives in prevention, mitigation, and postincident analysis. Fire marshals, operation officers, and fire and explosion investigators will individually and collectively have access to transformational 3-D systems that will improve visualization, evaluation, and integration of disparate data. The future ability to augment responders with legacy data stored in cloud computing resources that can be transmitted to interior crews and ICs will change strategic and tactical considerations in fires, active shooter incidents, technical rescues, building collapses, and natural disasters.
Where 3-D data is sparse or lacking, it should be collected during the response from helmet-mounted 3-D scanning hardware that can see in the dark so the data can be transmitted wirelessly to ICs and merged with legacy data for immediate mitigation uses and stored for future needs. Privacy concerns exist, and citizens must be given the opt in/out option, as the debate will continue. Opportunities include saving lives of law enforcement officers by using lapel cameras equipped with facial recognition software; the IoT 3-D will provide that opportunity. It will also assist field medical providers by converting a John Doe patient into a son, brother, or father by making available information on next of kin, medical history, and more.
Funding is available for 3-D technology for first responder communities. There are a few qualifying factors, however. Primarily, the DHS, NIST, and other federal grant and philanthropic programs allow first responders to acquire select hardware components and software suites. Access for these funds must be streamlined and made accessible to lean and efficient professionals who are providing the research and advancements in this new venue so that the technology can be developed rapidly and efficiently. Regardless of political orientation, venture capitalists, the federal government, insurance companies, and global corporate and private interests all play a role in the shifting of funds to invest in current and future generations by saving lives and property.
References
1. Roman, J. (2014). “Data driven in pursuit of smart.” NFPA Journal, 40-50. Retrieved from http://www.nfpa.org/research/fire-protection-research-foundation/current-projects/developing-a-research-roadmap-for-the-smart-fire-fighter-of-the-future/.
2. Hamins, A, N Bryner, A Jones, G Koepke, C Grant, & A Raghunathan. (2014). NIST Special publication 1174 smart firefighting workshop summary report. Retrieved from http://dx.doi.org/10.6028/NIST.SP.1174/.
3. Schönauer, C, E Vonach, G Gerstweiler, & H Kaufmann. (2013). 3D Building reconstruction and thermal mapping in fire brigade operations. Retrieved from https://www.ims.tuwien.ac.at/publications/tuw-217689/.
4. Oreifej, O, PhD; J Cramer, & A Zakhor, PhD. (n.d.). Automatic generation of 3D thermal maps of building interiors. Retrieved from http://www-video.eecs.berkeley.edu/papers/oreifej/oreifej_ashrae_2014.pdf/.
5. Vidas, S, P Moghadam, & M Bosse. (n.d.). 3D Thermal mapping of building interiors using an RGB-D and Thermal Camera. Retrieved from http://peymanmoghadam.com/pdf/ICRA20133dthermal.pdf.
6. Evarts, Ben. (October 2018). Fire loss in the United States during 2017. Retrieved from https://www.nfpa.org/News-and-Research/Data-research-and-tools/US-Fire-Problem/Fire-loss-in-the-United-States/.
7. Centers for Disease Control and Prevention. (2013). Leading causes of death. Retrieved from http://www.cdc.gov/injury/overview/leading_cod.html/.
8. Grant, C. (2015). The Fire Protection Research Foundation. CPS-Smart firefighting creating the research roadmap for smart firefighting (pp 1-72). http://www.nfpa.org/~/media/files/research/research-foundation/current-projects/smart-ff/smartffslides.pdf?la=en/.
Kirk McKinzie is a captain in the Cosumnes (CA) Fire Department. He is a SMART first responder technologist and a 30-year fire service veteran. He is working with international teams focused on Next Generation solutions. He functions as a subject matter expert, an advisor, a speaker, and an author to government, enterprise, academia, start-ups, and operators focused on SMART lifesaving technological solutions. His equity interest is life—the goal is to save one.
