Infrastructure Management and Extreme Events (IMEE) program
NSF Award #: 1361222
Team
San Diego State University – Department of Geography (lead)
Douglas Stow (Co-PI),
Lloyd (Pete) Coulter (Project Manager), Emanual Storey (PhD student), Sory Toure (PhD student), Andrew Kerr (MS student), Garrick MacDonald (MS student), Eugene Schweizer (MS student), Christopher Chen (Intern)
San Diego State University – Department of Electrical and Computer Engineering
Sunil Kumar (Professor)
University of New Mexico
Christopher Lippitt (Co-PI), Andy Loerch (MS student), Tammira Taylor (BS student)
Consultants
Richard McCreight (NEOS Ltd., airborne data consulting) and Nicholas Zamora (software development support)
Background
Within the first hours following a hazard event (e.g., earthquake, flood, tsunami, wildfire passage, nuclear accident, etc.) the key priority is to initiate life-saving activities. First responders and emergency managers need validated situational awareness of the status of critical infrastructure (e.g., utilities, bridges, hospitals, dams, etc.). The most reliable, detailed, and comprehensive means for early and documentable reconnaissance of post-event damage assessment is through low cost airborne imaging systems supported by semi-automated image processing and analysis, and coordinated image/map dissemination capabilities. As the only synoptic sensing technology available, remote sensing represents a critical source of information on the status of infrastructure following hazard events.
The focus of this study is on assessing damage to infrastructure following a major hazard event using airborne remote sensing. The premise is that some infrastructure, particular in cities, is so critical to saving human lives and supporting emergency response actions that near real-time information on the damage status of such infrastructure is essential and yet may be difficult to ascertain with conventional, ground observations and sensor networks. We hypothesize that the solution to this post-hazard information access challenge is to design flexible, ready-to-deploy, time-sensitive remote sensing systems (TSRSS) based on a network of airborne platforms and digital cameras. Our team is collaborating on research pertaining to important elements of end-to-end TSRSS that supports post-disaster assessment of damage to critical infrastructure and allocation of emergency response resources.
The seven critical elements of an end-to-end system for rapid infrastructure assessment from aerial image based damage detection are: (1) pre-event planning and baseline data preparation, (2) platform/sensor type and deployment (i.e., tasking), (3) capture and transmission of image data (or derived products) from airborne platforms to ground-based command and control centers, (4) precise registration (alignment) of time sequential images so that they may be compared (either on-board the aircraft or on the ground following transmission), (5) detection of changes evident within airborne images, (6) dissemination of primary data and derived information to analysts and first responders, and (7) the use of that information in a decision process of some value (e.g., recue prioritization, improved routing, evacuation, etc.).
An approach referred to as repeat station imaging (RSI) (formerly frame center matching) is used during airborne image capture and processing, in order to yield highly precise co-registration of airborne images captured over time. Precise co-registration of multitemporal airborne images is required for automated change detection, and also facilitates rapid visual change detection. RSI is based upon matching imaging stations in terms of horizontal position and altitude between multitemporal image acquisitions. When image frames are captured from exactly the same imaging station in the sky between multitemporal acquisitions, there is no parallax between images and they exhibit the same terrain related geometric distortions (they are essentially carbon copies of each other). SDSU has developed specialized procedures for precisely collecting airborne images from pre-determined image stations and co-registering multitemporal images collected from matched stations. The procedures enables automated and precise co-registration of multitemporal imagery with ultra-high spatial resolution. The simplified image registration approach enables automated and accurate image co-registration for near real-time change detection. Examples “b”-“g” in Figure 1 illustrate the differences between non-station matched image sets and station matched image sets.
Achieving reliable results from automated detection, delineation and/or identification of post-disaster damage based on registered, high spatial resolution multitemporal image sets is challenging. However, some form of automated, rapid detection and mapping is critical to meeting information timeliness requirements. Automated damage detection may be based on multitemporal differencing of image brightness, texture and/or spectral transforms derived from normalized repeat-pass imagery. Once changes are detected, maps of potential damage accompanied by before and after disaster images may be provided to an image analyst who may accept, edit or over-ride the detection results. As part of a Department of Homeland Security (DHS) funded project for post-earthquake disaster response, our team developed and demonstrated a system for rapid and automated change detection. We participated in the Research and Experimentation for Local and International Emergency and First-Responders (RELIEF) event at Camp Roberts, CA in August 2012, and demonstrated rapid multitemporal image collection with automated image co-registration and change detection (Figure 2).
Objectives
- Investigate the information requirements of infrastructure managers following hazard events.
- Examine factors affecting the timeliness of delivery from airborne remote sensing systems.
- Analyze factors affecting the reliability of infrastructure damage detection.
- Evaluate the capacity of current remote sensing technology and practice to address information requirements of various emergency response and facilities managers.
- Assess the viability of design and operation paradigms borrowed from systems engineering, operations research, and software engineering to enable the design of TSRSS and networks to meet user requirements.
- Research and develop tools and techniques for automated change detection.
Research Questions
- What infrastructure is critical, how soon after a natural disaster is information about critical infrastructure condition needed, and what information can be supplied by remote sensing?
- How can TSRSS be designed to satisfy the information needs of emergency and infrastructure managers following natural and anthropogenic hazard events?
- How well can geographic and engineering theories of communication and complex systems design improve the design and effectiveness of TSRSS of infrastructure damage assessment?
- What are the image collection requirements that enable detailed damage detection required by emergency response and infrastructure managers?
- Does faster information delivery from remote sensing necessitate a reduction in the reliability of the information provided?
- How much time would lapse from determination of an information need to delivery of sufficiently reliable information derived from an optimized airborne TSRSS?
- What are the components of a TSRSS that hold the most promise for minimizing time expenditure and what technical refinements are needed to generate reliable information?
- What types of damage features can be detected semi-automatically with a degree of reliability through implementation of a TSRSS based on the repeat station imaging approach?
Research Plan
The research plan includes theoretical exploration and empirical implementation of a remote sensing communication model (RSCM) developed by Dr. Lippitt, as well as testing pre-event flight planning, image registration, and automated damage detection tools, to determine the timeliness and accuracy characteristics of these TSRSS components. We are working with existing remotely sensed data sets already collected by the team and newly acquired imagery that more realistically represented damage features of primary interest. Surveys of participating emergency managers during the project (San Diego County Office of Emergency Services (OES), City of Albuquerque Office of Emergency Management, Los Angeles County Fire Department, and U.S. Civil Air Patrol) are guiding image data collection requirements. The research methodology is organized as follows:
Survey emergency managers
Optimal configuration of remote sensing systems requires that information delivered to the user be both relevant to and readily employed by that user. Therefore, it is critical to understand the types of information required by infrastructure managers following hazard events, the timeliness in which that information is required, and the levels of geometric and thematic accuracy necessary to facilitate effective decision making. To determine this information a series of interviews is being conducted with the hazard and infrastructure managers listed above. A two-stage interview process is being employed to: (1) determine the technical capacity, training, and decision processes of infrastructure manager in each jurisdiction and (2) determine the qualitative utility and desirability of products derived from TSRSS.
Quantify time for collection and delivery of primary image data
We are utilizing RSCM to simulate the timeliness of primary image data delivery for several possible TSRSS configurations under a range of hazard scenarios identified by hazard and infrastructure managers. We are also utilizing F-Planar flight planning software from TerraPan Labs LLC to enable rapid and interactive flight planning for repeat station imaging. This tool delivers flight plan information (e.g., number of flight lines and images) to simulate and estimate time required for the tasking, acquisition, and data transmission portions of the TSRSS for the available assets in each region.
Develop analytical model for image collection specifications
We are developing an analytical model based on photogrammetric principles to explain the interdependence between repeat-pass station matching accuracy, type of registration warping function, and the ultimate quality of multitemporal image co-alignment (i.e., co-registration error). We are validating the model using existing and newly acquired airborne image sets collected by our team.
Develop and test automated image co-registration software
We have engaged a consultant to assist us in developing and testing a software tool for automated co-registration of repeat station image sets. This tool is facilitating validation of the analytical model described above and testing of robust change detection algorithms with numerous multitemporal images by enabling automated co-registration of a range of image pairs collected from matched imaging stations. It also enables us to quantify timeliness and reliability of automated repeat station image alignment.
Evaluate optimal design and associated time for wireless air-to-ground image transfer
The data generated and captured by airborne imaging sensors must be transmitted to the ground-based command and control station(s) in a reliable and timely fashion while meeting the required quality of service (QoS) demands of the user and application. We are investigating: (1) required compression and target detection so that only the features of interest in an image frame are transmitted with higher fidelity; (2) considering the priority of source data and their packets based on their roles in the mission; and (3) cross-layer wireless protocols (e.g., admission control, bandwidth allocation and resource optimization) for reliable transmission of images and other sensor data for meeting latency constraints.
Develop and test automated damage detection algorithms
SDSU is building off of existing experience, reviewing current literature, and communicating with colleagues to identify effective techniques for achieving reliable detection of change associated with infrastructure damage and minimizing detection of false change. False change may be associated with such things as moving shadows, moving objects (e.g., vehicles), vegetation phenology, etc. Therefore, we are utilizing methods such as Maximally Stable Extremal Regions to detect and remove shadows. Methods for shadow restoration are being evaluated for restoring image data in shaded regions. We are also creating shadow and illumination invariant, reflectance-only panchromatic images from red, green, and blue (RGB) color composites from color digital cameras.
Change detection steps include radiometric/brightness normalization between images, differencing of brightness, local texture and indices, and/or other image transformations, and a series of post-classification filtering steps to refine results (e.g., majority filtering to remove isolated changes, specifying minimum object size, etc.).
Estimate total time to information product delivery for a matrix of TSRSS model scenarios
The total time to information delivery is being estimated for a range of TSRSS and for the likely hazard event scenarios identified through interviews of hazard response and infrastructure managers. Time to image delivery is being combined with empirically derived estimates of processing, distribution, and dissemination times to produce a matrix of potential TSRSS capable of assessing infrastructure damage. The matrix is being provided to hazard response and infrastructure managers along with derived damage detection products during the second phase of interviews to assess their prioritization of reliability or timeliness of information when responding to extreme hazard events.
Evaluation of theoretical approaches and design paradigms for optimizing TSRSS
System design paradigms from computer science and engineering is being critically evaluated based on their potential to (1) enable the design of TSRSS based on user information requirements and (2) to enable the reliable characterization of user requirements even in cases where users are unable to readily supply those requirements. Potential to enable reliable characterization of user requirements is being assessed by comparing the paradigm prescribed methodology for obtaining user requirements to our experience trying to obtain requirements from infrastructure managers in and near San Diego County, CA and Bernalillo County, NM.
Outcomes
Master’s Theses
Andrew Kerr (SDSU), in progress. Optimizing Radiometric Fidelity through Image Capture and Processing to Enhance Aerial Image Change Detection.
Book Chapters
Coulter, L., D. Stow, C.D. Lippitt and G. Fraley, 2015. Repeat Station Imaging for Rapid Airborne Change Detection, book chapter in Time Sensitive Remote Sensing, Springer-Verlag, 29-43.
Lippitt, C.D. and D. Stow, 2015. Remote Sensing Theory and Time-Sensitive Information Requirements, book chapter in Time Sensitive Remote Sensing, Springer-Verlag, pp. 1-9.
Stow, D., C.D. Lippitt, L. Coulter and B. Davis, 2015. Time Sensitive Remote Sensing Systems for Post-Hazard Damage Assessment, book chapter in Time Sensitive Remote Sensing, Springer-Verlag, pp. 13-28.
News Media
Homeland Security News Wire
San Diego Union Tribune
NBC San Diego News
San Diego 6 News
San Diego State University press release
Daily Aztec (SDSU campus newspaper)
UNM Project Web Page