Enhancing Passenger Safety in an Autonomous Bus: A Multimodal Fall Detection Approach for Effective Remote Monitoring

Abstract

With the rise of autonomous public transportation, passenger safety in autonomous buses is paramount. This paper introduces a novel Multimodal Long Short-Term Memory (LSTM) network-based fall detection system, to enhance passenger safety by accurately detecting falls, thereby assisting remote supervisors in efficiently monitoring multiple vehicles. It comprises two main processes: feature extraction and fall discrimination. Feature extraction utilizes YOLO (You Only Look Once) v7 for real-time pose estimation, combined with the SORT algorithm for tracking individuals across video frames. Fall discrimination leverages sequential data processing with LSTM networks in the proposed Multimodal approach, which employs a fusion of pose estimation and Inertial Measurement Unit (IMU) sensor data. Evaluations conducted in various fall and non-fall scenarios within a realistic bus setting yielded high recall and F2 scores. Specifically, the model attained 98% recall in single person fall scenarios and 95% in more complex multi-person fall scenarios, significantly surpassing traditional single-modality approaches such as Multilayer Perceptron (MLP) and simple LSTM. The paper also investigates a decision-level fusion approach, balancing predictive accuracy by optimizing the late integration of separate Pose and IMU models, despite its higher computational cost. The impact of varying frame rates on model performance was also explored, addressing practical implications for real-world implementations. The robustness of the model was affirmed even at reduced processing frame rates, ensuring effective real-time processing capabilities crucial for ensuring timely responses in emergency situations. The development of the fall detection system promises a safer and more efficient future for public transportation, especially benefiting elderly individuals.