SpringerLink
Log in

Reliable information system for identifying spatio-temporal continuity of kinetic deformed objects with big point cloud data

  • Original Research
  • Published:
Annals of Operations Research Aims and scope Submit manuscript

Abstract

In the context of Industry 4.0, a wide range of sensors are extensively deployed to gather production and equipment operation data, while also connecting human workforce information through the industrial Internet of Things technology. This integration enables effective improvements in sustainable, human-centric, and resilient productivity by leveraging industrial process control and automation. In this paper, we propose an intelligent information system for analyzing large point cloud data sets from depth sensors, which are used for detecting, representing, locating, and sha** monitored objects. To address privacy concerns, our system only considers de-identified information during analysis, using a newly proposed dynamic clustering method based on multivariate mixture Student’s t-distribution for monitoring human motions. The information system consists of two main blocks: segmentation and dynamic clustering for monitoring or tracking. The segmentation algorithm, utilizing a multivariate mixture Student’s t-distribution, groups points into homogeneous partitions based on spatial proximity and surface normal similarity, without relying on any semantic indicator or pre-determined shape. The dynamic clustering algorithm, powered by an online learning state-space model, efficiently incorporates and updates the centroid position and velocity of the object being monitored. To evaluate the reliability of our proposed method, we introduce two time-consistent measures that account for different illumination levels, drastic limb movements, and partial or full occlusions during object motion processing. We conduct empirical experiments using a large point cloud data set, comparing our method with several alternative methods. The results highlight the superiority of our proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Algorithm 2
Algorithm 3
Algorithm 4
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. Here, the term “frame" denotes a single image of video or temporal data. In this study, we treat a frame and the corresponding point cloud as interchangeable entities, representing the same information.

References

  • Abdel-Basset, M., Chang, V., & Hawash, H. (2021). Deep learning approaches for human-centered IOT applications in smart indoor environments: A contemporary survey. Annals of Operations Research. https://doi.org/10.1007/s10479-021-04164-3

    Article  Google Scholar 

  • Al Qundus, J., Dabbour, K., Gupta, S., Meissonier, R., & Paschke, A. (2020). Wireless sensor network for ai-based flood disaster detection. Annals of Operations Research, 391, 697–719.

    Google Scholar 

  • Arshad, N., Moon, K.S., Kim, J.N., (2010). Multiple ship detection and tracking using background registration and morphological operations, in: Pal, S.K., Grosky, W.I., Pissinou, N., Slezak, D. (Eds.), Signal Processing and Multimedia.

  • Berlińska, J. (2015). Scheduling for data gathering networks with data compression. European Journal of Operational Research, 246, 744–749.

    Article  Google Scholar 

  • Berlińska, J., & Przybylski, B. (2021). Scheduling for gathering multitype data with local computations. European Journal of Operational Research, 294, 453–459.

    Article  Google Scholar 

  • Bertinetto, L., Valmadre, J., Golodetz, S., Miksik, O., Torr, P., (2016). Staple: Complementary learners for real-time tracking.

  • Bouchachia, A. (2012). Dynamic clustering. Evolving Systems, 3, 133–134.

    Article  Google Scholar 

  • Boulemtafes, A., Derhab, A., & Challal, Y. (2020). A review of privacy-preserving techniques for deep learning. Neurocomputing, 384, 21–45.

    Article  Google Scholar 

  • Brigo, D., Pisani, C., & Rapisarda, F. (2021). The multivariate mixture dynamics model: Shifted dynamics and correlation skew. Annals of Operations Research, 299, 1411–1435.

    Article  Google Scholar 

  • Chaloun, T., Brandl, S., Ambrosius, N., Kröhnert, K., Maune, H., & Waldschmidt, C. (2023). Rf glass technology is going mainstream: Review and future applications. IEEE Journal of Microwaves, 3, 783–799.

    Article  Google Scholar 

  • Chen, B., Li, P., Sun, C., Wang, D., Yang, G., & Lu, H. (2019). Multi attention module for visual tracking. Pattern Recognition, 87, 80–93.

    Article  Google Scholar 

  • Chen, C. Y. T., Sun, E. W., Chang, M. F., & Lin, Y. B. (2023). Enhancing travel time prediction with deep learning on chronological and retrospective time order information of big traffic data. Annals of Operations Research. https://doi.org/10.1007/s10479-023-05223-7

    Article  Google Scholar 

  • Chen, Y. T., Sun, E. W., Chang, M. F., & Lin, Y. B. (2021). Pragmatic real-time logistics management with traffic iot infrastructure: Big data predictive analytics of freight travel time for logistics 4.0. International Journal of Production Economics, 238, 108157.

    Article  Google Scholar 

  • Danelljan, M., Bhat, G., Khan, F.S., Felsberg, M., (2017). Eco: Efficient convolution operators for tracking, in: IEEE conference on computer vision and pattern Recognition.

  • Delavernhe, F., Jaillet, P., Rossi, A., & Sevaux, M. (2021). Planning a multi-sensors search for a moving target considering traveling costs. European Journal of Operational Research, 292, 469–482.

    Article  Google Scholar 

  • Delavernhe, F., Lersteau, C., Rossi, A., & Sevaux, M. (2020). Robust scheduling for target tracking using wireless sensor networks. Computers and Operations Research, 116, 104873.

    Article  Google Scholar 

  • Delavernhe, F., Rossi, A., & Sevaux, M. (2021). Spatial and temporal robustness for scheduling a target tracking mission using wireless sensor networks. Computers and Operations Research, 132, 105321.

    Article  Google Scholar 

  • Elloumi, S., Hudry, O., Marie, E., Plateau, A., & Rovedakis, S. (2021). Optimization of wireless sensor networks deployment with coverage and connectivity constraints. Annals of Operations Research, 298, 183–206.

    Article  Google Scholar 

  • Frikha, A., & Moalla, H. (2015). Analytic hierarchy process for multi-sensor data fusion based on belief function theory. European Journal of Operational Research, 241, 133–147.

    Article  Google Scholar 

  • Fuentes-Jimenez, D., Martin-Lopez, R., Losada-Gutierrez, C., Casillas-Perez, D., Macias-Guarasa, J., Luna, C. A., & Pizarro, D. (2020). Dpdnet: A robust people detector using deep learning with an overhead depth camera. Expert Systems with Applications, 146, 113168.

    Article  Google Scholar 

  • Ghaleb, M., Zolfagharinia, H., & Taghipour, S. (2020). Real-time production scheduling in the industry-4.0 context: Addressing uncertainties in job arrivals and machine breakdowns. Computers and Operations Research, 123, 105031.

    Article  Google Scholar 

  • Giner, J. (2021). Orthant-based variance decomposition in investment portfolios. European Journal of Operational Research, 291, 497–511.

    Article  Google Scholar 

  • Gámiz, M. L., Limnios, N., & del Carmen Segovia-García, M. (2023). Hidden markov models in reliability and maintenance. European Journal of Operational Research, 304, 1242–1255.

    Article  Google Scholar 

  • Henriques, J. F., Caseiro, R., Martins, P., & Batista, J. (2015). High-speed tracking with kernelized correlation filters. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3), 583–96.

    Article  Google Scholar 

  • Hong, J., & Kwon, J. (2022). Optimal visual tracking using wasserstein transport proposals. Expert Systems with Applications, 209, 118251.

    Article  Google Scholar 

  • Horaud, R., Hansard, M., Evangelidis, G., & Ménier, C. (2016). An overview of depth cameras and range scanners based on time-of-flight technologies. Machine Vision and Applications, 27, 1005–1020.

    Article  Google Scholar 

  • Huang, D. H., Huang, C. F., & Lin, Y. K. (2020). A novel minimal cut-based algorithm to find all minimal capacity vectors for multi-state flow networks. European Journal of Operational Research, 282, 1107–1114.

    Article  Google Scholar 

  • Jamshed, M. A., Ali, K., Abbasi, Q. H., Imran, M. A., & Ur-Rehman, M. (2022). Challenges, applications, and future of wireless sensors in internet of things: A review. IEEE Sensors Journal, 22, 5482–5494.

    Article  Google Scholar 

  • Jiang, X., Bai, R., Wallace, S. W., Kendall, G., & Landa-Silva, D. (2021). Soft clustering-based scenario bundling for a progressive hedging heuristic in stochastic service network design. Computers and Operations Research, 128, 105182.

    Article  Google Scholar 

  • Jonasson, E. T., Ramos Pinto, L., & Vale, A. (2021). Comparison of three key remote sensing technologies for mobile robot localization in nuclear facilities. Fusion Engineering and Design, 172, 112691.

    Article  Google Scholar 

  • Kalal, Z., Mikolajczyk, K., & Matas, J. (2010). Tracking-learning-detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6, 1–14.

    Google Scholar 

  • Kammerdiner, A., & Guererro, A. (2019). Data-driven combinatorial optimization for sensor-based assessment of near falls. Annals of Operations Research, 276, 137–153.

    Article  Google Scholar 

  • Karabulut, E., Aras, N., & Altinel, I. K. (2017). Optimal sensor deployment to increase the security of the maximal breach path in border surveillance. European Journal of Operational Research, 259, 19–36.

    Article  Google Scholar 

  • Kart, U., Kamarainen, J.K., Matas, J., (2018). How to make an rgbd tracker. in: European Conference Computer Vision Workshop.

  • Kawamoto, A. L. S., & da Silva, F. S. C. (2018). Depth-sensor applications for the elderly: A viable option to promote a better quality of life. IEEE Consumer Electronics Magazine, 7, 47–56.

    Article  Google Scholar 

  • Keçeli, A. S. (2018). Viewpoint projection based deep feature learning for single and dyadic action recognition. Expert Systems with Applications, 104, 235–243.

    Article  Google Scholar 

  • Keskin, M. E. (2017). A column generation heuristic for optimal wireless sensor network design with mobile sinks. European Journal of Operational Research, 260, 291–304.

    Article  Google Scholar 

  • Konrad, K. A. (2020). Attacking and defending multiple valuable secrets in a big data world. European Journal of Operational Research, 280, 1122–1129.

    Article  Google Scholar 

  • Kozłowski, E., Mazurkiewicz, D., Żabiński, T., Prucnal, S., & Sep, J. (2020). Machining sensor data management for operation-level predictive model. Expert Systems with Applications, 159, 113600. https://doi.org/10.1016/j.eswa.2020.113600

    Article  Google Scholar 

  • Krishnamoorthy, S. (2022). A two-stage integer programming model considering transaction equivalence for privacy preservation. Computers and Operations Research, 148, 105997.

    Article  Google Scholar 

  • Lersteau, C., Rossi, A., & Sevaux, M. (2016). Robust scheduling of wireless sensor networks for target tracking under uncertainty. European Journal of Operational Research, 252, 407–417.

    Article  Google Scholar 

  • Li, Y., & Ibanez-Guzman, J. (2020). Lidar for autonomous driving: The principles, challenges, and trends for automotive lidar and perception systems. IEEE Signal Processing Magazine, 37, 50–61.

    Google Scholar 

  • Lin, Y. K., Fiondella, L., & Chang, P. C. (2022). Reliability of time-constrained multi-state network susceptible to correlated component faults. Annals of Operations Research, 311, 239–254.

    Article  Google Scholar 

  • Lin, Y. K., & Yeh, C. T. (2012). Multi-objective optimization for stochastic computer networks using nsga-ii and topsis. European Journal of Operational Research, 218, 735–746.

    Article  Google Scholar 

  • Liu, Y., **g, X. Y., Nie, J., Gao, H., Liu, J., & Jiang, G. P. (2019). Context-aware three-dimensional mean-shift with occlusion handling for robust object tracking in rgb-d videos. IEEE Transactions on Multimedia, 21, 664–677.

    Article  Google Scholar 

  • Lukezic, A., Vojir, T., Cehovin, L., Matas, J., Kristan, M., (2018). Discriminative correlation filter tracker with channel and spatial reliability. in: IEEE conference on computer vision and pattern Recognition.

  • Luna, C. A., Losada-Gutierrez, C., Fuentes-Jimenez, D., Fernandez-Rincon, A., Mazo, M., & Macias-Guarasa, J. (2017). Robust people detection using depth information from an overhead time-of-flight camera. Expert Systems with Applications, 71, 240–256.

    Article  Google Scholar 

  • Luo, W., Gu, B., & Lin, G. (2018). Communication scheduling in data gathering networks of heterogeneous sensors with data compression: Algorithms and empirical experiments. European Journal of Operational Research, 271, 462–473.

    Article  Google Scholar 

  • Nam, H., Han, B., (2016). Learning multi-domain convolutional neural networks for visual tracking, in: IEEE conference on computer vision and pattern Recognition.

  • Ning, X., Zhang, X., Wang, Y., Jaeger, M., (2009). Segmentation of architecture shape information from 3D point clouds. in: Proceedings of the 8th International Conference on Virtual Reality Continuum and its Applications in Industry, pp. 127–132.

  • Peel, D., & McLachlan, G. J. (2000). Robust mixture modeling using the t distribution. Statistics and Computing, 10, 339–348.

    Article  Google Scholar 

  • Rajput, A. S., Raman, B., & Imran, J. (2020). Privacy-preserving human action recognition as a remote cloud service using RGB-D sensors and deep cnn. Expert Systems with Applications, 152, 113349.

    Article  Google Scholar 

  • Redmond, M., Campbell, A. M., & Ehmke, J. F. (2022). Reliability in public transit networks considering backup itineraries. European Journal of Operational Research, 300, 852–864.

    Article  Google Scholar 

  • Shukri, S., Faris, H., Aljarah, I., Mirjalili, S., & Abraham, A. (2018). Evolutionary static and dynamic clustering algorithms based on multi-verse optimizer. Engineering Applications of Artificial Intelligence, 72, 54–66.

    Article  Google Scholar 

  • Solti, A., Raffel, M., Romagnoli, G., & Mendling, J. (2018). Misplaced product detection using sensor data without planograms. Decision Support Systems, 112, 76–87.

    Article  Google Scholar 

  • Song, S., **ao, J., (2013). Tracking revisited using RGBD camera: unified benchmark and baselines, in: Proceedings of the IEEE conference on computer vision.

  • Sun, D., Sudderth, E.B., Pfister, H., (2015). Layered RGBD scene flow estimation, in: Proceedings of the IEEE conference on computer vision and pattern Recognition.

  • Swanson, E. B. (2020). How information systems came to rule the world: Reflections on the information systems field. The Information Society, 36, 109–123.

    Article  Google Scholar 

  • Tan, A. J. Y., Ng, S. M., Stoddart, P. R., & Chua, H. S. (2021). Trends and applications of u-shaped fiber optic sensors: A review. IEEE Sensors Journal, 21, 120–131.

    Article  Google Scholar 

  • Taylor, S. J. (2019). Distributed simulation: State-of-the-art and potential for operational research. European Journal of Operational Research, 273, 1–19.

    Article  Google Scholar 

  • Tsionas, M. G. (2023). Clustering and meta-envelopment in data envelopment analysis. European Journal of Operational Research, 304, 763–778.

    Article  Google Scholar 

  • Ullmann, I., Guendel, R. G., Kruse, N. C., Fioranelli, F., & Yarovoy, A. (2023). A survey on radar-based continuous human activity recognition. IEEE Journal of Microwaves. https://doi.org/10.1109/JMW.2023.3264494

    Article  Google Scholar 

  • van Staden, H. E., & Boute, R. N. (2021). The effect of multi-sensor data on condition-based maintenance policies. European Journal of Operational Research, 290, 585–600.

    Article  Google Scholar 

  • Wang, L., Meng, W., **, R., Zhang, Y., Ma, C., Lu, L., Zhang, X., (2019). 3D point cloud analysis and classification in large-scale scene based on deep learning. IEEE Access.

  • Wu, Y., Wang, Y., Zhang, S., & Ogai, H. (2021). Deep 3d object detection networks using lidar data: A review. IEEE Sensors Journal, 21, 1152–1171.

    Article  Google Scholar 

  • **ao, J., Stolkin, R., Gao, Y., & Leonardis, A. (2018). Robust fusion of color and depth data for rgb-d target tracking using adaptive range-invariant depth models and spatio-temporal consistency constraints. IEEE Transactions on Cybernetics, 48, 2485–2499.

    Article  Google Scholar 

  • Yue, C., Wang, Y., Tang, X., & Chen, Q. (2022). Drgcnn: Dynamic region graph convolutional neural network for point clouds. Expert Systems with Applications, 205, 117663.

    Article  Google Scholar 

  • Zhang, S., Lu, W., **ng, W., & Zhang, L. (2018). Using fuzzy least squares support vector machine with metric learning for object tracking. Pattern Recognition, 84, 112–125.

    Article  Google Scholar 

Download references

Acknowledgements

We would like to extend our sincere gratitude to Professor W. Brent Lindquist for organizing the Mathematics & Statistics seminars at Texas Tech University, USA. We are immensely grateful to all the participants for their valuable time, thoughtful reviews, and constructive feedback on our paper. Their insightful comments have significantly enriched our research. Additionally, we would like to express our heartfelt appreciation to the two guest editors for their invaluable feedback and insightful comments, which have greatly contributed to improving the quality of our work.

Funding

This work was supported in part by the Center for Open Intelligent Connectivity from The Featured Areas Research Center Program within the framework of the Higher Education Sprout Project by the Ministry of Education, National Science and Technology Council (NSTC 110-2622-8-A49-022, 112-2221-E-A49-049), and NCKU Miin Wu School of Computing in Taiwan and supported by InfoTech from 2018 to 2021 under USt-IdNr. DE320245686 in Germany.

Author information

Authors and Affiliations

  1. Montpellier Business School, Montpellier, France

    Claire Y. T. Chen

  2. MRM, University of Montpellier, Montpellier, France

    Claire Y. T. Chen

  3. KEDGE Business School, Talence, France

    Edward W. Sun

  4. College of AI, National Yang Ming Chiao Tung University, Hsinchu, Taiwan

    Yi-Bing Lin

  5. Miin Wu School of Computing, National Cheng Kung University, Tainan City, Taiwan

    Yi-Bing Lin

  6. China Medical University Hospital, Taichung, Taiwan

    Yi-Bing Lin

Authors
  1. Claire Y. T. Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Edward W. Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yi-Bing Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Edward W. Sun.

Ethics declarations

Conflict of interest

Authors declare that they have no conflict of interest.

Human or animal rights

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, C.Y.T., Sun, E.W. & Lin, YB. Reliable information system for identifying spatio-temporal continuity of kinetic deformed objects with big point cloud data. Ann Oper Res (2023). https://doi.org/10.1007/s10479-023-05522-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10479-023-05522-z

Keywords

Search

Navigation