SpringerLink
Log in

A multiple-UAV architecture for autonomous media production

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Cinematography with Unmanned Aerial Vehicles (UAVs) is an emerging technology promising to revolutionize media production. On the one hand, manually controlled drones already provide advantages, such as flexible shot setup, opportunities for novel shot types and access to difficult-to-reach spaces and/or viewpoints. Moreover, little additional ground infrastructure is required. On the other hand, enhanced UAV cognitive autonomy would allow both easier cinematography planning (from the Director’s perspective) and safer execution of that plan during actual filming; while integrating multiple UAVs can additionally augment the cinematic potential. In this paper, a novel multiple-UAV software/hardware architecture for media production in outdoor settings is proposed. The architecture encompasses mission planning and control under safety constraints, enhanced cognitive autonomy through visual analysis, human-computer interfaces and communication infrastructure for platform scalability with Quality-of-Service provisions. Finally, the architecture is demonstrated via a relevant subjective study on the adequacy of UAV and camera parameters for different cinematography shot types, as well as with field experiments where multiple UAVs film outdoor sports events.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. https://multidrone.eu/

  2. https://aiia.csd.auth.gr/LAB_PROJECTS/MULTIDRONE/AUTH_MULTIDRONE_Software.html

  3. http://poseidon.csd.auth.gr/LAB_PROJECTS/MULTIDRONE/AUTH_MULTIDRONE_Dataset.html

References

  1. Alcantara A, Capitan J, Cunha R, Ollero A (2021) Optimal trajectory planning for cinematography with multiple unmanned aerial vehicles. Robot Auton Syst 140:103778. https://doi.org/10.1016/j.robot.2021.103778

    Article  Google Scholar 

  2. Alcantara A, Capitan J, Torres-Gonzalez A, Cunha R, Ollero A (2020) Autonomous execution of cinematographic shots with multiple drones. IEEE Access, 201300–201316. https://doi.org/10.1109/ACCESS.2020.3036239, https://ieeexplore.ieee.org/document/9249238/

  3. Arev I, Park H S, Sheikh Y, Hodgins J K, Shamir A (2014) Automatic editing of footage from multiple social cameras. ACM Trans Graph 4 (33):81:1–81:11

    Google Scholar 

  4. Bonatti R, Ho C, Wang W, Choudhury S, Scherer S (2019) Towards a robust aerial cinematography platform: localizing and tracking moving targets in unstructured environments. In: International conference on intelligent robots and systems (IROS), pp 229–236

  5. Bonatti R, Wang W, Ho C, Ahuja A, Gschwindt M, Camci E, Kayacan E, Choudhury S, Scherer S (2020) Autonomous aerial cinematography in unstructured environments with learned artistic decision-making. J Field Robot 37(4):606–641. https://doi.org/10.1002/rob.21931

    Article  Google Scholar 

  6. Bucker A, Bonatti R, Scherer S (2020) Do you see what i see? Coordinating multiple aerial cameras for robot cinematography. Ar**v:https://arxiv.org/abs/2011.05437

  7. Caraballo L E, Montes-Romero A, Diaz-Bañez J M, Capitan J, Torres-Gonzalez A, Ollero A (2020) Autonomous planning for multiple aerial cinematographers. In: International conference on intelligent robots and systems (IROS), Las Vegas

  8. Cunha R, Malaca M, Sampaio V, Guerreiro B, Nousi P, Mademlis I, Tefas A, Pitas I (2019) Gimbal control for vision-based target tracking. In: EUSIPCO. Satellite workshop on signal processing, computer vision and deep learning for autonomous systems, La Coruña

  9. Daniyal F, Cavallaro A (2011) Multi-camera scheduling for video production. In: European conference on visual media production (CVMP)

  10. DJI (2020) DJI products, https://www.dji.com/

  11. FreeSkies (2020) FreeSkies CoPilot, https://freeskies.pr.co/

  12. Galvane Q, Lino C, Christie M, Fleureau J, Servant F, Tariolle F-, Guillotel P (July 2018) Directing cinematographic drones. ACM Trans Graph 37:3. https://doi.org/10.1145/3181975

    Article  Google Scholar 

  13. Gandhi V, Ronfard R (2015) A computational framework for vertical video editing. In: Proceedings of the eurographics workshop on intelligent cinematography and editing (WICED). Eurographics

  14. Garage A (2020) Skywand, https://skywand.com/

  15. Gebhardt C, Hepp B, Nägeli T, Stevšić S, Hilliges O (2016) Airways: optimization-based planning of quadrotor trajectories according to high-level user goals. In: Proceedings of the conference on human factors in computing systems (CHI), New York

  16. Gschwindt M, Camci E, Bonatti R, Wang W, Kayacan E, Scherer S (2019) Can a robot become a movie director? Learning artistic principles for aerial cinematography. In: IEEE IROS. https://doi.org/10.1109/IROS40897.2019.8967592, pp 1107–1114

  17. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)

  18. Hornung A, Wurm K M, Bennewitz M, Stachniss C, Burgard W (2013) OctoMap: an efficient probabilistic 3d map** framework based on octrees. Auton Robot 34(3):189–206

    Article  Google Scholar 

  19. Huang C, Yang Z, Kong Y, Chen P, Yang X, Cheng K-TT (2019) Learning to capture a film-look video with a camera drone. In: IEEE ICRA. https://doi.org/10.1109/ICRA.2019.8793915, pp 1871–1877

  20. Joubert N, E J L, Goldman D B, Berthouzoz F, Roberts M, Landay J A, Hanrahan P (2016) Towards a drone cinematographer: guiding quadrotor cameras using visual composition principles. Ar**v:1610.01691

  21. Kakaletsis E, Mademlis I, Nikolaidis N, Pitas I (2020) Bayesian fusion of multiview human crowd detections for autonomous UAV fleet safety. In: Proceedings of the European signal processing conference (EUSIPCO). IEEE

  22. Kakaletsis E, Tzelepi M, Kaplanoglou P I, Symeonidis C, Nikolaidis N, Tefas A, Pitas I (2019) Semantic map annotation through UAV video analysis using deep learning models in ROS. In: Proceedings of the international conference on multimedia modeling (MMM). Springer

  23. Karakostas I, Mademlis I, Nikolaidis N, Pitas I (2018) UAV cinematography constraints imposed by visual target tracking. In: Proceedings of the IEEE international conference on image processing (ICIP)

  24. Karakostas I, Mademlis I, Nikolaidis N, Pitas I (2019) Shot type constraints in UAV cinematography for target tracking applications. Inf Sci 506:273–294

    Article  Google Scholar 

  25. Karakostas I, Mademlis I, Nikolaidis N, Pitas I (2019) Shot type feasibility in autonomous UAV cinematography. In: IEEE International conference on acoustics, speech and signal processing (ICASSP)

  26. Li B, Yan J, Wu W, Zhu Z, Hu X (2018) High-performance visual tracking with siamese region proposal network. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)

  27. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C-Y, Berg A C (2016) SSD: single-shot multibox detector. In: Proceedings of the European conference on computer vision (ECCV). Springer

  28. Mademlis I, Mygdalis V, Nikolaidis N, Montagnuolo M, Negro F, Messina A, Pitas I (2019) High-level multiple-UAV cinematography tools for covering outdoor events. IEEE Trans Broadcast 65(3):627–635

    Article  Google Scholar 

  29. Mademlis I, Mygdalis V, Nikolaidis N, Pitas I (2018) Challenges in autonomous UAV cinematography: an overview. In: Proceedings of the IEEE international conference on multimedia and expo (ICME)

  30. Mademlis I, Nikolaidis N, Tefas A, Pitas I, Wagner T, Messina A (2018) Autonomous unmanned aerial vehicles filming in dynamic unstructured outdoor environments. IEEE Signal Proc Mag 36:147–153

    Article  Google Scholar 

  31. Mademlis I, Nikolaidis N, Tefas A, Pitas I, Wagner T, Messina A (2019) Autonomous UAV cinematography: a tutorial and a formalized shot type taxonomy. ACM Comput Surv 52(5):105:1–105:33

    Google Scholar 

  32. Mercer Moss F, Wang K, Zhang F, Baddeley R, Bull D R (2016) On the optimal presentation duration for subjective video quality assessment. IEEE Trans Circ Syst Video Technol 26(11):1977–1987. https://doi.org/10.1109/TCSVT.2015.2461971

    Article  Google Scholar 

  33. Mercer Moss F, Yeh C-T, Zhang F, Baddeley R, Bull D R (2016) Support for reduced presentation durations in subjective video quality assessment. Signal Process Image Commun 48:38–49

    Article  Google Scholar 

  34. Messina A, Metta S, Montagnuolo M, Negro F, Mygdalis V, Pitas I, Capitán J, Torres A, Boyle S, Bull D (2018) The future of media production through multi-drones’ eyes. In: International broadcasting convention (IBC)

  35. Montes-Romero A, Torres-González A, Capitán J, Montagnuolo M, Metta S, Negro F, Messina A, Ollero A (2020) Director tools for autonomous media production with a team of drones. Appl Sci 10:4. https://doi.org/10.3390/app10041494, https://www.mdpi.com/2076-3417/10/4/1494

    Article  Google Scholar 

  36. MULTIDRONE-Consortium (2017) D2.1: Multidrone media production requirements, https://multidrone.eu/wp-content/uploads/2017/01/MultiDrone_D2.1_Multidrone-media-production-requirements.pdf

  37. Nägeli T, Alonso-Mora J, Domahidi A, Rus D, Hilliges O (2017) Real-time motion planning for aerial videography with dynamic obstacle avoidance and viewpoint optimization. IEEE Robot Autom Lett 2(3):1696–1703

    Article  Google Scholar 

  38. Nägeli T, Meier L, Domahidi A, Alonso-Mora J, Hilliges O (2017) Real-time planning for automated multi-view drone cinematography. ACM Trans Graph 36(4):1–10. https://doi.org/10.1145/3072959.3073712, http://dl.acm.org/citation.cfm?doid=3072959.3073712

    Article  Google Scholar 

  39. Nousi P, Mademlis I, Karakostas I, Tefas A, Pitas I (2019) Embedded UAV real-time visual object detection and tracking. In: Proceedings of the IEEE international conference on real-time computing and robotics (RCAR)

  40. Nousi P, Patsiouras E, Tefas A, Pitas I (2018) Convolutional neural networks for visual information analysis with limited computing resources. In: Proceedings of the IEEE international conference on image processing (ICIP)

  41. Nousi P, Triantafyllidou D, Tefas A, Pitas I (2020) Re-identification framework for long-term visual object tracking based on object detection and classification. Signal Process Image Commun 88:115969

    Article  Google Scholar 

  42. Papaioannidis C, Mademlis I, Pitas I (2021) Autonomous UAV safety by visual human crowd detection using multi-task deep neural networks. In: Proceedings of the IEEE international conference on robotics and automation (ICRA)

  43. Passalis N, Tefas A (2019) Deep reinforcement learning for controlling frontal person close-up shooting. Neurocomputing 335:37–47

    Article  Google Scholar 

  44. Patrona F, Nousi P, Mademlis I, Tefas A, Pitas I (2020) Visual object detection for autonomous UAV cinematography. In: Proceedings of the northern lights deep learning workshop

  45. Real F, Torres-González A, Soria P R, Capitán J, Ollero A (2020) Unmanned aerial vehicle abstraction layer: An abstraction layer to operate unmanned aerial vehicles. Int J Adv Robot Syst 17(4):1–13. https://doi.org/10.1177/1729881420925011

    Article  Google Scholar 

  46. Recommendation ITU-R BT.500-11 (2002) Methodology for the subjective assessment of the quality of television pictures. Geneva

  47. Redmon J, Farhadi A (2017) YOLO9000: better, faster, stronger. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)

  48. Saeed A, Abdelkader A, Khan M, Neishaboori A, Harras K A, Mohamed A (2017) On realistic target coverage by autonomous drones. ar**v:1702.03456

  49. Skydio (2020) Skydio products, https://www.skydio.com/

  50. Symeonidis C, Mademlis I, Nikolaidis N, Pitas I (2019) Improving neural non-maximum suppression for object detection by exploiting interest-point detectors. In: Proceedings of IEEE international workshop on machine learning for signal processing (MLSP)

  51. Tzelepi M, Tefas A (2017) Human crowd detection for drone flight safety using Convolutional Neural Networks. In: Proceedings of the EURASIP signal processing conference (EUSIPCO). IEEE

  52. Yuneec Yuneec products. http://us.yuneec.com

  53. Zhu Z, Wang Q, Li B, Wu W, Yan J, Hu W (2018) Distractor-aware siamese networks for visual object tracking. In: Proceedings of the European conference on computer vision (ECCV). Springer

Download references

Acknowledgments

The authors would like to thank Prof. Anibal Ollero (Head of GRVC, University of Seville) for his integral scientific contribution to the MULTIDRONE project and architecture.

Funding

The research leading to these results has received funding from the European Union’s European Union Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE). This publication reflects only the author’s views. The European Union is not liable for any use that may be made of the information contained therein.

Author information

Authors and Affiliations

  1. Department of Informatics, Aristotle University of Thessaloniki, Thessaloniki, Greece

    Ioannis Mademlis, Anastasios Tefas, Nikos Nikolaidis & Ioannis Pitas

  2. GRVC Robotics Lab, University of Seville, Seville, Spain

    Arturo Torres-González & Jesús Capitán

  3. RAI - Centre for Research and Technological Innovation, Torino, Italy

    Maurizio Montagnuolo, Alberto Messina & Fulvio Negro

  4. Thales - Advanced Studies Department THERESIS, Palaiseau, France

    Cedric Le Barz & Tiago Gonçalves

  5. Institute for Systems and Robotics (ISR/LARSyS), Instituto Superior Tecnico, Lisbon, Portugal

    Rita Cunha & Bruno Guerreiro

  6. Department of Electrical and Electronic Engineering, University of Bristol, Bristol, UK

    Fan Zhang, Stephen Boyle & David Bull

  7. Alerion, Nancy, France

    Gregoire Guerout

Authors
  1. Ioannis Mademlis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Arturo Torres-González
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jesús Capitán
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Maurizio Montagnuolo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alberto Messina
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Fulvio Negro
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Cedric Le Barz
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Tiago Gonçalves
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Rita Cunha
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Bruno Guerreiro
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Fan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Stephen Boyle
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Gregoire Guerout
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Anastasios Tefas
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Nikos Nikolaidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. David Bull
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Ioannis Pitas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ioannis Mademlis.

Ethics declarations

Competing interests

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE). This publication reflects the authors’ views only. The European Commission is not responsible for any use that may be made of the information it contains.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mademlis, I., Torres-González, A., Capitán, J. et al. A multiple-UAV architecture for autonomous media production. Multimed Tools Appl 82, 1905–1934 (2023). https://doi.org/10.1007/s11042-022-13319-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13319-8

Keywords

Search

Navigation