SpringerLink
Log in

Urban tree generator: spatio-temporal and generative deep learning for urban tree localization and modeling

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

We present a vision-based algorithm that uses spatio-temporal satellite imagery, pattern recognition, procedural modeling, and deep learning to perform tree localization in urban settings. Our method resolves two primary challenges. First, automated city-scale tree localization at high accuracy typically requires significant acquisition/user intervention. Second, vegetation-index segmentation methods from satellites require manual thresholding, which varies across geographic areas, and are not robust across cities with varying terrain, geometry, altitude, and canopy. In our work, we compensate for the lack of visual detail by using satellite snapshots across twelve months and segment cities into various vegetation clusters. Then, we use multiple GAN-based networks to plant trees by recognizing placement patterns inside segmented regions procedurally. We present comprehensive experiments over four cities (Chicago, Austin, Indianapolis, Lagos), achieving tree count accuracies of 87–97%. Finally, we show that the knowledge accumulated from each model (trained on a particular city) can be transferred to a different city.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Canada)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., Kim, B.: Sanity checks for saliency maps. In: NeurIPS, NIPS’18, pp. 9525–9536 (2018)

  2. Aliaga, D.G., Vanegas, C., Lei, M., Niyogi, D.: Visualization-based decision tool for urban meteorological modeling. Environ. Plann. B. Plann. Des. 40(2), 271–288 (2013)

    Article  Google Scholar 

  3. Arief, H.A., Strand, G.H., Tveite, H., Indahl, U.G.: Land cover segmentation of airborne lidar data using stochastic atrous network. Remote Sens. 10(6), (2018)

  4. Benes, B., Massih, M.A., Jarvis, P., Aliaga, D.G., Vanegas, C.A.: Urban ecosystem design. In: Symposium on Interactive 3D Graphics and Games, I3D ’11, pp. 167–174. Association for Computing Machinery (2011)

  5. Bovik, A.: A visual information fidelity approach to video quality assesment (2005)

  6. Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: ECCV (2018)

  7. Chicago Department of Transportation: Street and site plan design standards (2007). https://www.chicago.gov/dam/city/depts/cdot/StreetandSitePlanDesignStandards407.pdf

  8. City of Austin Texas: Downtown Tree Inventory (2013). https://data.austintexas.gov/Locations-and-Maps/Downtown-Tree-Inventory-2013

  9. City of Indianapolis/Marion, Indiana: Code of ordinances—chapter 701—trees and flora (2021). https://library.municode.com/in/indianapolis-marion_county/codes/code_of_ordinances

  10. City of New York: NYC open data map tiles (2021). https://maps.nyc.gov/tiles

  11. Earth Observing System (EOS), NASA: The Ice, Cloud, and Land Elevation Satellite-2 (2021). https://icesat-2.gsfc.nasa.gov/icesat-2-data

  12. Food and Agriculture Organization of the United Nations: Global forest resources assessment country report series—Nigeria. Tech. rep, United Nations (2010)

  13. Guo, J., Jiang, H., Benes, B., Deussen, O., Zhang, X., Lischinski, D., Huang, H.: Inverse procedural modeling of branching structures by inferring l-systems. ACM Trans. Graph. 39(5) (2020)

  14. Hahsler, M., Piekenbrock, M., Doran, D.: dbscan: Fast density-based clustering with R. J. Stat. Softw. 91(1), 1–30 (2019)

    Article  Google Scholar 

  15. Hojas-Gascon, L., Eva, H.: Field guide for forest map** with high resolution satellite data. monitoring deforestation and forest degradation in the context of the un-redd programme. the tanzania redd + initiative (2014). https://doi.org/10.2788/657954

  16. Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., Le, Q.V., Adam, H.: Searching for mobilenetv3. In: ICCV (2019)

  17. Hu, Q., Yang, B., Khalid, S., **ao, W., Trigoni, N., Markham, A.: Towards semantic segmentation of urban-scale 3d point clouds: a dataset, benchmarks and challenges (2020). https://doi.org/10.48550/ARXIV.2009.03137

  18. Huang, S., Tang, L., Hupy, J., Wang, Y., Shao, G.: A commentary review on the use of normalized difference vegetation index (ndvi) in the era of popular remote sensing. J. For. Res. 32 (2020)

  19. Huang, Z., Arian, A., Yuan, Y., Chiu, Y.C.: Using conditional generative adversarial nets and heat maps with simulation-accelerated training to predict the spatiotemporal impacts of highway incidents. Transp. Res. Rec. 2674(8), 836–849 (2020)

    Article  Google Scholar 

  20. Inc., G.: Google Earth (2021). http://earth.google.com

  21. Japan Aerospace Exploration Agency—JAXA: JAXA (2021). https://global.jaxa.jp

  22. Jiang, Z., Huete, A., Chen, J., Chen, Y., Li, J., Yan, G., Zou, Y.: Analysis of ndvi and scaled difference vegetation index retrievals of vegetation fraction. Remote Sens. Environ. 101, 366–378 (2006)

    Article  Google Scholar 

  23. Keren, S.: Modeling tree species count data in the understory and canopy layer of two mixed old-growth forests in the dinaric region. Forests 11(5) (2020)

  24. Kim, J.S., Kavak, H., Crooks, A.: Procedural city generation beyond game development. SIGSPATIAL Special 10(2), 34–41 (2018)

    Article  Google Scholar 

  25. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NeurIPS, vol. 25 (2012)

  26. Lee, S., Park, S., Son, S., Han, J., Kim, S., Kim, J.: Land cover segmentation of aerial imagery using segnet. In: Earth Resources and Environmental Remote Sensing/GIS Applications X, vol. 11156, pp. 313–318. Intl. Society for Optics and Photonics, SPIE (2019)

  27. Li, B., Kałużny, J., Klein, J., Michels, D.L., Pałubicki, W., Benes, B., Pirk, S.: Learning to reconstruct botanical trees from single images. ACM Trans. Graph. 40(6) (2021)

  28. Lian, D., Li, J., Zheng, J., Luo, W., Gao, S.: Density map regression guided detection network for rgb-d crowd counting and localization. In: CVPR, pp. 1821–1830 (2019). https://doi.org/10.1109/CVPR.2019.00192

  29. Liu, L., Chen, J., Wu, H., Li, G., Li, C., Lin, L.: Cross-modal collaborative representation learning and a large-scale rgbt benchmark for crowd counting. In: CVPR, pp. 4823–4833 (2021)

  30. Liu, Y., Guo, J., Benes, B., Deussen, O., Zhang, X., Huang, H.: Treepartnet: Neural decomposition of point clouds for 3d tree reconstruction. ACM Trans. Graph. 40(6) (2021)

  31. Lu, D., Weng, Q.: A survey of image classification methods and techniques for improving classification performance. Int. J. Remote Sens. 28(5), 823–70 (2007)

    Article  Google Scholar 

  32. Maggiori, E., Tarabalka, Y., Charpiat, G., Alliez, P.: The inria aerial image labeling benchmark. In: 2017 IEEE Intl. Geoscience and Remote Sensing Symposium (IGARSS) (2017)

  33. Martinovic, A., Van Gool, L.: Bayesian grammar learning for inverse procedural modeling. In: CVPR, pp. 201–208 (2013)

  34. McBride, J.: Map** chicago area urban tree canopy using color infrared imagery. Ph.D. thesis, Lund University, Lund, Sweden (2011)

  35. Musialski, P., Wonka, P., Aliaga, D.G., Wimmer, M., Van Gool, L., Purgathofer, W.: A survey of urban reconstruction. In: Computer Graphics Forum, vol. 32, pp. 146–177. Wiley Online Library (2013)

  36. Niese, T., Pirk, S., Albrecht, M., Benes, B., Deussen, O.: Procedural urban forestry. ACM Trans. Graph. 41(2) (2022)

  37. Nowak, D., Bodine, A.R., Hoehn, R.E., Edgar, C., Hartel, D., Lister, T., Brandeis, T.: Austin’s urban forest (2016)

  38. Nowak, D., Heisler, N.: Air quality effects of urban trees and parks. Research Series Monograph. National Recreation and Parks Association Research Series Monograph, Ashburn, VA (2010). https://www.fs.usda.gov/treesearch/pubs/52881

  39. Open Street Map Contributors: Planet Dump (2021). https://planet.osm.org. https://www.openstreetmap.org

  40. Paula, P., Gregory, M., James, S., Kelaine, V., Qingfu, X.: City of Indianapolis, Indiana Municipal Forest Resource Analysis. Tech. rep. Center for Urban Forest Research USDA Forest Service (2008)

  41. Planet: Planet Explorer (2021). http://planet.com/explorer

  42. Potapov, P., Li, X., Hernandez-Serna, A., Tyukavina, A., Hansen, M., Kommareddy, A., Pickens, A., Turubanova, S., Tang, H., Silva, C.E., Armston, J., Dubayah, R., Blair, J.B., Hofton, M.: Map** global forest canopy height through integration of gedi and landsat data. Remote Sens. Environ. 253, 112–165 (2021)

    Article  Google Scholar 

  43. Pretzsch, H., Biber, P., Uhl, E., Dahlhausen, J., Rötzer, T., Caldentey, J., Koike, T., van Con, T., Chavanne, A., Seifert, T., du Toit, B., Farnden, C., Pauleit, S.: Crown size and growing space requirement of common tree species in urban centres, parks, and forests. Urban For. Urban Greening 14(3), 466–479 (2015)

    Article  Google Scholar 

  44. Prusinkiewicz, P., Lindenmayer, A.: The Algorithmic Beauty of Plants. Springer-Verlag, New York (1990)

    Book  Google Scholar 

  45. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI 2015, pp. 234–241. Cham (2015)

  46. Rußwurm, M., Körner, M.: Temporal vegetation modelling using long short-term memory networks for crop identification from medium-resolution multi-spectral satellite images. In: CVPR, pp. 1496–1504 (2017)

  47. Shen, B., Fang, S., Li, G.: Vegetation coverage changes and their response to meteorological variables from 2000 to 2009 in Naqu, Tibet, China. Can. J. Remote Sens. 40, 67–74 (2014)

    Article  Google Scholar 

  48. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. ar**v:1409.1556 (2014)

  49. Tinchev, G., Nobili, S., Fallon, M.: Seeing the Wood for the Trees: Reliable Localization in Urban and Natural Environments (2018). https://doi.org/10.48550/ARXIV.1809.02846

  50. Tree Care Industry Association (TCIA): ANSI A300 Standards: Tree, Shrub and Other Woody Plant Management—Standard Practices. American National Standards Institute (2017). https://www.tcia.org/TCIA/Build_Your_Business/A300_Standards/A300_Standards.aspx

  51. Tree counting with high spatial-resolution satellite imagery based on deep neural networks. Ecol. Ind. 125, 107591 (2021)

  52. Urban Design Division/Planning & Development Review Department: City of austin great streets (2012)

  53. USDA Forest Service: iTree (2020). https://www.itreetools.org

  54. Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Img. Proc. 13(4), 600–612 (2004)

    Article  Google Scholar 

  55. Wang, Z., Simoncelli, E., Bovik, A.: Multiscale structural similarity for image quality assessment. Conf. Record Asilomar Conf. Signals Syst. Comp. 2 (2003)

  56. Weier, J., Herring, D.: Measuring vegetation (ndvi and evi (2000). https://earthobservatory.nasa.gov/features/MeasuringVegetation

  57. Yang, S.D., Su, H.T., Hsu, W.H., Chen, W.C.: Class-agnostic few-shot object counting. In: IEEE/CVF WACV, pp. 870–878 (2021)

  58. Zabelskyte, G., Kabisch, N., Stasiskiene, Z.: Patterns of urban green space use applying social media data: a systematic literature review. Land 11(2) (2022)

  59. Zhang, L., Zhang, L., Mou, X., Zhang, D.: Fsim: A feature similarity index for image quality assessment. IEEE Trans. Img. Proc. 20(8) (2011)

  60. Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: CVPR, pp. 6230–6239 (2017)

Download references

Acknowledgements

This research was funded in part by National Science Foundation grant #10001387, Functional Proceduralization of 3D Geometric Models and by the Foundation for Food and Agriculture Research Grant ID: 602757. We thank the Integrated Digital Forestry Initiative (iDIF) at Purdue University for their partial support. We also thank NSF grant #1835739 “U-Cube: A Cyberinfrastructure for Unified and Ubiquitous Urban Canopy Parameterization” and NSF grant #2106717 “Deep Generative Modeling for Urban and Archaeological Recovery.”

Author information

Authors and Affiliations

  1. Department of Computer Science, Purdue University, 305 N. University St, West Lafayette, IN, 47907, USA

    Adnan Firoze, Bedrich Benes & Daniel Aliaga

Authors
  1. Adnan Firoze
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bedrich Benes
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daniel Aliaga
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adnan Firoze.

Ethics declarations

Conflict of interest

The authors claim and announce no conflict of interest with any entity or party in relation to this work.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Firoze, A., Benes, B. & Aliaga, D. Urban tree generator: spatio-temporal and generative deep learning for urban tree localization and modeling. Vis Comput 38, 3327–3339 (2022). https://doi.org/10.1007/s00371-022-02526-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02526-x

Keywords

Search

Navigation