1 Introduction

The Vibrant Fields project endeavors to construct novel representational tectonics of urbanization aimed at comprehending the materiality of forms and the genesis of multispectral relations as the outcome of a complex information system governing the interplay of atmospheric and bodily embedded energy cycles within the biosphere. Vibrant Fields takes to the fore, that any (experienceable) condition of nature is an expression of the interplay of dynamic processes limited to grasp by the sensitivity of the human senses.

The urban environment is epitomized by a systemic model rooted in theoretical biology. The concept of modularity (Callebaut, 2005) posits the biosphere as a dynamic interplay of chemical and physical elements, engaged in information exchange within the boundaries of ecological systems and subsystems. These systems and their interconnected subsystems operate at the intersection of biology and technology, influenced by geographic and atmospheric conditions. Consequently, the city, much like any intricate life form, grapples with the challenge of achieving a cohesive structure amid the multitude of vibrant (Bennett, 2010) objectives and multiplicities it encompasses.

Sourcing from this various interrelated information layers, a research paper led by complexity scientist Luis M. A. Bettencourt on physical laws of quantities and qualities of urban growth (Bettencourt et al. 2007; Balzar 2021) suggests and argues that Cities are consumers of energy and resources and producers of artifacts, information, and waste which have often been compared with biological entities. The findings by the research team at the Santa Fe institute fall in line with the earlier findings in the 1960ies by the regional German Geographer Walter Christaller. His research of emerging functional patterns of infrastructures at cities became known as Central Place Theory (Christaller 2009; van Meeteren and Poorthuis 2018). Accordingly, various physiological states of the urban environment are results of an interplay of complex interrelated ecological and social systems. Ecological and energetic optimization and efficiency (Bejan and Zane 2012) are embedded strategies of this `geochemical energy´ flow also researched by environmental system scientist Howard T. Odum. Climate change is according to the systems ecologist Odum an expression of an imbalance within the earth flow of matter and energy governed by the second law of thermodynamics – therefore a constant change in element ratios (Odum, 2007).

There are multiple layers of observation systems that build information networks translating the enormous complex biosphere, geosphere, hydrosphere, cryosphere, and atmosphere (Society, 2022) into various technical data streams. These layers of divergent realities separate the continuous flux into dimensionally reduced relations.

Innumerous Satellites oversee the global changes of elements - the physical-chemical processes as well as particles and substances that contribute to global climate - in various geolocations circling rhythmically around the globe. The global system of the earth is therefore wrapped into the different satellite paths sensing highly precise information at the macroscale, mostly with a pixel resolution between 0,25 km2 - 4,0 km2. Vibrant Fields therefore developed a set of flexible tools as a complementary family of devices bringing together additional sets of yet non existing data (3D point cloud of the urban environment, thermal camera, various gases not reflected in satellite datasets, …) that try to fill the gap between global data sets and local datasets to gain a better understanding of small-scale conditions of microclimatic phenomena (Balzar, Aksöz Balzar, & Moncayo Asan, OTTO, 2023). With these devices called OTTO and ANA, we have developed a methodology for disentangling the various layers of information that collectively constitute what we commonly refer to as the "environment." These devices allow us to collect and analyse local data within a 100-meter diameter area. Unlike satellite recordings, which offer a global perspective over a brief timeframe, our recordings provide a detailed view of the immediate environmental dynamics in real-time over the course of a typical human day, allowing us to uncover intricate patterns and correlations.

The goal of Vibrant Fields is to investigate the multiple simultaneous realities and temporalities of information of an urban field in relation to its ecological milieu. It is an investigation and observation of a city’s architecture, technology, flora and fauna in relation to their geographic, geologic, and ecologic condition (Balzar & Aksöz Balzar, Narrating Urban DATA, 2023). The core of this investigation is conducted through the analysis of representational techniques for data correlations. These are then transformed into immersive experiences designed to depict the shifting energy dynamics within the urban space, presenting them in an abstract yet more comprehensible manner.

In addition to multi-scalar levels of effects of climate change and related energetic imbalances, another level of complexity is added through the multiple temporalities of observation. Different effects of climate change and climate interrelations can be observed in different overlap** synchronous time periods. As the Russian geologist, mineralogist and geochemist Vladimir Vernadsky states, these multiple temporalities can be observed at the historical time frame and at the geological time frame. As the geological scale adapts its domain to changes of earth’s planetary system covering a larger period, the historical time domain is adapted to the changes of human life (Vernardsky, 1997).

In this different systemic analyzing periods, it is hard to follow the local shifts of relations in accordance with the global patterns it leads to. At the same time, local changes within the immediate environment, are not directly expressed through the outbursts of local or global phenomena.

Locally, the pre-existing meteorological network relies on a handful of strategically positioned weather stations within the city's boundaries. To grant an overview to the specific urban condition under active research and observation, an aforementioned set of 2 different devices has been developed. The “stationary” device OTTO (Orthographic Trans Territorial Operator) (Fig. 1) (Balzar, Aksöz Balzar, & Moncayo Asan, OTTO, 2023) and the portable device ANA (Analytic Navigating Apparatus) (Fig. 2). Both devices together build a network of sensing points with a circular span of max. 100m in diameter to cover different zones within the realm of observation. In this regard the diverse zones at one particular urban location can be covered by their sensible differences.

Fig. 1
figure 1

OTTO (Orthographic Trans Territorial Operator) by Vibrant Fields

Full size image
Fig. 2
figure 2

ANA (Analytic Navigating Apparatus) by Vibrant Fields

Full size image

The focus of Vibrant Fields lies on urban environments as well as its infrastructures and organizations within the urban periphery and rural landscape. Instead of observing these systems as isolated closed aggregations, the project focuses on holistic perspectives using a metabolism-oriented approach of materialized energy cycles. This research employs classical analytical methodologies to examine extensive datasets of existing atmospheric data at the macroscale, which are publicly accessible through platforms like Google Earth Engine and QGIS. The primary objective lies in discovering innovative means of artistic output and expression to effectively convey both local and global data, thereby contributing to the dynamic and evolving realms of these vibrant fields.

Historically, the influence of climatic conditions had various effects on population and population density. Geologists observed a direct relationship between the world distribution of urban sprawl and the respected isotherms. Isotherms delineate layers or regions offset by 2 degrees Celsius across the globe, characterizing the average temperature during a specific time period. These isotherms also correspond with various ecosystems and wildlife habitats. While isotherms were originally discovered by Alexander von Humboldt, the historical Russian geographer Lev Mechnikov found out that those layers were also linked to the process of urban sprawl during the 19th century (Reclus, 2013). In the early 20th Century Geologist Vladimir Vernadsky approached the terminology of the “biosphere” (Suess, 1875), considering it as a planetary system of multiplicities of interconnected, interrelated, and interdependent events. It is part of Vibrant Fields research argument, that these isotherms are constantly moving and put tension on the current state of urbanization. Human society can only put two strategic logics into the process: Moving with the isotherm or adapting to the new climatic conditions.

Vibrant Fields is an operational investigation and observation of physical and chemical traces of architecture, technology, animal species and plants in relation to their geographic, geologic, and ecologic conditions (also represented by isotherms) bringing to the fore climatic conditions of urban sprawl. These investigations are set within cities located at zones exposed to the most extreme climatic challenges on earth. Of the originally three chosen cities - Phoenix (Arizona) as one of the hottest cities on earth, Jakutsk as the coldest big city in the world with temperatures going as low as more than minus 50 degrees Celsius in wintertime and reaching more than plus 30 degrees Celsius in summertime, and Jakarta as one of the largest metropolitan areas on earth situated within the most humid earth zone and already exposed to various challenges caused by climate change – Phoenix and Jakarta have been actively researched through field surveys. The assembled data have been further used as a basis for investigation represented within this research paper.

2 State of the art: imaging the invisible

We might easily recall Walter Benjamin concluding that the media shapes human information processing which he describes in certain ways as to yield the optical unconscious (“das Optisch-Unbewußte").

‘... For it is another nature which speaks to the camera rather than to the eye: "other" above all in the sense that a space informed by human consciousness gives way to a space informed by the unconscious. Whereas it is a commonplace that, for example, we have some idea what is involved in the act of walking (if only in general terms), we have no idea at all what happens during the fraction of a second when a person actually takes a step.’

(Benjamin, 1963)

Benjamin's central thesis revolves around his keen observation of a profound shift in the realm of art and its reception, driven by the progressive evolution of photography and film (Fig. 3).

Fig. 3
figure 3

William Turner’s style dematerializes the representational subject, focusing instead on making the light and the colors of sunlight, fire, and water the true artwork of life. Representative Painting: The Fighting Temeraire (Turner, 1839)

Full size image

Specifically, in Benjamin's view, this transformative process unfolds through the unprecedented reproduction capabilities offered by these emerging technologies. These capabilities, in turn, exert a decisive influence on the collective perception and subsequently reshape the very essence of art's social function. The artistic object lost its “aura” of an individual masterpiece, but in exchange offered the transgressional path to ephemeral concepts as artistic seed instead (Fig. 4). Later on, Karan Barad will employ Nils Borg’s apparatus as an augmentation of human sensory faculties which enables to delve into the realm of nano-physics and quantum physics, registering proof to the abstract scientific mathematical concepts of the un-known reality of these forementioned microscales of existence.

“To be entangled is not simply to be intertwined with another, as in the joining of separate entities, but to lack an independent, self-contained existence. Existence is not an individual affair. Individuals do not preexist their interactions; rather, individuals emerge through and as part of their entangled intra-relating. Which is not to say that emergence happens once and for all, as an event or as a process that takes place according to some external measure of space and of time, but rather that time and space, like matter and meaning, come into existence, are iteratively reconfigured through each intra-action, thereby making it impossible to differentiate in any absolute sense between creation and renewal, beginning and returning, continuity and discontinuity, here and there, past and future.”

(Barad, 2007)

Fig. 4
figure 4

Gerhard Richter, overpainted photographs (Richter, 2015). “Photography has almost no reality, it is foremost just an image. A painting always has reality, the color can be touched, it has presence; It always creates a picture – no matter how good or bad.“ (Translation by Mark Balzar)

Full size image

The pioneering artistic scientist, Steve Mann, renowned as the world's first cyborg, has been exploring an unconventional perspective of the world for over three decades through his personally crafted wearable device, permanently affixed to his skull (Mann, Fung, & Aimone, 2005). This cybernetic lens enables him to perceive the world in a manner distinctly removed from the natural human experience. Mann's device empowers him to view reality through the lens of coded digital filters, creating a unique amalgamation of digital and physical information (Fig. 5). These layers seamlessly merge within his mind, forming a coherent, multi-layered image that unveils a context typically concealed from organic eyes.

Fig. 5
figure 5

Steve Mann, Eyetap (University of Toronto, 2023) (Left) Eyetap for Visual Collaboration (EyeTap for Visual Collaboration, 2023)

Full size image

In terms of imaging the world, technology went through a huge amount of innovation accelerating the informational precision and density given by its model of representation. Especially digital technology hence digital image production and processing has leveled up embedded information transfer.

With our research we carry on Steve Manns approach to offer fresh means of comprehending the urban and architectural landscape. This human initiated amalgam of the Technosphere and the Biosphere (Vernardsky, 1997) profoundly shapes our physical bodies, immediate ecological environments, and societal norms.

The fusion of technological tools with the organic body and digital codes shapes and informs our social environment and the intricate framework of potentially observable hyper-textuality. Diverging from the traditional scientific methodology of comprehending cities by dissecting them into distinct functional layers, each with its unique physical attributes and operational networks, these recently accessible layers in microscale of information sampling hold the potential to redefine and influence the understanding of the urban landscape and the informative patterns underneath. They can function as invaluable tools for research and scientific oversight. In this context, we have the capability to gather chemical and atmospheric samples from sediments spanning across various scales, effectively preserving the remnants of past energetic events.

Drawing a parallel with geological research, this process resembles how geographers obtain excavation tubes of earth sediments from beneath the earth surface in order to gain information from the earth history at a designated geographical location. This practice allows us to uncover parts of the geographical history of a given area.

Contemporary technologies can bring to the fore these geographical layers, transforming those translated data streams into a visible network of relations through pattern recognition. The physical urban fabric itself no longer solely represents the spatial urban context - it has evolved into programmatic capsules encapsulating profound complexities.

Archiving, classifying, and indexing the earths data of these related complex events represents an act of understanding the Earth as a geo-informatic construct (Mattern, 2017) which aids researchers to monitor these synchronous events in different time scales and enables them to formulate predictions concerning future events. In the present day, the accessibility of these observations and data to the public elevates our collective awareness regarding the intricate interplay between the biosphere and the Anthropocene era, fostering a deeper understanding of our planet's evolving dynamics (Balzar & Aksöz Balzar, Narrating Urban DATA, 2023).

A central focus within the Vibrant Fields project's research agenda is the investigation of the concealed realm of information flow and its intricate patterns to draw conclusions and to find potential for future urban developments and innovative approaches for architectural design to maintain a sustainable future. This exploration is conducted through the utilization of specialized sensing objects known as OTTO and ANA, which collectively contribute to sha** the observer's sensory experience (Table 1). The primary mode of inquiry involves capturing and visualizing the fluid-like qualities of these data streams of the transient nature. These data streams are shaped and informed by active interactions with their surrounding environment, akin to an ephemeral painting in constant transformation.

Table 1 Sensors included in OTTO apparatus
Full size table

In the past, historical moments were captured with representative paintings, then with photographs and film. Within the science community relationships and research results were represented with numbers, diagrams, and charts. With the OTTO and ANA recordings we intend to chronicle the qualities of local energy cycles through various data representation methods to render and archive the energetic footprint that relates to the global stream of materials and gases within the biosphere and atmosphere. Both instruments contribute to the Atlas of Climate Change.

2.1 Objectives and methodologies

Creating new methodologies of multifaceted representation technics of correlating multispectral data sets is a priority of the research project Vibrant Fields.

The project addresses these goals by generating own datasets through self-developed sensory apparatus and testing these in selected areas of survey. These areas of survey also build the case studies of the research, where multispectral datasets, approached as sedimental layers are used to explore new visualization techniques that transmit the environmental properties and energetic interactions of the area of survey as an interactive ephemeral painting. These technics include the correlation of local data of urban microclimates and new representations of urban satellite data.

A more detailed description of our approach to urban data correlation will follow at section 4.1.

2.2 A new kind of documentation of places under climatic pressure from the lens of Energy Flows

This methodology encompasses a comprehensive examination of urban environments by employing a multi-faceted approach that integrates satellite data and readily available urban data. Its primary objective is to gain a holistic understanding of global challenges faced by cities in extreme environments. The investigated cities Phoenix and Jakarta are at the forefront of selected metropolitan areas that face ecological pressures caused by climate change and the immediate impact by their local urban microclimates. Additionally, it involves localized data collection of energy dynamics through recordings facilitated by OTTO and ANA.

The utilization of satellite data serves a pivotal role in establishing correlations between urban areas situated in diverse geographical locations. This approach allows for a comparative assessment of the challenges encountered by these urban spaces, albeit with a limited spatial resolution. The in-depth analysis of urban regions also provides valuable insights into local variations in air composition, the presence of urban heat islands, and the occurrence of meteorological events influenced by the unique characteristics of urban morphology.

By dissecting these areas of interest during field trips, the methodology enables the map** of significant environmental disparities in terms of heat, humidity, and air pollution. These disparities are further correlated with population density and building density, offering a comprehensive overview of the intricate relationships between urban form and environmental conditions.

Through the field trips acquired energetic information is turned into interactive point clouds representing the data acquired during several hours of acquisition. Overlaying this information with sound recordings and dynamic 3dscans, we create immersive environments, that can be explored through VR or AR technologies. This way we aim to bring the field surveys to a new level of physical representation creating these multispectral documentaries.

Vibrant Fields employs two distinct rendering methodologies in its pursuit of elucidating intricate ecological phenomena. The first method leverages interactive immersive environments, which serve as a canvas for ephemeral depictions of local energetic footprints. This approach aims to bridge the perceptual gap between the invisible nuances of these ecosystems and human sensory perception.

The second method of the Vibrant Fields project is dedicated to investigating environments subjected to extreme climatic pressures. Employing an energetic documentation technique using OTTO and ANA, this facet of the project seeks to democratize access to these remote and harsh landscapes. By transforming them into immersive, transient digital environments, it offers a means to engage a wider audience and convey a profound understanding of the ecological challenges faced by these locales.

2.3 The realtime rendering of local of energetic footprint as a dynamic immersive map

This rendering method was developed as a representative experiment for an exhibition format, exploring the different immersive visualization techniques within the micro environments of the exhibition space. The real time visualization aims to represent the environment and the agents within as an interconnected system of energetic exchange by visualizing the acquired data using OTTO and ANA in real time. By interacting with the sensors, the visitors are able to change projections representing the data acquired by the devices.

In this setup, data was collected through various environmental sensors in 36 multispectral dimensions, a thermal camera, and a lidar scanner, which was then transformed into two projection maps depicting the energy footprint of the space (Fig. 6).

Fig. 6
figure 6

The real-time 3d scan of the space overlayed with thermal image

Full size image

First projection featured a real-time 3D lidar scan overlaid with the thermal image displaying the exhibitions space and the visitors as a dynamic point cloud. Creating this real-time visualization presented various challenges. The Ouster LIDAR Scanner has its own visualization environment Ouster Studio. However, this program lacks the capability to integrate additional information or data from other sensors. To overcome this limitation, a custom Python code was developed to access the lidar sensor directly, capturing the point cloud's coordinates and writing them to a CSV file. The CSV file was then streamed to Rhino Grasshopper, where a new point cloud was generated to overlay the thermal information in Grasshopper.

The lidar scan consistently emits the same number of laser beams per rotational angle. Our Ouster Lidar system comprises 1024 steps in a 360-degree horizontal rotation, with each step covering a 45° angle and dividing it into 512 beams. This configuration results in 512x1024 points per frame. The specific points recorded depend on the obstacles the laser beams encounter. As a result, objects at greater distances receive fewer points, leading to variable point density in the 3D point cloud, which is not the case in 2D images captured by the thermal camera (Fig. 7).

Fig. 7
figure 7

Step by step creation of the exhibition projection from static 3dScan (top left) contouring to represent four dimensions of gas data (top right) representation of humidity, pressure, temperature, and visible light in shape shifting bubbles placed in the virtual 3d space of contours (bottom)

Full size image

To achieve an accurate alignment between the thermal image and the point cloud projection, several geometric operations must be performed in the code. First, we need to determine the depth of field with respect to the angle of the thermal camera lens, then we select the Lidar Points within this depth of field and adjust their positions to align with the camera lens, matching the point indices with the pixels. Once this alignment is achieved, we can accurately associate the correct 3D points with their corresponding thermal values. The visitors can view themselves as thermal point clouds in the projection and observe their thermal exchange with the exhibition space in an abstract projection.

The second projection displays the acquired data from environmental sensors of OTTO and ANA.

The graphical composition merged a static 3D scan of the existing environment (as a canvas) with the real-time sensor monitoring. Sectional lines following the spatial digital parameters (3D footprint) indicate the four gas values (CO, VOC, NO2 and C2H5CH) through a colour separation. These 4 colour lines shifting in intervals of 4 change thickness and colour brightness according to the shifts in value. Additionally, each of these lines was activated through a different audio spectrum which all together moved in waves according to the sound frequencies within the recorded space.

Within the physical space, two portable ANA devices where separately arranged according to the diverse zonings which found their equivalent within the 3D realm. In the 3d space, these spots were indicated through pixel bubbles which changed their shape in flux according to the changing humidity and temperature furthermore the visible light value in Lux controlled the pixel size of the points representing the bubbles where the pressure defined the overall size of the bubble. In this manner all data flux orchestrated the fluctuating energy landscape painting (Fig. 8).

Fig. 8
figure 8

Overlay of interactive energy footprint visualization using AR. Overlay within the space(left) Representation of the acquired data on small scale on desktop. (right)

Full size image

2.4 From resolution to temporality

Satellite data operates on a temporal scale that may not directly align with the human perceptible spectrum. These satellite observations aggregate data over extended periods, such as days, months, or even years, and distil them into a single comprehensive image from a planetary perspective. In this context, satellites serve as planetary observers, capturing long-term global trends and changes affecting our earth crust and the earth's atmosphere (Fig. 9).

Fig. 9
figure 9

Image by © NASA displaying environmental changes during a Covid 19 Lock Down in early 2020. (Stevens, 23 n.d.)

Full size image

However, the human sensory experience operates on a more localized and immediate scale. Our daily impressions arise from a multitude of sensory inputs that we often perceive but may struggle to fully comprehend or attribute. To bridge the gap between environmental changes and our sensory perceptions, it is essential to conduct a spectral analysis of the energetic information present in our surroundings. The challenge of getting high-res data is tackled with portable sensors. Several initiatives aim to gather urban data using extensive sensor networks (Diez Ladera and Posada 2013).

In addition to already existing smart city devices and initiatives, our multispectral real time approach with OTTO and ANA facilitates a deeper understanding of how the environment influences our daily experiences by dissecting the complexities of the sensory information within our immediate surroundings.

2.4.1 Satellite data and urban correlations

The Vibrant Fields project involves filtering, correlating, and analyzing publicly accessible satellite data through platforms like Google Earth Engine and QGIS, translating selected bands into red, green, and blue values for visualization. This endeavor utilizes the Landsat program, which has evolved through nine satellite cycles, providing 50 years of planetary observations with increasingly advanced spectral bands, demonstrating substantial progress in planetary data collection (Table 2). Each distinct sensor is finely tuned to detect electromagnetic wavelengths, including light beyond the visible light range. The combination of these diverse spectral bands, organized into RGB channels, enables the extraction of various environmental features and phenomena.

Table 2 Summary of different spectral bands
Full size table

Through various combinations of these bands, it becomes possible to render specific surface features visible to the human eye. To visualize aspects like vegetation, clouds, soil moisture, and hotspots, satellite bands located both above and below the visible range are employed to generate what is commonly referred to as a "false" colour image. This image transforms the imperceptible wavelengths into RGB channels, effectively highlighting the desired features for observation and analysis.

In addition to satellite imagery, Google Earth Engine also encompasses a wealth of additional planetary information. This platform offers access to various datasets that provide insights into atmospheric and climatic conditions, as well as human-related information, such as population density and the locations of human settlements (Table 3).

Table 3 Vibrant Fields uses following datasets among others were used to establish correlations
Full size table

In addition to the datasets mentioned, we tailor our data analysis to each field research and its specific global challenges, with a focus on climate change. By incorporating meteorological maps and spectral band visualizations from Landsat 8 and 9, we enhance our environmental data.

We integrate anthropogenic data, like human settlement patterns, with atmospheric variables such as temperature and gas concentrations, enabling a swift understanding of how densely populated urban centres impact atmospheric dynamics (Figs. 10 and 11). Crucially, our analysis considers time as a vital dimension, harnessing extensive long-term environmental records to capture temporal changes.

Fig. 10
figure 10

QGIS, Overlay of Phoenix Human Settlement, Nitrogen Dioxid, Slope ALOS Elevation Terrain, LCZ-Filter

Full size image
Fig. 11
figure 11

East Java and Jakarta CO concentration 2018-2020 Mean(top) 2020-2022 Mean(Bottom) The examination of carbon monoxide levels in Jakarta during different time intervals reveals significant insights, demonstrating a correlation between human activities and shifts in atmospheric composition, both pre-pandemic, during the COVID-19 pandemic. (Union/ESA/Copernicus, Copernicus Sentinel data, 2020a, 2020b)

Full size image

Conversely, higher resolution images, particularly at (10 miles per 67 pixels), provide detailed representations of urban density, population distribution, and architectural morphologies. At this finer scale, we can readily discern correlations between urban heat island effects and population density. However, it is important to note that datasets related to air quality and particulate matter such as methane and carbon monoxide at this scale possess significantly lower spatial resolution. Consequently, these datasets serve as global indicators rather than offering localized insights (Fig. 12).

Fig. 12
figure 12

QGIS, Overlay of Phoenix Flow Accumulation, Soil Water Content at 33kPa, Slope ALOS Elevation Terrain, Crop Mask

Full size image

2.4.2 Satellite data correlation using machine learning

Visual and numerical analysis across various temporal scales serves as a valuable tool for monitoring changes in urban structures over time. However, comprehending the intricate correlations and recurrent patterns within multiple layers of data can pose challenges when relying on traditional analytical and visualization methods.

Our approach incorporates two distinct machine learning methodologies for an enhanced understanding of the relationships between different data dimensions drawn from various global datasets. First, we utilize K-Means (MacQueen, 1967) clustering to subsample the extensive array of features within the datasets. This technique is applied across multiple spectral bands within a dataset, facilitating the map** of correlations between different data dimensions. It is important to note that this method is effective when working with integral bands within the features provided by Google Earth Engine's developer system. To correlate multiple datasets effectively, information from different sources is integrated as additional bands into a primary dataset, merging various types of satellite data. In doing so, Landsat bands can be amalgamated with offline methane recordings and supplementary atmospheric composition data. This integration allows us to discern correlations and similarities within urban areas, particularly in terms of material attributes such as building density, as determined through surface radiation analysis, in relation to atmospheric composition and heat. Through this image segmentation approach, the satellite image can be simplified to highlight different features.

A secondary method for satellite data analysis involves the application of supervised learning to extract urban information that may not be classified in certain areas. Leveraging landcover classification datasets from the USA, we trained a Classification Model capable of extracting land cover classifications from Landsat images in areas where such information is lacking (Fig. 13).

Fig. 13
figure 13

K-means clustering was applied to two Thermal Infrared Bands of a Landsat satellite image of Jakarta. As the number of clusters increased, distinct thermal radiation features became discernible. In the case of fewer clusters, the urban area and water bodies, along with their surroundings, were distinguishable. However, as the number of clusters increased, additional details such as highways, built environments, wetlands, and densely populated areas of the city became visible

Full size image

For the supervised classification algorithm training, the 2021 release of the USGS National Land Cover Database was utilized (Dewitz, 2023). This comprehensive database offers a detailed classification of land cover across the USA, encompassing 20 distinct classes at a spatial resolution of 30 meters (Multi-Resolution Land Characteristics Consortium, 2024).

Given the complex, nonlinear relationships between multispectral band values and land cover classes, a CART (Classification and Regression Trees) (Breiman, Friedman, Olshen, & Stone, 1984) classifier was chosen for its proficiency in handling such intricacies. This decision was informed by the nature of the classification task, where the feature vector, comprising values from various spectral bands, exhibits a complex correlation with land cover classes.

The training process involved constructing a feature vector for each location, represented as a data point. These vectors included the spectral band values from satellite imagery, paired with a corresponding target land cover class for the same location. Experimentation involved testing different spectral bands from the Landsat-8 dataset (Landsat-8 image courtesy of the U.S. Geological Survey) to optimize the composition of the feature vector. The validity of the trained model was tested in US data sets, using one city as training set and another with an existing landcover classification for validation.

Throughout our experiments, it became evident that the accuracy of land cover classification is highly dependent on the quality and representativeness of the training data. Two critical factors were identified to enhance the predictive accuracy for undocumented and unclassified areas.

First, the choice of spectral bands for the feature vector was paramount. Each band captures unique surface characteristics, affecting the classification efficacy. The most successful feature vector combined Bands 1 (coastal/aerosol), 5 (near-infrared), 6 (short-wave infrared 1), and 7 (short-wave infrared 2). This selection was based on their comprehensive ability to represent various land surface attributes.

Second, selecting training locations with characteristics similar to the target prediction areas was crucial. For instance, constructing a model with training data from a U.S. city that shares geographical features with the city under study ensures better prediction accuracy. Training with coastal data, for example, may yield ineffective results for inland regions due to differences in landscape features and the potential absence of relevant classes in the dataset.

By adhering to these principles, we developed models that predicted land cover information for our survey zones, facilitating more effective planning and execution of field expeditions.

It's important to acknowledge that this method is still evolving. Recent experiments have led to the fine-tuning of the feature vector specifically for coastal areas with wetlands. However, in regions with different landscape characteristics, adjustments to the feature vector may be necessary. For example, removing the coastal aerosol band (Band 1) from the input vector could yield better results in non-coastal areas.

Moreover, the current training set is limited to a single satellite image. Expanding this to include multiple images, or potentially the entire satellite imagery dataset of the USA, might enhance the algorithm's ability to generalize. This broader dataset could provide a more diverse and comprehensive range of land cover types and conditions, potentially improving the accuracy and applicability of the model across various geographical locations.

3 Field surveys

3.1 Field survey: Jakarta

The Jakarta metropolitan area with an area of 6,802.10 km2 and a population of 31.24 million inhabitants as of 2020, stands as one of the world's largest and most humid cities affected by climate change. Located near river mouths and parts of the city situated below sea level, Jakarta grapples with tidal variations, canal floods and hurricanes, which unfortunately lead to loss of life and property. Surprisingly, despite these challenges, the city's densest areas are still situated along the canals and near the coastline.

Within this diverse urban landscape encompassing a wide range of economic capabilities, Jakarta boasts a distinctive network of communities known as "Kampungs." These Kampungs represent tightly knit communities with limited economic resources that intricately interweave with the city's residential areas. Often characterized by less-than-optimal living conditions due to their high population density, these Kampungs also foster a unique social lifestyle that predominantly unfolds in the streets and open spaces.

Kampungs, as communities, possess robust social and political cohesion while being acutely conscious of their environmental challenges. Our collaborators at the Rujak Center for Urban Studies play a pivotal role in this political framework, serving as architects who assist Kampungs and cooperatives within them in the development of both existing and new structures that prioritize resilience and environmental consciousness.

Rujak, as an architectural community, assumes an advisory role within the Kampungs, aiding them in the advancement of their settlements. This assistance is rendered with due consideration for preserving their distinctive lifestyles while concurrently incorporating hygienic aspects into the construction of buildings.

In our collaboration with Rujak, we had the unique opportunity to actively explore the energy footprint of Kampungs on-site in Jakarta. Our devices, OTTO and ANA, were introduced to the community, and their functionalities were thoroughly explained to the Kampung residents. Community leaders, serving as key political figures, expressed a keen interest in accessing environmental information regarding their settlements (Fig. 14). As a result, six distinct communities expressed their interest in our data collection, allowing us to conduct measurements within their respective areas. Ultimately, we selected four communities for recording based on considerations of the built environment and settlement variations.

Fig. 14
figure 14

Photographs of newly developed vertical Kampung Kunir(Top) in comparison to older kampungs. Kampung Marlina(Left) Kampung Jedong Pompa (right)

Full size image

Two of these selected Kampungs, Kampung Akuarium and Kampung Kunir, represent new building typologies known as vertical Kampungs. These vertical Kampungs have been constructed in place of older settlements that were demolished. Notably, these residential buildings feature a unique network of spacious, open corridors that replicate the street experience on higher levels. In this context, the corridors extend beyond merely serving as pathways; they also function as an integral part of the living space and social life within the community. It is common to find small restaurants and shops run by the inhabitants seamlessly integrated into these corridors.

The other two Kampungs, namely Kampung Marlina and Kampung Jedong Pompa, represent older settlements characterized by dense informal housing and a highly intricate network of streets. These areas face significant challenges, including poor living conditions. In both of these Kampungs, streets serve dual roles as living spaces and communal areas, but they also function as transportation routes. Unfortunately, this multifunctionality leads to environmental issues, primarily stemming from the emission of harmful gases by motorbikes and problematic canal situations, which frequently subject inhabitants to flooding.

A common practice in these Kampungs is the extension of indoor spaces by extending structures on the first floor over the street. However, this architectural approach can have adverse effects on air circulation, further exacerbating the environmental challenges faced by the residents.

During the field trip urban energetic footprint samples were not only collected in the Kampungs but also in the vibrant areas of the city with different urban properties including residential areas, vivid urban zones, and touristic attractions.

3.1.1 Data correlation and visualization

The collected samples provide a concise comparative analysis of environmental attributes across various locations within Jakarta during the same season. The acquired dataset encompasses air quality, climatic information, as well as sound and live 3D scans. These data sources enable the establishment of a comprehensive documentation of specific locations during a particular time frame (Fig. 15).

Fig 15
figure 15

Dynamic 3DScan of Kampung Marlina overlayed with thermal information and information acquired from different ANA devices distributed along the area

Full size image

This data serves a dual purpose: firstly, it facilitates the comparison of environmental characteristics among different kampungs; secondly, it introduces a novel documentary approach for capturing the essence of a place that transcends conventional sound and camera-based documentation methods (Fig. 16).

Fig. 16
figure 16

Jakarta, Kampung 3D Scan inkl. Representation of Sound frequincies, Humidity and Heat. Sequences of a movie through the area

Full size image

Environmental correlations hold significant importance for map** various types of urban settlements inhabited by diverse societal groups. This map** is essential to assess access to healthy environmental conditions. Additionally, investigating the environmental attributes of various kampungs aids in gaining insight into the implications of newly designed vertical kampungs as compared to traditional informal kampungs. The sharing of this data with our collaborators at the Rujak Center also assists them in comprehending the consequences of their design choices and empowers kampungs to advocate for their rights to improved living conditions. However, this only helps the local authorities to act on these challenges (Fig. 17).

Fig. 17
figure 17

Correlation of environmental recordings in between city center Cikini and all Kampungs of OTTO from Jakarta CO(top-left) Temperature (top-right) O2 Concentration(Bottom-left) Humidity(Bottom-right)

Full size image

With our project, Vibrant Fields, our primary objective is to raise awareness of these challenges on a global scale. To achieve this, we are dedicated to develo** an immersive documentation technique that presents immersive representations of local conditions. Through these documentaries, our aim is to capture a novel perspective of places by focusing on energy flows.

In the context of Jakarta, we have generated virtual reality (VR) environments from the 3D scans of the surveyed locations. These VR environments enable observers to explore these places from an immersive perspective. Additionally, we overlay environmental data within these VR environments, introducing a secondary layer of visual information. Here, changes in climatic and air quality values are represented as distinct geometric elements in this transient environment. The ever-evolving geometries and colours in these representations immerse the observer in a realm of energetic information.

3.2 Field Survey Phoenix Arizona

Phoenix, Arizona, ranks among the hottest cities on Earth, experiencing peak temperatures exceeding 43.3°C (110°F) (Cusick, 2023) throughout the year, totalling 53 days in 2020 alone. Furthermore, the humidity levels in Phoenix fluctuate annually, ranging from 17% to 44%, with an average of approximately 31.3% (ClimaTemps, 2023). Phoenix is already grappling with the effects of climate change, notably exemplified by the urban heat island phenomenon (Nazarian, et al., 2022). This microscale phenomenon within the metropolitan area already effected the macroclimate (Theeuwes, Barlow, Teuling, & Kotthaus, 2019).

These numbers in tow lifecycles in the city organize around these peek temperature ranges outside airconditioned environments.

Devoted runners strap on headlamps to go jogging at 4 a.m., when it is still only 90 degrees, come home drenched in sweat and promptly roll down the sun shutters. Neighbourhoods feel like ghost towns at midday, with rumbling rooftop air-conditioners offering the only sign of life. (Holloway, 2019)

The team quickly discovered large microclimatic differences within the city, which interact with cultural characteristics, demographic and functional differences, and architecture.

Within this diverse urban landscape encompassing a wide range of different urban environments and neighbourhoods with different architectural qualities, Phoenix boasts a distinctive patchwork of materialisation effecting urban microclimates (Figs. 18, 19 and 20). A tremendous amount of hardcovered solid surface conditions interweave with the city's residential areas where only in the wealthier areas greenery and green parks can be maintained. Conversely, in less affluent regions, the urban landscape is often marked by suboptimal conditions. Here, landscape design predominantly features elements such as stone, cacti, or artificial plastic grass and plants, which contribute to heat retention within the city's atmosphere. Additionally, the exhaust emissions from older vehicles release fine particulate matter and elevated levels of gases that lead to exacerbating air quality concerns (Fig. 21).

Fig. 18
figure 18

QGIS, Overlay of Phoenix Human Settlement, Carbon Monoxide, Basic Demographic Characteristics, Slope ALOS Elevation Terrain

Full size image
Fig. 19
figure 19

QGIS, Overlay of Phoenix Land Surface Temperature, Slope ALOS Elevation Terrain, Basic Demographic Characteristics

Full size image
Fig. 20
figure 20

QGIS, Overlay of Phoenix Human Settlement, Nitrogen Dioxid, Landcover-10m

Full size image
Fig. 21
figure 21

Survey Areas in Phoenix were selected in the contrasting heat islands of city. On the one hand hottest areas such as highways and economically challenged neighborhoods were compared to green areas and touristic locations

Full size image

Hence, it becomes imperative to explore passive cooling strategies aimed at mitigating both building energy usage and external heat radiation. Some experimental areas have started addressing this issue by applying white paint to pavements. Through such measures, it has been determined that the urban heat island effect can be significantly diminished, albeit with daytime air temperature reductions typically at around max. 1 degrees Celsius due to the increased surface reflectivity (Schneider, Ortiz, Vanos, Sailor, & Middel, 2023). This efficacy is primarily attributed to the white color, which promptly reflects incoming heat back into the atmosphere rather than absorbing and retaining it within the material.

One of our key observations was that evaporative cooling through vegetation represents one of the simplest and most ancient forms of air conditioning within urban environments, effectively alleviating climatic stress within the targeted areas. Evaporative cooling has gained considerable traction in the field of air conditioning over the past decade owing to its straightforward design, cost-effectiveness, and utilization of natural resources. Notably, it has found practical application in places like Phoenix, Arizona, where affluent communities can afford the expenses associated with watering and upkeep.

Commencing our research expeditions in Phoenix, Arizona, swiftly illuminated the distinction between raw contextual data readings and the nuanced insights derived from actual, on-the-ground experiences. Even though the cultural explorations and coverage of our measurements have been very poor due to the climatic preconditions, our understanding of data-able-nature has been rich and unfolded new directions for further studies.

4 Conclusion

This paper explores documentation techniques aimed at comprehending the essence of places from an energetic standpoint through material exchange captured by the chemical and physical traces they leave through interaction. Information sampling as practiced through the research of Vibrant Fields stores digital protocols of environmental changes in the biosphere to build data resources for further analytic innovation. Those digital archives of the multispectral recordings might also give future geologist, climate specialists as well as architects and urbanist incorporated answers to future questions not conducted in research yet (Fig. 22).

Fig. 22
figure 22

Visualization of all the data dimensions in correlation in this case from the acquisition in Phoenix Arizona in Biltmore Area during sunrise

Full size image

The self-developed tools, OTTO and ANA, play a central role in acquiring data across multiple layers yet not accessible at once through a single multispectral recording device. Conducting acquisition surveys in various Earth locations characterized by extreme climatic conditions sheds light on the local challenges posed by climate change. These tools excel in capturing environmental conditions through sensory data acquisition within specific time frames.

In contrast to the information inherent in satellite datasets, which covers broader time periods at a global scale, the acquired data delves into the environment with higher resolution with multiple levels of information, providing a more human-centric map** of places. By seamlessly transitioning between temporal and spatial scales of information, this project aims to comprehend climate change from multiple perspectives with artistic means.

Such documentation further serves a dual purpose: firstly, it aids local authorities and communities in gaining a deeper understanding of their environment, thereby empowering them to advocate for improved living conditions. Secondly, these documentaries offer a novel perspective on gras** the essence of various locales, effectively translating the implications of climate change into a more humanly perceptible scale.

The goal is to craft interactive landscape paintings that vividly portray our physical surroundings and illustrate the repercussions of the climate crisis through Anthropocene on nature through new forms of information.

The way an ecosystem conveys its information to an observer is contingent upon both the sensory "tools" of the recipient and the nature of the contained information. Innovating approaches for the collection of diverse urban data and discovering fresh methods for representing this information holds the potential to reveal new perspectives on the physical manifestations of architecture within interconnected ecosystems, and their intricate interplay with the micro and macroclimates.

As a result, there is a pressing need to expand the scope of sensing, data acquisition, and data analysis beyond the conventional realm of climate analysis. This expansion is crucial to construct a more precise model encompassing a city's architecture, infrastructure, and ecosystem, along with their underlying information networks.

This research presents a novel yet experimental approach to understand urban microclimates, focusing on the intricate energy exchanges that occur within them. The developed devices, while prototypical, have been instrumental in exploring the significance of various dimensions of energetic data. This data is crucial for painting a comprehensive picture of urban microclimates.

Moving forward, the primary goal is to refine these devices and the sensors within them. Taking into account the proof-of-concept phase of the project, the budget friendly Arduino based sensor electronics are capable of enabling qualitative data comparison between the individual recordings, whilst a scientific quantitative recording needs higher standards in the quality control of the microelectronics and stability of measuring results. Unfortunately, the variation between the induvial Arduino sensors was relatively high. By enhancing sensing precision, we aim to reach higher quality and standards in quantitative measurements.

Moreover, the inclusion of a high-resolution Lidar sensor is highly advantageous, as it would significantly improve the precision of long-distance recordings recognizing tracked objects and nature. Furthermore, this enhancement would streamline the process of linking the recording profile to its immediate sources. In addition, future research should not only focus on continuously map** these microclimates but also harness this data to inform new urban developments. With its heightened precision, this measurement methodology holds the potential to provide more profound insights into the influence of architectural decisions on the urban environment and its material as well as social characteristics beyond the individual autarch building performance.