Latent Dirichlet Allocation and Hidden Markov Models to Identify Public Perception of Sustainability in Social Media Data

Abstract

To help guide a just transition to a sustainable society and onboard the local communities, researchers can identify events of public interest through access to data from community engagement activities and social media content. However, novel analytic methods are required to process and analyse data in unstructured formats (e.g. transcripts, text and images) and to extract useful information for decision-making. This paper proposes an analytics pipeline combining latent Dirichlet allocation and hidden Markov models for automatically detecting multiple latent changepoints in topics over time, without prior knowledge of their occurrence. Analysing social media content (i.e., tweets) related to Glasgow, we identified events that captured social media users’ public interest, demonstrating the potential of our method to inform timely and relevant policy making.