Abstract
This chapter discusses methods appropriate to the study of social protection in Latin America. It reviews and assesses the main methodological approaches employed by researchers in the past. Building on these, it identifies methods appropriate to develo** a theoretical perspective capable of explaining the evolution and current configuration of social protection institutions in the region. Recent trends in social research methods reveal a renewed emphasis on causal inference. The application of quasi-experimental methods in the study of the outcomes of social protection interventions, initially in the context of conditional income transfers, has focused attention on the causal effects of social protection interventions. The chapter assesses the challenges associated with the application of causal inference models in the context of research on institutions relying on observational data. It argues the potential outcomes approach offers a systematic framework for incorporating attention to counterfactuals. It makes a case for the use of graphical casual models to help discriminate causal versus non-causal association between variables, thus refining researchers’ hypotheses and linking the model to potential empirical counterparts.
You have full access to this open access chapter, Download chapter PDF
This chapter discusses methods appropriate to the study of social protection in Latin America. It reviews and assesses the main methodological approaches employed by researchers in the past and, building on these, it identifies methods appropriate to develo** a theoretical perspective capable of explaining the evolution and current configuration of social protection institutions in the region. The task is to select methods of data collection and analysis that will connect theoretical propositions to empirical counterparts.
By and large, the literature on social protection in Latin America has relied on case studies, cross-country institutional comparisons, linear structural equations estimated on aggregate annual data, and quasi-experimental evaluation of social protection interventions. After the turn of the century, critical improvements in data availability have supported comparative analysis. The spread, regularity, and accessibility of individual and household dataFootnote 1 since the mid 1990s has facilitated the application of a wider range of analytical tools for the comparative study of social protection. Dedicated beneficiary surveys in some countries in the region provide good quality detailed data on social protection.Footnote 2 More recently, impact evaluation survey data collected to assess the effects of social assistance programmes have been available to researchers. They offer an excellent resource for research on social protection outcomes.Footnote 3 Region-wide attitudinal survey data provide an essential resource to study public preferences on social protection.Footnote 4 There are also notable improvements in the availability of administrative data.Footnote 5
Recent trends in social research methods reveal a renewed emphasis on causal inference. This is the outcome of push and pull factors. On the one hand, a growing awareness of the deficiencies associated with correlation bias in quantitative analysis. On the other, the growing application of experimental approaches to data collection and analysis (Angrist & Pischke, 2008; Banerjee & Duflo, 2008; Gelman & Imbens, 2013). The application of quasi-experimental methods in the study of the outcomes of social protection interventions, initially in the context of conditional income transfers, has focused attention on the causal effects of social protection interventions.
The issue to be discussed in this chapter is the extent to which these methodological developments can guide research on social protection institutions. There are significant challenges to causal inference in social protection research. Arguably, the study of social protection institutions could be seen as fundamentally different to the study of whether conditional income transfers have specific effects on their recipients. There are obvious limits to the application of experimental techniques in the study of social protection institutions and policies. Institutions evolve in time making it hard to keep conditions unchanged. Controls groups can be hard to identify in the context of social policy. Implementation issues can be significant in the context of decentralised programmes affected by territorial diversity (Niedzwiecki, 2018).
Nevertheless, causal inference studies are increasingly being implemented in the study of social policy (Morgan & Winship, 2015). Taking causal inference seriously promises to deliver significant gains. The implementation of graphical causal models forces researchers to examine critically the relationships between the variables of interest on the basis of their theoretical frameworks (Elwert, 2013). Sifting causal from non-causal factors associated with social protection systematically, as opposed plugging multiple control variables in regression exercises, is a welcomed discipline. The potential outcomes approach can be usefully applied in the context of observational data, providing the preconditions for its applicability are considered carefully (Imbens, 1990). This approach distinguishes welfare systems according to the institution that dominates their welfare provision: markets, the state, and families. This comparative approach has generated a great deal of research interest because it claims to unveil fundamental differences in the structure of capitalism among high income countries.Footnote 8 The main analytical tools employed in this literature cluster countries by extracting an index of differences or similarities from multivariate data (Barrientos, 2015). Cluster analysis is a commonly used approach (Abu Sharkh & Gough, 2010; Gough, 2001; Hirschberg et al., 1991; Powell & Barrientos, 2004). Martínez Franzoni (2008) has applied this approach to social policy in Latin America. Alternative data reduction techniques have been implemented in the region. Cruz-Martínez (2014, 2017b) employs principal components analysis to reduce several indicators supporting the construction of indexes which are then combined into a multidimensional welfare index. Countries are then ranked by their scores on this welfare index. Two points from this literature need to be underlined at this juncture. First, a crucial underlying assumption is that welfare institutions are multidimensional, hence the need to implement data reduction. Second, country grou**s are helpful, analytically, in distinguishing multiple configurations of welfare institutions in the region, but they also have a normative content is as much as they reveal good and bad outcomes.
Qualitative Comparative Analysis (QCA) has proved especially useful in the study of institutions, their configuration, and change over time (Kangas, 1994; Kvist, 2007; Vis, 2012). It has been applied to the study of welfare institutions in Latin America (Cruz-Martínez, 2017a, 2019; Segura-Ubiergo, 2007). Simulation studies are also effective in providing a comparative perspective on social protection (Altamirano Montoya et al., 2018).
Time-series-cross-section and panel data are employed extensively in economic and political studies which include social protection institutions. Studies have considered whether globalisation has undermined social expenditure (Avelino et al., 2005; Huber et al., 2008) or whether democracy or conflict are associated with social programmes and expenditure (Yörük, 2022). Studies have relied on time-series-cross-section data to investigate the factors associated with the origins and evolution of welfare institutions (Haggard & Kaufman, 2008; Huber et al., 2008; Kaufman & Segura-Ubiergo, 2001; Segura-Ubiergo, 2007). Time-series-cross-section and panel data analysis are grounded on structural linear equation models unveiling correlations among variables of interest. The statistical models estimated are scrutinised as regards the significance of the estimated coefficients and the strength of the model’s ability to reduce unexplained variance.
Event studies are helpful in identifying the factors constraining or facilitating the emergence of specific social protection institutions. They focus on unveiling correlates of foundational events (Knutsen & Rasmussen, 2018; Rasmussen & Knutsen, 2017; Schmitt, 2015; Schmitt et al., 2015).
More recently, the availability of evaluation and observational data has encouraged quasi-experimental techniques to study the impact of social protection interventions. In Latin America the spread of conditional income transfers helped bed in the collection of evaluation data and techniques for their evaluation. Mexico’s Progresa was the pioneer (Skoufias, 2005). Exploiting a scheduled implementation of the programme, difference in difference estimates of its impact on poverty helped to protect it from contracting government budgets (Levy, 2006). Impact evaluations of old age transfers estimate the effects of the programmes by focusing on the differences in social indicators around the age of eligibility, a regression discontinuity design (Galiani et al., 2014). Regression discontinuity design can be implemented where programme regulations or their change over time generate exogenous breaks in benefit entitlements (Barrientos & Villa, 2015). Comparative studies extract information from the estimated impact of specific programme. Meta studies of the impact of conditional income transfers on education (Saavedra & Garcia, 2017); child mortality (Cavalcanti, 2023); labour supply (Alzúa et al., 2010); or elections (Araújo, 2021) provide a comparative perspective on social assistance. The rapid spread of evaluation data and quasi-experimental analytical techniques have greatly refined our understanding of the impact of social protection interventions in the region.
2.2 Shifting Attention to Causality
The growing use of experimental and quasi-experimental in social protection, especially in the study of the effects of social assistance programmes is part of a methodological shift focused on causal inference. In the context of social assistance programmes this focus on causality has greatly benefited from the design of impact evaluations and the availability of quasi-experimental data. This will be discussed in detail in the Protection chapter. This approach is predicated on the presence of an ad hoc intervention, social assistance transfers for example, capable of leading to a change in the behaviour of recipients. Dividing potential participants into two equivalent groups and implementing the intervention to the treatment group makes it possible to measure the difference in outcomes across the treatment and control groups after the intervention. This measure is a representation of the causal effect of the intervention.
There are multiple ways in which social protection institutions, like occupational pension funds or individual savings plans, are likely to have more complex effects than conditional income transfers. It is harder, and often unfeasible, for governments to experiment with large scale pension schemes, if anything because of the time horizons required. To a significant extent, researchers can only rely on observational data to try to understand social protection institutions. Generalising from local experiments is perilous as the causal effects identified by impact evaluation studies trade off external validity for external validity. These and other issues have persuaded researchers examining social protection institutions to sidestep causal inference in favour of associational models, qualitative and quantitative.
The aim of the materials discussed here is to argue there is path towards a causal examination of social protection institutions in Latin America. The main argument is that causal models are essential to unveil the factors explaining existing social protection in the region. The causal framework should not be limited to the study of the effects of local interventions but can also be deployed to study the configuration of social protection institutions in the region. To do this successfully, we need to integrate a set of methodological tools into our research together with an understanding of their limitations or, what is the same, an understanding of the conditions in which they can help advance understanding.
2.3 ‘Effects of Causes’ and ‘Causes of Effects’ Explanations
Gelman and Imbens (2013) make a distinction between two types of causal explanations in the social sciences. First, a forward explanation that answers to the question: ‘what are the effects of causes’? This is at the core of the experimental methods revolution. A second set of explanations consist of backward explanations answering the question: ‘what are the causes of effects? They start from a set of effects and seek to understand what factors caused them. Impact evaluation studies of conditional income transfers belong to the first set of explanations, the ‘effects of causes’, while studies aiming to explain the origins of occupational insurance funds in Latin America in terms of associated features of society, politics, or the economy belong to the second set of explanations, the ‘causes of effects’. Yamamoto (2012) argues that ‘causes of effects’ explanations “are about attribution, instead of effects, because their primary concern is the extent to which the actual occurrence of events can be attributed to a suspected cause” (2012, p. 1). This is closer to the term in common use. This distinction is relevant to the methodological approach in this book.
The ‘experimental revolution’ involved shifting the focus from ‘causes of effects’ to ‘effects of causes. In Gelman and Imbens’ view, estimating causal relations can only be done with forward causal questions. Studies attempting to explain causes of phenomena usually begin by identifying a set of potential causal factors and then test for the presence or absence of these factors in the antecedents of the phenomena. The presence of the effect often precludes consideration of possible counterfactual. Even when attention is paid to counterfactuals, reverse causation studies find multiple explanatory factors capable of confounding the potential link between causes and the effect. Research findings are on stronger ground if they can dismiss, falsify, the influence of some factors on the presence of the phenomena.
The current dominance of ‘effects of causes’ methodologies, especially experimental methods, has cast doubt on the contribution that ‘causes of effects’ studies can make (Goertz & Mahoney, 2012) and on whether this approach can be accommodated within statistical or econometric estimation models (Gelman & Imbens, 2013). Some researchers go further and argue that only an external intervention or manipulation can support causal inference. Experimental data is needed to support causal inference, observational data lacking an intervention in the data generating process will not be capable of supporting causal inference.
It can be argued this view is too restrictive, that ‘causes of effects’ explanations have a role to play in advancing causal knowledge. Experimental methods are unfeasible in many social protection or social policy contexts that motivate social protection researchers in Latin America. However, a ‘causes of effects’ approach could be helpful in the task of identifying non-causal factors, the factors that are not causally linked to social protection institutions; in refining existing models capable of capturing the relationships involved, and perhaps in hel** formulate hypotheses to be explored using ‘effects of causes’ research methods. Identification of causal and non-causal factors is a first step (Morgan & Winship, 2015, Chapter 3). It makes explicit the assumptions that are required to claim causal status. Searching for causes of effects or inverse causal inference could lead to improvements in the maintained model or suggest its replacement with a better one. Gelman and Imbens suggest that ‘causes of effects’ questions fit into statistics and econometrics not as inferential questions to be answered with estimates or confidence intervals, but as the identification of statistical anomalies that motivate improved models” (Gelman & Imbens, 2013, p. 4).
In fact, the identification of relevant factors capable of explaining the emergence of social protection institutions has been the primary objective in the quantitative literature. In some contributions, this relates to conditions required for social protection institutions, for example the level of economic development, or democracy, or openness, or urbanisation. Studies interrogating times series cross-section data have focused on the identifying the main correlates of social expenditure. The underlying statistical model is a linear structural equation model including a range of variables to ‘control’ for potential confounders and estimated using regression analysis. Studies using times-series-cross-section data (Segura-Ubiergo, 2007) and panel data (Haggard & Kaufman, 2008) provide some examples.
‘Causes of effects’ or reverse causation studies have contributed to refining the identification of the factors explaining the emergence and evolution of social protection institutions in the region. They have helped to clear the ground for ‘effects of causes’ research leading to reliable causal inference. But perhaps the most significant methodological challenge is to extend the application of ‘effects of causes’ to the study of social protection institutions in the region. It requires shifting the focus of research from conditions to effects. The following sections introduce some of the relevant tools.
2.4 The Potential Outcomes Framework
The potential outcomes framework provides a foundation for causal inference (Imbens,
Where yi is the outcome for individual i; yi1 denotes the outcome if treated and yi0 the outcome if not treated.Footnote 9 Aggregating across the sample, it is possible to evaluate the sample average treatment effect (SATE).
Several assumptions are required to ensure validity for the SATE (Hernán, 2020). The ignorability assumption refers to the requirement that the random assignment is independent of the potential outcomes. This assumption can be formalised as t ⊥ y0, y1. The stable unit treatment assumption requires that the potential outcome for unit i depends only in the treatment. This implies the requirement that there are no hidden treatments and no spillovers among units.
The potential outcomes framework facilitates thinking through causality in the context of social protection institutions. It emphasises the role of counterfactuals, that is the road not taken.