Abstract
Multivariate pattern analysis, or MVPA, has become one of the most popular analytic methods in cognitive neuroscience. Since its inception, MVPA has been heralded as offering much more than regular univariate analyses, for—we are told—it not only can tell us which brain regions are engaged while processing particular stimuli, but also which patterns of neural activity represent the categories the stimuli are selected from. We disagree, and in the current paper we offer four conceptual challenges to the use of MVPA to make claims about neural representation. Our view is that the use of MVPA to make claims about neural representation is problematic.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Quine sometimes uses the expressions “ontological relativity”, “indeterminacy of translation” and “inscrutability of reference” to refer to the same underlying notion. Although some philosophers tried to pull apart these ideas as different doctrines, later in life Quine made clear that these terms expressed the same idea (Quine, 1986).
The term intensional comes from mathematical logic, and refers to definitions that are not extensional. A set is extensionally defined when its elements are listed. Thus, the set of all even number is extensionally defined as the set containing 2, 4, 6, 8, 10 and so on. But one could also intensionally define the same set with the function f(n) = n. 2, where n \(\in \,{\mathbb{N}}\). Chisholm (1957) famously argued that intentional statements are also intensional because (1) they don’t admit existential generalization—viz., from “Ana believes Santa is stingy” it does not follow that there exists an x such that x is Santa and x is stingy—and (2) co-referential terms in intensional contexts cannot be substituted salva veritate—viz., I cannot substitute “Clark Kent” for “Superman” in “Louis Lane believes that Clark Kent is a coward” without changing the truth value of the whole statement. Critical for our discussion is the fact that two terms can be co-extensional and differ in their intensional definition. For instance, the set of all cordata (animals with hearts) is co-extensional with that of renata (animals with kidneys), but it is definitively different to entertain thoughts about cordata as opposed to renata (Quine, 1951). The elements in the set—the referent—do not determine how you think about them for, as Frege put it, “sense determines reference”: two expressions with the same sense will have the same referent, but two expressions with the same referent need not have the same sense (Frege, 1892).
So closely related they are, that a reviewer suggested to combine this challenge with the previous one. However, we decided to keep them separate. We reasoned that, for many philosophers of mind with a naturalistic bent, the issue of misrepresentation is critical (Neander, 1995), and it is likely that at least some of them may be unmoved by Quinean considerations about radical translation. In fact, a number of philosophers of mind and cognitive scientists have rejected Quine’s concerns (e.g., Chomsky, 1968; Soames, 1999), and yet they remain open to the possibility of naturalizing intentionality and representation (e.g., Fodor, 1990). Thus, although the points that the problem of cross-cut categories and the problem of misrepresentation make are closely related, the motivation behind them differs. We thank a reviewer for inviting us to clarify this issue.
Perhaps unsurprisingly, there are several websites dedicated to collecting pictures of buildings that look like living things (https://images.app.goo.gl/2NoL9eidmJtSPSQr7).
A reviewer suggested an intriguing possibility to try to circumvent—or, at least, ameliorate—the problem of misrepresentation, namely the inclusion of error trials in the analysis. In fact, as pointed out by the reviewer, this is a strategy that has been employed, for instance, by Woolgar et al. (2019) in a challenging stimulus–response task with a stable error rate. The reviewer’s suggestion is that these sorts of analyses may be able to tell us when a misrepresentation may occur by, say, “showing that patterns on trials where X is an error are similar to patterns where X is not”. Although intriguing, we remain somewhat skeptical. For one, error trials are extremely difficult to analyze, as it is often not clear why a participant erred in each trial. Perhaps in one trial a participant was confused, in another one tired, or even simply pushed the wrong button by accident. Indeed, it is because of the difficulty of determining the reasons behind error trials that most researchers simply discard them. A second reason to be skeptical, is that the few instances (we are aware of) in which error trials have been included in MVPA analyses, is for extremely tight experimental paradigms where there are no more than a couple of options to hit or miss. It is not clear how these kinds of paradigms and analytic strategies can scale up to larger stimuli set. Finally, even if, for the sake of argument, we were to assume that we knew why a participant makes an error—say, a participant incorrectly identifies a dog as a cat—we may still face similar difficulties as those pointed out in the current paper. After all, such a result will only reverse the map** for error trials (i.e., error trials will be thought to match the incorrect response and not the actual target), and thus problems like that of cross-cut categories and orthogonalization (discussed below) would still apply. Nevertheless, we acknowledge that our brief rebuttal is far from being a knock-down counterargument against this intriguing possibility. Further research is needed to evaluate the extent to which the inclusion of error trials can solve the problem of misrepresentation for MVPA.
Indeed, today it may be really hard to get a paper accepted using a SVM algorithm with a softmax loss function without the use of a neutral class, even though this was a common strategy just a few years ago.
For example, the theory of activity-silent working memory has begun to replace the earlier established hypothesis that working memory is dependent upon sustained neural activity. The activity-silent working memory hypothesis arose from theoretical work at the cellular level (Mongillo et al., 2008), but it was supposed to be later confirmed with EEG (Wolff et al., 2015) and fMRI (Rose et al., 2016) MVPA-style analyses via null results (i.e., the inability to decode the contents of working memory). Our argument in this section does not imply that something like the activity-silent working memory hypothesis is wrong; we simply want to suggest that the structure of the inference from null results alone to corroborate such a view is, if not inadequate, at best insufficient.
To be sure, the first two challenges likely apply to a weaker notion of representation as well. We are assuming here that partisans of a weaker notion of representation may feel unaffected by the first two challenges, but it is possible that don’t. We thank Zina Ward for mentioning this point.
References
Anderson, M. L., & Oates, T. (2010). A critique of multi-voxel pattern analysis. Proceedings of the Annual Meeting of the Cognitive Science Society, 32(32), 7.
Carlson, T., Goddard, E., Kaplan, D. M., Klein, C., & Ritchie, J. B. (2018). Ghosts in machine learning for cognitive neuroscience: Moving from data to theory. NeuroImage, 180, 88–100.
Carlson, T. A., Schrater, P., & He, S. (2003). Patterns of activity in the categorical representations of objects. Journal of Cognitive Neuroscience, 15(5), 21.
Carlson, T. A., & Wardle, S. G. (2015). Sensible decoding. NeuroImage, 110, 217–218. https://doi.org/10.1016/j.neuroimage.2015.02.009
Chisholm, R. M. (1957). Perceiving: A philosophical study. Cornell University Press.
Chomsky, N. (1968). Language and mind. Cambridge University Press.
Cremers, H. R., Wager, T. D., & Yarkoni, T. (2017). The relation between statistical power and inference in fMRI. Plos ONE, 12(11), e0184923.
Cummins, R. (1996). Representations. MIT Press.
Davis, T., & Poldrack, R. A. (2013). Measuring neural representations with fMRI: Practices and pitfalls— Representational analysis using fMRI. Annals of the New York Academy of Sciences, 1296(1), 108–134. https://doi.org/10.1111/nyas.12156
Dretske, F. (1986). Misrepresentation. In R. Bogdan (Ed.), Belief: Form (pp. 17–36). Content and Function.
Dubois, J., de Berker, A. O., & Tsao, D. Y. (2015). Single-unit recordings in the macaque face patch system reveal limitations of fMRI MVPA. Journal of Neuroscience, 35(6), 2791–2802. https://doi.org/10.1523/JNEUROSCI.4037-14.2015
Fan, M., & Chou, C.-A. (2016). Exploring stability-based voxel selection methods in MVPA using cognitive neuroimaging data: A comprehensive study. Brain Informatics, 3(3), 193–203. https://doi.org/10.1007/s40708-016-0048-0
Fodor, J. A. (1987). Psychosemantics: The problem of meaning in the philosophy of mind. MIT Press.
Fodor, J. A. (1990). A theory of content and other essays. MIT Press.
Frege, F. L. G. (1892). Über Sinn und Bedeutung. Zeitschrift Für Philosophie Und Philosophische Kritik, 100, 25–50.
Gauthier, I., Skudlarski, P., Gore, J. C., & Anderson, A. W. (2000). Expertise for cars and birds recruits brain areas involved in face recognition. Nature Neuroscience, 3(2), 8.
Gauthier, I., Tarr, M. J., Anderson, A. W., Skudlarski, P., & Gore, J. C. (1999). Activation of the middle fusiform “face area” increases with expertise in recognizing novel objects. Nature Neuroscience, 2(6), 568–573. https://doi.org/10.1038/9224
Goddard, E., Klein, C., Solomon, S. G., Hogendoorn, H., & Carlson, T. A. (2018). Interpreting the dimensions of neural feature representations revealed by dimensionality reduction. NeuroImage, 180, 41–67.
Grice, H. P. (1957). Meaning. The Philosophical Review, 66(3), 377–388. https://doi.org/10.2307/2182440
Haxby, J. V. (2001). Distributed and overlap** representations of faces and objects in ventral temporal cortex. Science, 293(5539), 2425–2430. https://doi.org/10.1126/science.1063736
Haxby, J. V., Connolly, A. C., & Guntupalli, J. S. (2014). Decoding neural representational spaces using multivariate pattern analysis. Annual Review of Neuroscience, 37(1), 435–456. https://doi.org/10.1146/annurev-neuro-062012-170325
Hubel, D. H., & Wiesel, T. N. (1959). Receptive fields of single neurones in the cat’s striate cortex. The Journal of Physiology, 148(3), 574–591. https://doi.org/10.1113/jphysiol.1959.sp006308
Johnson, M. R., & Johnson, M. K. (2014). Decoding individual natural scene representations during perception and imagery. Frontiers in Human Neuroscience. https://doi.org/10.3389/fnhum.2014.00059
Kanwisher, N., McDermott, J., & Chun, M. M. (1997). The fusiform face area: A module in human extrastriate cortex specialized for face perception. Journal of Neuroscience, 17(11), 4302–4311.
Konkle, T., & Caramazza, A. (2013). Tripartite organization of the ventral stream by animacy and object size. Journal of Neuroscience, 33(25), 10235–10242. https://doi.org/10.1523/JNEUROSCI.0983-13.2013
Konkle, T., & Oliva, A. (2012). A real-world size organization of object responses in occipitotemporal cortex. Neuron, 74(6), 1114–1124. https://doi.org/10.1016/j.neuron.2012.04.036
Lent, R., Azevedo, F. A. C., Andrade-Moraes, C. H., & Pinto, A. V. O. (2012). How many neurons do you have? Some dogmas of quantitative neuroscience under revision. European Journal of Neuroscience, 35(1), 1–9. https://doi.org/10.1111/j.1460-9568.2011.07923.x
Loewer, B. (2017). A guide to naturalizing semantics. In B. Hale, C. Wright, & A. Miller (Eds.), A companion to the philosophy of language (pp. 174–196). Wiley.
Mongillo, G., Barak, O., & Tsodyks, M. (2008). Synaptic theory of working memory. Science, 319(5869), 1543–1546. https://doi.org/10.1126/science.1150769
Naselaris, T., & Kay, K. N. (2015). Resolving ambiguities of MVPA using explicit models of representation. Trends in Cognitive Sciences, 19(10), 551–554. https://doi.org/10.1016/j.tics.2015.07.005
Neander, K. and Schulte, P. (2021). Teleological Theories of Mental Content. The Stanford Encyclopedia of Philosophy, Edward N. Zalta (ed.), forthcoming URL = https://plato.stanford.edu/archives/spr2021/entries/content-teleological/.
Neander, K. (1995). Misrepresenting & malfunctioning. Philosophical Studies, 79(2), 109–141. https://doi.org/10.1007/BF00989706
Norman, K. A., Polyn, S. M., Detre, G. J., & Haxby, J. V. (2006). Beyond mind-reading: Multi-voxel pattern analysis of fMRI data. Trends in Cognitive Sciences, 10(9), 424–430. https://doi.org/10.1016/j.tics.2006.07.005
Pereira, F., Mitchell, T., & Botvinick, M. (2009). Machine learning classifiers and fMRI: A tutorial overview. NeuroImage, 45(1, Supplement 1), S199–S209. https://doi.org/10.1016/j.neuroimage.2008.11.007
Quine, W. V. O. (1960). Word and object. M.I.T. Press.
Quine, W.V.O. (1986). .Reply to Paul A. Roth. in Hahn and Schilpp (eds.), The Philosophy of W. V. Quine. Peru, IL: Open Court. pp 469–461.
Quine, W. V. O. (1951). Two dogmas of empiricism. Philosophical Review, 60, 20–43.
Rao, R. P. N., & Ballard, D. H. (1999). Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, 2(1), 79–87.
Ritchie, J. B., Kaplan, D. M., & Klein, C. (2019). Decoding the brain: Neural representation and the limits of multivariate pattern analysis in cognitive neuroscience. The British Journal for the Philosophy of Science, 70(2), 581–607. https://doi.org/10.1093/bjps/axx023
Rose, N. S., LaRocque, J. J., Riggall, A. C., Gosseries, O., Starrett, M. J., Meyering, E. E., & Postle, B. R. (2016). Reactivation of latent working memories with transcranial magnetic stimulation. Science, 354(6316), 1136–1139. https://doi.org/10.1126/science.aah7011
Serences, J. T., & Boynton, G. M. (2007). The representation of behavioral choice for motion in human visual cortex. Journal of Neuroscience, 27(47), 12893–12899. https://doi.org/10.1523/JNEUROSCI.4021-07.2007
Soames, S. (1999). The indeterminacy of translation and the inscrutability of reference. Canadian Journal of Philosophy, 29(3), 321–370.
Sullivan, J. A. (2010). A role for representation in cognitive neurobiology. Philosophy of Science, 77(5), 875–887. https://doi.org/10.1086/656818
Visser, R. M., Scholte, H. S., & Kindt, M. (2011). Associative learning increases trial-by-trial similarity of BOLD-MRI patterns. Journal of Neuroscience, 31(33), 12021–12028. https://doi.org/10.1523/JNEUROSCI.2178-11.2011
Wang, J., Baucom, L. B., & Shinkareva, S. V. (2013). Decoding abstract and concrete concept representations based on single-trial fMRI data. Human Brain Map**, 34(5), 1133–1147. https://doi.org/10.1002/hbm.21498
Weaverdyck, M. E., Lieberman, M. D., & Parkinson, C. (2020). Tools of the Trade Multivoxel pattern analysis in fMRI: A practical introduction for social and affective neuroscientists. Social Cognitive and Affective Neuroscience, 15(4), 487–509. https://doi.org/10.1093/scan/nsaa057
Wolff, M. J., Ding, J., Myers, N. E., & Stokes, M. G. (2015). Revealing hidden states in visual working memory using electroencephalography. Frontiers in Systems Neuroscience. https://doi.org/10.3389/fnsys.2015.00123
Woolgar, A., Dermody, N., Afshar, S., Williams, M.A., & Rich, A.N. (2019). Meaningful patterns of information in the brain revealed through analysis of errors. bioRxiv 673681. https://doi.org/10.1101/673681.
Acknowledgements
Previous versions of this paper were presented at the 43rd Annual Meeting of the Society for Philosophy and Psychology at Johns Hopkins University, the workshop on Locating Representations in the Brain at Stanford University, and the meeting of the Deep South Philosophy and Neuroscience Workgroup at the Central APA. We thank the participants in these events for their feedback. We also thank the members of the Imagination and Modal Cognition lab at Duke University for many helpful discussions.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This article belongs to the topical collection “Interdisciplinary Perspectives on Locating Representations in the Brain”, edited by Jessey Wright and Sarah Robins.
Rights and permissions
About this article
Cite this article
Gessell, B., Geib, B. & De Brigard, F. Multivariate pattern analysis and the search for neural representations. Synthese 199, 12869–12889 (2021). https://doi.org/10.1007/s11229-021-03358-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11229-021-03358-3