Elisabeth Tipton visited the show on episode 080 Growth Mindset & Music Distractions to discuss her recent paper in a special issue debating the efficacy of growth mindset interventions (and their research). We hosted Dr. Tipton as a direct follow-up to our previous episode 079 Meta-analyses: Co-teaching and Growth Mindset.
Laurence Woodruff: Hello, my name is Laurence Woodruff. And I wish I had more time to play Starfield.
Michael Ralph: And I’m Michael Ralph, and I should probably spend less time playing Starfield.
LW: Professional development requires ongoing reflection and dialogue.
MR: So join us as we spend our Saturday discussing education research and drinking beer.
LW: Today, we’re drinking the Rochefort trappist, triple extra
LW: in the second…
MR: Oh, whoa.
LW: I didn’t. Yeah, that happened to me, too. Oh my gosh.
MR: So listeners, we opened our beers. And for the first time, actually, now that it’s happened, I’m kind of shocked that it’s taken us seven years for this to happen. Both of our beers erupted in our hands. I’d like to tell you, we’re usually more professional than this. But we’re not. This is exactly what the show is like.
Elizabeth Tipton: I love it. It’s a good beer. Yeah, I was gonna say I mean, I am really bad in that, what I’m saying is I recognize the bottle and I probably had it but could I put that label to a taste? Probably not. But I would say my husband is, about a decade ago, went through a pretty heavy like, Belgian beer period where we drink a lot of Belgian beer.
We even went to Belgium, we went to a place in, I think it’s in Brussels called beer world that has hundreds of bottles of beer of different kinds.
LW: That’s a podcast goals, Ralph, is reading some European education research in a European beer garden.
MR: I think we’re probably going to use some of that tape because that was fun. And so why don’t we…., Laurence want to roll us forward?
LW: What are we doing today, Dr. Ralph?
MR: This month, we talked with Dr. Elizabeth Tipton, about the research support for growth mindset interventions, and the flaws and last month’s meta analysis. Together, we consider how growth mindset should be part of a more comprehensive approach to helping students improve.
MR: Later, we read how listening to music reduces our ability to use our working memory for academic tasks. Their laboratory study shows music has a cost, but we wonder whether the cost of background classroom distractions might be higher.
MR: Let’s get started.
MR: For our first segment, we read why meta analyses of growth mindset and other interventions should follow best practices for examining heterogeneity commentary on McNamara and Burgoyne, and Burnett, et all.
LW: This was written by Elizabeth Tipton, Christopher Bryan, Jared Murray, Mark McDaniel, Barbara Schneider, and David Yeager.
MR: And this segment, you may notice, is very closely related to the segment that we did last month where we read one of those meta analyses. And some of the feedback and engagement that we got from some prominent researchers in the field, including folks whose work was commented upon in that meta analysis suggested that we bring on a guest who knows a lot about this topic so that we can discuss it even further. And so to that end, we have with us as a guest, the lead author of this paper, Dr. Elizabeth Tipton, welcome. ‘
ET: Thank you. I’m happy to be here.
MR: Dr. Elizabeth Tipton is a professor of statistics at Northwestern University and the co-director of the statistics for evidence based policy and practice Center. She is a statistician who works in education and social science, who is interested in thinking about how we use evidence to inform what we do in education. I cued up this paper because we were told to read this paper by folks who wanted us to have a better understanding of what was going on in the growth mindset literature. So Dr. Tipton, can you tell us maybe in broad strokes, why do you work on growth mindset stuff?
ET: Oh, thank you, man. It’s a long story. But I’ll try to be brief. I have been interested in evidence-based practices in education for a long time, I do a lot of work, thinking about how we design studies in order to test interventions. And I’ve been particularly interested in how we design such studies to test interventions in the environments in which the evidence will be used. So you might call it generalizability, or external validity or at scale. But I want to know, not just does this work in some schools that I’ve picked in my neighborhood that are willing to be in my study, but will it work in schools in other parts of the country as well as around the world? So
to that end, probably back in 2014, or 2015, I was connected with David Yeager, who was planning a national study of learning mindsets, which was a big evaluation of growth mindset intervention in a random sample of us high schools, which is maybe the only study of this type. We randomly sampled. Think ultimately, we ended up with like 79 high schools in the United States in the study, and then randomly assigned students to receive either a growth mindset intervention or something comparable. But they were very interested in these generalizability issues, the random sampling, maybe making sure we over sampled certain subgroups that they thought it was likely to work with. So certain kinds of students at risk students, so they really thought this intervention would work with, as well as kinds of students in schools that they thought it wouldn’t work to really test a hypothesis about, under what conditions of growth mindset really matters. So there’s as much a longer answer than you probably wanted. And, and so I got involved with this community of scholars who were, I thought, trying to really do the cutting edge best practice science. They wanted to test not just does it work everywhere, but does it work under these particular conditions, and even doesn’t maybe not work in these other conditions. So I got involved with a group, several of whom are authors on this paper, that was multidisciplinary. And we did this study, I think the results of that study came out in nature in 2019. And then I also do work in meta analysis. So I also have been working as a statistician in methods for combining results across studies, to try to make sense of a literature in which some papers say things work, and some in which they don’t. And I had been kind of starting to be on a sort of a tear of being kind of a grumpy, middle aged professor who’s like, why, why are people doing bad meta analyses? Why are they not doing what we know are the right kinds of things to do. And in particular, my gripe in meta analysis is about people focusing on the average effect. So David Yeager contacted me and said, Hey, there’s these two meta analyses that are going to come out and they’re competing. And I want you to look at them, I think that one of them is really poorly done. And the other one seems to me to be one you’d be happy with. Will you think about this and figure out what’s going on? And so that’s what I did.
MR: You mentioned that national growth mindset study, and we did read it and discuss it on a segment of the show back, Episode 32, or something like that. It was a long time ago, but it’s in the show notes. So we did a segment on that big old paper, and we were very impressed with it. But at the time, what are the implications of these two competing meta analyses? And what you’re seeing, especially with your expertise on thinking about policy implications, what do we say to classroom teachers, who many, many folks have been excited about growth mindsets have seen positive impacts in their classrooms? And now they’re asking, do we keep going? Do we change the course, what do we do with all of this?
ET: My sense of the literature, and from this meta analysis, as well as from the National Study, which I think are fairly consistent findings across both, is that the growth mindset is the kind of intervention that works if you’re the right kind of student that if you are a student who’s already getting A’s, already taking hard classes, you could think of this as you’re already doing well, so how am I going to improve that with a growth mindset? I mean, maybe I can improve many aspects of that, but it’s going to be hard for me to see improvement, we might call that a ceiling effect. There’s not, you can’t get something better than As. So it’s gonna be hard for us to see if that made an improvement. On the other hand, if you’re at the very bottom end, where you don’t know how to read, but you’re in high school, for example, no amount of me telling you, “Work harder, your brain can grow” without extra resources is going to help you. But if you’re a student that’s sort of in the middle or that is struggling, and that sort of maybe has given up on school, who think, “Well, I’m not smart, I’m not good at math. That’s why I’m not doing well at this,” that the intervention works. It actually can, though, and the intervention can help students see that they don’t need to give up, that learning involves failure, learning involves, you know, some sort of muscle pain things are growing, and that putting in the work can pay dividends. And I think that’s partly, so it’s sort of an intervention that works when it needs to and when it doesn’t work. It’s not it’s not. It’s not a negative effect. It’s not like hurting people. It’s either helping you or it is a neutral.
LW: I’m glad that you mentioned about the ceiling effect, because we don’t really even know that it’s not helping them. Just the things that we’re measuring, We can’t see that they’re not. And so if you can imagine a type of student who is a high achieving fixed mindset student who has been maybe skating through school, but they haven’t actually hit a challenge that has made them question their abilities. Well, assuming they continue on to higher education or even postgraduate education, they can and, hopefully in their life, they will hit something that is new to them, because that’s where the growth will be either in the sum of human knowledge as they’re finding out new things, or just personal growth, where they hit something like, wow, this isn’t easy for me. And so, the possibility that these have effects later, which we, you know, I don’t know, I don’t design studies, I’m not really invested in that. So I don’t, I don’t know how we would measure it. It’s not easy to measure downstream psychological effects. And we know that from papers, we’ve read that say that when you do, this is slightly different. But when you directly teach elementary school students self regulation techniques and practice, they don’t show statistically significant increase in changing behavior until two years later, but they do! If you don’t do them, they don’t change their behavior. If you do give them the interventions, they will change their behavior two years later. So when you have the like, well, it helps these kinds of kids, but not those it’s, maybe. Maybe it doesn’t help those, but maybe it does in ways that we haven’t figured out how to assess yet.
MR: Well, and I appreciate that you’re referencing that paper, I cued up another one from just a couple of months ago, where we were reading this was now reading research, where they had promising previously validated interventions for reading instruction, that when you go from a contextualized, responsive intervention application to a global, like, everybody does this all the time, kind of an intervention, then the effects disappeared, and they’re like, so what’s good for some is not is very, very rarely good for everybody all the time. Okay. And that to any educator is not going to be surprising, like people who listened to show like, oh, it depends, is kind of always the answer. And like we laugh about that in the research world sometimes. But it is absolutely true that education is a social and contextual endeavor. And so like, I don’t think, you know, maybe somebody can read back the discussion from last month, I don’t we were never all that interested in this idea that the impact of growth mindset, as it is understood in the literature right now is zero. We were never, we were never on that boat.
LW: And that wasn’t even like, that’s not even what I thought the paper was, I thought the paper was telling us that it doesn’t really matter if the kids believe whether their intelligence can grow or not. The effect is probably coming from their commitment and scaffolding of effort. And I’m like, so don’t change anything that I’m doing in the classroom. And as a consequence of this paper, even if we have no problems with the findings, like the practicality of improving student cognition and developing their abilities, and helping them achieve more, all comes from the same behaviors. So it’s not really as in and like that sidesteps the question of anti essentialism, right? Like you still want to teach people that they are not reduced to certain qualities. And so like, you can still say, hey, that smart was kind of a fixed mindset compliment, can you think of another kind of compliment that you can use to describe the situation. It doesn’t challenge me to change that behavior, either. So whether or not the kids have a particular, like, definition, semantic understanding of what intelligence is and able to do is kind of irrelevant for my goals. So even if they had the best methods on Earth, their findings weren’t really influencing me in a specific way.
MR: This does lead me to a question from one of your comments a moment ago, but how there’s no there’s no cost. There’s no harm. We’re not harming students by doing some of these growth and growth mindset interventions. And while generally that’s true, you don’t see a decrease in performance, I would argue there is an opportunity cost, especially when a piece of their consideration was how many how many, like capitalist products have been created around growth mindset material, and so there are dollars that you can be spending on any number of things. And one thing that I noticed in your commentary that I’d like you to comment on is a lot of your particular commentary was focused on the measurement considerations being as our that’s your area of expertise. But I read in the McNamara review, that another major criticism was the operationalization of growth mindset and how the intervention is very often not one intervention, or one that is consistent. And they call out a couple of specific examples where mindset is, as it is narrowly or more concretely defined, as my view of whether my intelligence is malleable is very close. You know, it’s so proximal and interwoven with ideas of grit and ideas of what’s the role of practice and the quality of practice and like all. Have those things that are out there? Like, again, I think an educator wouldn’t be nodding right now like, I feel like yeah, all those things are sort of part and parcel of if we’re going to go towards a growth mindset. However, if we are not precise in defining in the research, which elements of it are having an impact, I can imagine school districts who spend all their time and energy trying to persuade a student, you can get smarter, you can get smarter, you can get smarter. And they’re not attending to the fact that they’re giving bad practice. And so what they’re doing is not actually leading to improvement, and it’s undermining their credibility. Do you have any comments on their critique of the operationalization of a growth mindset?
ET: Yeah, that’s a great, that’s a great question. So this is actually a real problem for most interventions. So, you know, most meta analyses, most like combinations, looking across the literature, what is hard, is that the intervention is not exactly the same. In every study, it’s not measured exactly the same. It’s not implemented exactly the same. And that’s, you know, that’s just a fact of education in the social sciences, compared to medicine, where it’s like, take the drug or don’t take the drug. Right. Like, that’s like a pill that you take. Here. It’s different. And it’s, it’s hard. I mean, it’s a hard problem. I mean, I think it’s also a hard problem once you move to school. So I think it’s not surprising that a psychologist would make this critique, because the very scientific critique of science, I want to isolate what exactly part of this intervention is the cause? Or is not the cause of this effect. But the reality is, once you move to schools, it’s very hard to isolate that exact mechanism in the same way, because you’re not going to get a growth mindset. In the you know, there are a variety of ways that you can do, you know, very variety of programs and a variety of approaches that can be built into an English curriculum, it can be on its own, there’s a lot of different opera operationalization of it. And so I think, to some degree, I think for the scientists that matters a lot, I’m not sure it matters that much the nitpicky parts of that as much for the people actually making decisions are using it. Now, you had another point in there that I think was also really important, which is that people are spending money on this and time on this. And so there’s an opportunity cost, right, like what I’m doing here, I’m not doing something else. And I think there’s a couple of things within that. The intervention itself is actually a very brief intervention, right? It doesn’t have to be something that takes a lot of time. The intervention also, from what I’ve learned, and this is very much me sort of extrapolating some things that I learned from the national study in learning mindsets, but also from some other conversations with people in the field, is it’s not enough just to tell kids that, you know, you can do better, like, you know, this is the critique of grit, too, right? Like you don’t, you know, it’s not enough to be like, Yeah, you’re really struggling. And you’re in this under-resourced school, and you’re from a historically excluded population, and you’ve got all of these things working against you, but you should just know that your brain grows, and that’s going to solve it right. That’s, that’s not how I’ve seen it. operationalized by the scientists doing this, it’s more, it’s supposed to be more your brain can grow. And I’m going to help you learn what the resources are to help your brain grow. So there’s a behavior component to it, there’s not just the belief, the belief and the words are not enough, you need the behavioral component to it as well. And that’s about being able to seek out resources, seek out learning, like seek out the things that you need in order to learn and have in that means for schools, they have to can’t just have the words, you have to have the resources available in the environment in which you’ve set up classrooms, not just in what you say that kids can, their minds can grow, but also that you’ve shown that in the behaviors you set up in the classroom, right? So if I tell you, your kids, yeah, kids, your brains can grow. But then I have only high stakes tests. And all of the decisions are based on these high stakes tests in the classroom. And I put all the smart kids, “smart” working quotes over on one side, and I’m ranking kids continually in the classroom, no matter how much I say, I believe in growth mindset, the kids can see from my actions that I don’t actually believe that they’re in growth mindset. Right? So there has to be that classroom activities, allowing kids to, you know, learn things, make mistakes and redo them, for example, that really shows it’s a culture of growth mindset. So that’s one part. And then the other part, I think we’ll keep adding on more parts. The other part is, I think that’s a good question. In general for interventions. You mean this question of I have to buy this one or that one. Some of them are just a framework or a way of thinking about the school I think you know, about the, you know, the environment you’ve created in your school, whereas others are much bigger commitments. You know, I’m buying a reading program and it’s going to be expensive and time can, you know, take a lot of time. And so there is a reality in which people are comparing things. And I don’t think scientists do a great job at that, thinking about that we’re often thinking about our one intervention versus something else, and not thinking about the whole environment.
MR: I want to do just a quick response. And then again, I know that I’m taking up more Mic time than I intend, but then some of your comments, just now, especially around the operationalization. And like it, things are messy, right. And I really want to carry that that water right, as a researcher, I really, I want to continue to identify with practitioners and as a practitioner, and embracing the, ecologically nested nature of what we’re doing, as opposed to the sterilized experimental nature of how many psychologists are trained, I just, I want to, I’m with you, I really am with you. But I also think that if I’m understanding with, even though I’m not formally trained in meta analyses, I really want to understand some of that mismatch and operationalization, which I suspect explains some of the heterogeneity and success of the interventions. And so I think from a research standpoint, accepting that distribution is an important thing to understand in the social sciences. And, to your credit, specifically, your paper makes that argument, I think, as clearly as anything I’ve ever seen in my life. Like, if I still taught methods, I’d use that paragraph. It’s a great explanation. And I think that that’s true. And I think that better clarifying what exactly is being measured in individual studies, will allow for future meta analyses to better explain some of that heterogeneity, because from a grounded perspective, somebody who is in a lot of classrooms, thinking about those research characteristics, I think it explains a lot. I think that the quality of the interventions, and at really subtle levels, one of the notes that I made is that I would pay folding money to read a good qualitative study of how some of these, very, very lightweight interventions of growth mindset, or otherwise, some of these very lightweight interventions, what does it look like, when a school adopts them? What are some of the ripple effects on what they touch? Or do not touch? And how do those things happen? In our discussion last month, I think one of the things that Laurence and I were kind of thinking about was so much of this hinges on if we’re doing a growth mindset, you know, 20 minute intervention with with students, but does that prompt the teacher to be thinking differently about what kinds of activities are providing in the classroom, and what kind of language they choose in their student interactions, some of those things that are incredibly subtle, that are really difficult to quantify, and are incredibly important for what’s going to happen at a cultural level and ultimately, for individual learners in that room and their success or lack thereof. And so I think that is both really, really important and incredibly difficult to quantify, which I think is consistent with some of your comments. And so I don’t think that the quibble about how we operationalize growth mindset is purely a scientific question. I was telling Laurence and some of our pre-recording time, there are examples from my work my life this month, where I, I’ve been sitting in on discussions where we’re going to measure growth mindset. And I had to be like, well, this is not actually growth mindset, like this is important. This is what we care about. This is not actually growth mindset. And we need to do a more precise description of this thing. So that we can go and measure it well, and then we can report it faithfully. So that then a future meta analysis doesn’t get it wrong. So I think, I think that whole body of critique, I do think it’s got some merit. I do, so don’t know.
ET: No, I would actually say that’s true. It was making me think about implementation fidelity of implementation just in general, and adaptation, right? So often people schools often will take things in the name, adapt them to their environment. And a lot of what we’ve studied in Encino, in scientific studies are very clean versions of an intervention. We’ve really made sure it is the full intervention, but we don’t have studies of those other things. So I was just part of a national academy of science, engineering and medicine companies, some committee that looked at the future of education research and sort of sent me these arguments for what the future of education research should look like. And one of the things we called out in that report was adaptation and implementation, being able to tell people not just this is what this evaluation found under ideal conditions. But what schools really need to know is sort of what are the things that are essential about this intervention? What are the things that are kind of oh, you can do them or you can’t do them. But if you don’t do these essential components, you’re not going to see the results of this. And so that’s asking researchers to really think about mechanisms and hone in on which things are essential components and which ones aren’t. So, which I think is great. I think it’s actually very important. The other thing I was gonna say as a little bit of a story, which is sort of hits this, this attitude versus behavior component. In the national study of learning mindsets, we had a series we asked teachers, so there was both a student component. And then of the sampled students, we then looked at, we found who their math teacher was, and then math teachers did like, a bunch of stuff, too. And so the math teachers, we asked them, you know, have you read Carol Dweck’s book? Are you familiar with the growth mindset? Do you have a growth mindset, right, all of these questions kind of about attitudes. But then we also asked two other things that we did one, we asked them to give them these like case study things and ask them how they would handle them, which was sort of getting at behaviors. And the other was, we asked students in their math class, do you think your math teacher has a growth mindset?
And, and I feel like, you know, where this is gonna go, right? That’s not much of a relationship, right? I read the books, and I think I have a growth mindset. Now, that’s not actually enough, that very often people do those, and then the behaviors that they say, and what their students say, Do not map on to that. So to me, that points out that you can change, you can read a book, you can think you’ve changed your ideas. But if you’re not changing your practice, then who cares, right? Like, who cares, the intervention is not just in your brain, it has to be also in your class.
LW: When I read your paper, this paper, it reminded me of something of another paper we read a while ago, and that, in your conclusion, you kind of made a comment about how there have been improvements to statistical analysis approaches that have existed for like 30 years, and people are still not using them. And that we feel that in education too, because sometimes, you know, you’re at this period of your life, or you’re deeply entrenched in learning about education, and then you transition to practice. And then you phase out of that, and you kind of get crystallized at this particular place. But you’re not burned out. So you know, you’re, you’re still doing pretty good. But it can be frustrating when you see practice that you know is outdated. So I wanted to identify with that a little bit. And then another just specifically, I appreciated that in your approach. You said, Okay, we’re going to take the data of this paper, and then we’re going to reanalyze it with these other modern approaches to see what happens then. And that reminded me of the I don’t have any idea how long ago we read this, but the summer slide paper Ralph, where we took the stats that like the Yeah, the 70s. You know, they’re like, Hey, man, our kids are just terrible. They have three months off in the summer, and then they come back and they don’t know anything. And here’s all the stats to prove it. And they reran all that data with modern analysis, and like now there’s not really a summer slide. That’s not really a real thing. So it’s just like, we can look at old information with better tools to get better information. I mean, it’s true and right. Archaeology, right. It’s true in molecular science, and it can be true in data analysis as well. So I just wanted to say I appreciate those perspectives.
MR: Thanks for coming on. This has been a really, really interesting and thought provoking conversation. I’m looking forward to sharing with all of our listeners. If there are folks out there who have enjoyed hearing your perspective on using statistics and using research and policy, where can they find more of the kinds of work that you do?
ET: Oh, thanks. I’d say the easiest is to go to my website, which is www.bethtipton.com. Or to just Google Elizabeth Tipton at Northwestern and you’ll find my university websites.