Dr. Virginia Clinton joined us on episode 027 Reading and Metacognition to share her research on using open education resources (OERs), and examining how paper materials compare to digital materials in her courses.
Michael Ralph: For our first segment we welcome Dr. Virginia Clinton who is an assistant professor of educational foundations and research at the University of North Dakota. She teaches courses on research methods and on how people learn.
Laurence Woodruff: I read “Savings Without Sacrifice: A Case Report On Open Source Textbook Adoption,” where we compare the use of open source textbooks versus traditional commercial textbooks at a collegiate classroom.
M.R: I read, “Reading From Paper Compared To Screens, A Systematic Review And Meta-analysis” and those are only two of the papers that I know that you have put out recently. You’ve been doing a lot of work on research and reading so why don’t you tell us about how your work includes writing all of these many papers.
Virginia Clinton: Yes. I became interested in the open source textbook adoption because, like many faculty members, I was really shocked and a little disturbed by how expensive textbooks where. I started out teaching at community college several years ago. The cost to take my course was about $540.00 which is pretty reasonable. A lot of students who were lower SES. Who were just paying out of pocket. I was really shocked when I found out how much the textbooks cost because the textbook I had it up getting was $230.00. I basically increased the cost of my class by 50 percent with the textbook. It’s not like I picked a pricy textbook. That’s just what they cost. Moving forward I later became an instructor and I had a very heavy course load and I had large enrollment classes, including Introduction To Psychology. I had hundreds of students which made me a very appealing person to market to, by a publishing company. One week I had six publishing reps come to my office unannounced. I remember just thinking, “Is this really the best way for us to have students learn, or get the best materials?” I had looked into a lower priced textbooks. I had taught my students all kinds of different tricks for getting cheaper textbooks, but I just really didn’t feel like what they were getting was anywhere near worth the cost. I found out about open educational resources through an initiative at my institution. The University of North Dakota is part of the North Dakota University system which is getting more interested in open educational resources. I went to a training and I learned about the different ways they were made and vetted because I was really concerned about the quality. A lot of people have this idea that free means that they’re not any good. That definitely was my initial concern. I went. I reviewed a textbook very thoroughly. I wrote a review and it’s actually on the open textbook library at the University of Minnesota. I noted a number of mistakes and I contacted the publisher because they actually had a link where you could click on it and inform the publisher that there were mistakes. The publisher being Open Stacks, it’s a different model than with commercial publishing industry. They got back to me right away and they fixed it right away. One of the reasons they could is an electronic book so it’s easy to make changes and updates and corrections. They wanted to make sure that everything was fixed. When I had a few years ago contacted a commercial publisher to point out a mistake they just flat out were defining things wrong. It wasn’t a, “This is a controversy in the field.” It was, “This is inaccurate.” They contacted me back, “Thank you. Can we talk to you about this.” OK They called me up and they just started pushing for me to make my students pay for another one of their products. It was really upsetting. I adopted the textbook, but being a researcher, I really wanted to do a thorough examination and not just be like, “Well OK. It seemed like it went OK for me.” If I was going to do this kind of change I wanted to take advantage of the opportunity to really analyze it and use empirical data collection and look at student academic performance before hand and look at their grades after and see if there were any differences, if that $200.00 textbook actually made an improvement on their learning. I did the study and had the students report how much they spent on their textbook, how much they used their textbook, and also what they thought of the textbook. That’s a survey and I also compared grades in the course. To get an idea of how they did before the course, I got their high school GPAs. I’ve found that they, not surprisingly, spent a lot less money on the open source textbook. I asked them to estimate if you printed it out to estimate paper and ink costs, or if you bought a bound copy, estimate that. Or if you had to buy a tablet because you felt like that was the best way to read it, estimate those costs as well. They spent a lot more obviously on the commercial textbook. As far as use and perceptions, it’s about the same. They have the same ratings of quality and this was a nice addition to the literature because a lot of the studies beforehand that were really informative, were limited in that they asked students, “OK. You’re taking Econ 101 now and you have an open source textbook. Compare it to your commercial textbooks you’ve used before. You’d be asking somebody to compare their econ textbook with what they took last semester in history or sociology or chemistry. It’s a different beast.
M.R.: Different comparison, but also self report stuff over time becomes a lot weaker. Yeah that makes sense to me.
V.C.: And you also tend to like what you have right now. But this was students asking right now, “What do you think the textbook you’re using?” and comparing different students.
L.W.: One of the things that I liked about this was that there’s a kind of a “be the change” kind of feeling that I have here. You’re not comfortable with the commercialization of education. There are alternatives but we need to make a legitimate comparison and find some measurements that we can agree upon are meaningful about these alternatives. That’s something that I appreciated. What I’m interested in, you know, you are one of the variables held constant in this. You are teaching in the fall and teaching in the spring, and what was it like trying, I mean you had to come to decisions. The books are different so how do you control the directions that you’re giving them, or recommendations regarding how to use the books? When did you call for to use for the books? Was the material that you asked them to read about in this book highlighting it differently than this other book? And then, I’m sure, you probably have this internal, “Am I making sure that I myself am not acting differently between these 2 semesters?” How do you navigate those complications of trying to be a variable that you’re supposed to hold constant?
V.C.: Oh yeah. This is a strength and a weakness in scholarship and teaching and learning research, which is basically the same as action research for K through 12 teachers, except at the post-secondary level, you’re the researcher, so you have special insight into your classrooms but that’s also the confound. I’m not naïve to what’s going on and there is of course that concern, “Am I teaching differently?” I will say, If anything I wouldn’t have been teaching as well with the open source textbook because I was super pregnant, compared to the Spring. I wasn’t quite as energetic and engaged because I was exhausted and I actually gave birth two days after grades were entered in the fall. I also started a new position. I got a tenure line position. Personally, I feel even if I had wanted to teach better with the open source textbook, I don’t know if that would have even been possible. When I converted my course to the open source textbook, one thing I did is I tried to make it as similar. I know a lot of times converting to an open source textbook is a great opportunity for doing a course overhaul and restructuring and customizing and blending your class. That’s awesome. There’s lots of studies on that, but I really wanted to control it so the biggest difference would be that one semester there was a commercial textbook and another semester there was an open source textbook.
L.W.: One of the things that I actually appreciated about your writing, specifically, was it was really easy to read your literature.
V.C.: For that open source textbook adoption paper, I was extra careful to be as non-technical as possible because I knew it was not to a psych audience or an Ed research audience. It was very broad. It’s “Open Learning,” which is a really broad Journal.
L.W.: If there is a kernel of take away, what do you want people to take away after having consumed this research? What’s the message?
V.C.: I would say that my study, in conjunction with numerous other studies, I’ve actually did another meta-analysis, this time looking at open source textbook adoption. I included my study and every study I could find that was similar to it and I believe I found 25 different studies that qualified and in the Meta analysis with over 100000 students overall there was no difference. I think if you look at that and you look at so many other studies, there’s one or two where learning seems to be lower with an open source textbook, but one study is just that, it’s one study. So even my study is just one study. If you look at the research syntheses I’ve done on this, the pretty clear take home message is that open source textbooks reduce the financial burden of a college education without any real impact on learning, In terms of the difference between a commercial textbook. That is very practical direct, controllable factor that most post-secondary instructors have. They can help their students achieve their college goals by using an open source textbook, provided that there is one in that particular area. The work in K. through 12 really needs to be expanded on. It is starting to take off. I actually have a few citations for you. I saw your tweet this morning. In my meta-analysis review, I only could find one K. through 12 study that met my inclusion criteria. Actually, it’s kind of a discussion with my reviewers whether I should even include it just because that population is so different because they don’t pay for their textbooks. There’s only one study, I’d originally thought I’d have a cluster of a them and I could look at them separately and then look at…
M.R.: Why do you think that there is this gap in research for the effect of open source material with the K. 12 community because making University and post-secondary education more accessible to people is good full stop period. OK, but there’s also this financial crunch on what K-12 can do with the resources they have available and I feel like liberating some of the investment that they have to make on commercial textbook products, which is considerable. We look at that research a while back with Chattergee, in 2018. It’s a huge fraction of their budgets going to textbooks. We can liberate that money to do something else. That’s a big deal also. Why do we not have the research to help them do some of that.
V.C.: One reason is that researchers tend to be at universities. If you hear about a new initiative on your campus or you want to do a study with your own classroom most of the studies that I come across were instructors doing their own classrooms. Not all of them, but I would say a good portion of them, or they were instructional designers who had a colleague who let them use their data. It gets more complicated with K. through 12 because with K-12 you don’t have researchers in the building necessarily who are hearing about things and thinking, “Oh, you know we should really study that?” or we should really have the empirical comparison of that. It’s just kind of the nature of being on a college campus makes it more likely that somebody is going to be thinking the idea, “I need to do my dissertation and there is this Initiative on my campus. Maybe I could look into that for my dissertation.” So that’s one reason. The other big reason which I think is a huge motivating factor is that K-12, the user doesn’t see the cost. I have never seen the budget for my kids’ school. I’m sure I could if I really wanted to dig for it, but I have a feeling if I got sent home a bill say here is what your kids textbooks cost and every other parent got that and I had to write a check, I have a strong suspicion that there would be a lot more research in K. through 12 because people would really be clamoring to have it looked at. The little bit of research that has been done is similar to the post-secondary research. That it’s just as effective.
L.W: Now just to be, I don’t know, contrarian. There was no significant differences in terms of student gain between one textbook or another. Are you confident you would see a loss in gain if you just didn’t use the textbook at all? If there’s no difference between these two text books maybe textbooks don’t matter?
V.C.: That, yeah. Honestly, that is a really good question. That is something that I pointed out in my meta-analysis that one reason we’re not seeing a difference is maybe textbooks just aren’t that important. Really all this time and you are putting into it isn’t that critical. I mean if you think about valuable factors in student learning, the number one predictor of student success is the student. Yeah. Their background knowledge. Their motivation. Their prior training. That’s different than background knowledge, that’s knowing just kind of how school culture works and knowing how to interact with faculty and all these unspoken rules.
L.W.: Yeah. It kind of speaks to how to use, how to identify valuable resources from not valuable resources, and how to use those resources in that academic education space. Maybe part of that culture is the skill of knowing when this is a textbook I actually need, It’s going to be really good for me, and when to say this textbook is a waste of my time and I need to do something else in order to prepare myself.
M.R.: From my experimental standpoint that makes sense. We should get that control data does it even matter at all, but from an ethical perspective, can we? Especially because you’re working with convenience samples.
V.C.: The thing is, my hunch is, there’s a hypothesis called the access hypothesis that goes with this, is that there is a small, Non-zero, but not large percentage of college students who really do need that textbook, but really can’t afford it. They are benefited from the open source textbook. The thing is that is probably a fairly small percentage because these are the students who need the book or the materials in addition to going to class and hearing the professor. On top of that, can’t afford the textbooks, they’re the ones who really benefit from open source textbooks. I think there really is a critical number of students who need that, either for second language issues, that you know they need the information again because the information so challenging. I Remember when I took statistics I had to sit down with my textbook before class, go to class, and then review may notes after class in order to get the material, just because it was really challenging for me. I’m sure I had peers who probably could have never cracked open the textbook the whole semester and done fine, but I really needed it. There’s been other classes where I barely read the textbook because it wasn’t hard for me. I’m going to class was plenty so there are groups of students who really do need it. One reason I think that is if you look at the differences in withdrawal rate. If a college course is using an open source textbook, what my study found and what I found when I meta-analyzed all the studies I could find, that had course withdrawal rates and then also even followed up with researchers and said, “Hey do you have your withdrawal rate information that I can include?” There was a significantly lower withdrawal rate with an open source textbook. What I think that is that segment of students that needs the textbook but can’t afford it, or realizing that they were behind in the class and they just did not have the money to get a hold of the materials.
M.R.: I think it’s a really important point and in your paper you pointed out that the GPAs of those who were withdrawing in the 2 treatment groups were similar, but I may have overlooked it but what was the G.P.A. of the students who remained in the class? Did you see there were some students…
V.C.: There actually was a significant difference. The high school G.P.A. of the students with the commercial textbook was a little lower than the students with the open source text book. It was about the same difference as their overall grade at the end. I argue in my paper that once you factor in the previous G.P.A. it’s the same. The groups who remain were also high comparable G.P.As. If that’s true because that’s not what I predicted, but if that’s true then which students are students who would withdraw in the commercial textbook but who remain in the open source textbook, if you have as I think I heard you say a moment ago, the students who can’t afford textbooks don’t have the support so they leave, but are students in that state homogeneously distributed across all of the SES strata of the class?
V.C.: There is very little data on students who withdraw from class. The thing is they withdraw. They’re not there. It is hard to ask them. The handful of studies that I have found have shown that the cost of a textbook is a reason students say they withdrew from a class.
M.R.: There’s something else that I made in my notes that I wanted to ask because I just found it amusing maybe is the words to say there. In students who were comparing or who were reporting their perceptions of the text that were available commercial versus open source, you mentioned that there were some things that were comparable. Outcomes were similar and their usage cases were similar, but that the reports of their perception of the writing quality in the open source textbooks was slightly higher, but the top comment on commercial textbooks was how well it was written. I found that to be really interesting, those two things juxtaposed against each other.
V.C.: I think one reason I thought of is there just wasn’t much they could think of to wright about with what they liked. They certainly had a lot to say about what they didn’t like.
L.W.: Which actually kind of relates to this little bit. One of the many concerns about adopting open source textbooks in the first place is that the primary mode to interact with them is digital.
V.C.: And that was the number one complaint about the open source textbook is it was electronic and they didn’t want to pay for a paper copy, and I’m like, “Look, you’re not getting the free paper book.”
M.R.: Hold on. The number one complaint of student using the open source textbook was that it was digital?
V.C.: Yes.
M.R.: I overlook that.
V.C.: I believe it was.
L.W.: Yeah. That’s what I got to. I read your paper two hours ago, great depth, and that’s what I took away as well. That it was well written, but they would prefer not to have to access it digitally.
M.R.: Which makes sense. You also did a meta-analysis of comparing papers vs screens you you compiled lots of this research. Generally, we found that papers are superior to screens? Is that generally true?
V.C.: Yes. Looking aggregating over the studies, and these were all randomized experiments, the gold standard in research. There was a paper control and a screen version and it many studies, the paper and the screen conditions had to be the same. This was a little more sterile than real life, but they were not allowed to have access to the Internet. It’s not like now, where I walk by students studying and they have their phone next to them. But I found that overall there was a benefit of paper over screens in terms of comprehension. It’s the same if you’re looking at the memory for what you read so more literal comprehension, just memorizing it, but not really thinking about it. It’s similar for inferential comprehension, where you’re actually putting together ideas and having to come up with a conclusion that is not explicitly stated in the text but supported by the texts. Honestly, that surprised me a little bit. I thought going into this that students would do the same. My Ph D. work was on the psychology of reading and I got back into it the past few years with my faculty position. Just thinking about the theories of reading, it didn’t make sense to me that reading from a screen or paper, if all else is the same would make a difference. I had a lot of students say how they felt like they learned better from paper, or they didn’t want an electronic textbook, because even before I got into OER I would encourage students to get the ebook because it’s so much cheaper. My students were like, No I just don’t like the e-book. I really want the paper. OK that’s another $100.00.
M.R.: But it’s a not wasted $1200.00 right?
V.C.: Yeah. Apparently, it was not wasted. A book you don’t read it’s worthless.
L.W.: Right. Exactly.
M.R.: Or course you fail is a waste of $900.00.
V.C.: Yeah. Exactly.
M.R.: What jumped out to me in your meta-analysis because your conclusions are consistent with the other stuff that we’ve discussed and read and that’s all fine, but what I wanted to ask was your inclusion criteria was 2008 to 2018, Our presentation of material in the digital format is much more immature than the centuries we’ve been able to spend finding the right fonts, and the right margins, in the right organisation, and all the things that go into making material presentable and consumable on the printed page and a lot of has been just straight ported over into the digital space. Did your reviewers bring up, or what’s your what’s your thoughts on comparing something that is digital material in 2008 compared to 2018 and their usability so that it’s not that it is on a screen but that it is in a less productive format for that setting and we are still learning what that looks like. How does that factor in?
V.C.: First I want to clarify. I didn’t look at the date of publication and how the findings were changed, but there are actually two other, this is the year of meta-analyses between paper and screens, literally three came out, not including mine. All of our inclusion criteria were a little different. Our analysis approaches were a little different, and our specific research questions varied, but we all came down to the same conclusion, that paper had a benefit over screens for comprehension. Two of them looked at the date of publication, and one said that as time is going on,the gap between paper and screens is narrowing. The other one said the other thing they said it’s actually getting worse. I have not been able to really sit down and tease out the methods to figure out why but. To say it’s inconclusive is putting it mildly.
M.R.: I’m going to conjecture. I think that paper and screen are too big of bins. I think that saying that all screen and all digital media is too heterogeneous to lump together. It sounds like that’s kind of where you’re going as well.
V.C.: The whole point with the meta-analysis is you are consolidating the work from dozens of other studies, thousands of other participants, and you’re trying to simplify things to a single effect size that people can take and understand, Because there’s so much research out there that even professional researchers just can’t follow all of it. You certainly can’t expect the general public who has jobs outside of academia, and doesn’t read research for a living to keep up with it. That’s where it gets tricky as I’m going through studies and trying to figure out how to define digital tools and how to define the benefits of screens have that paper doesn’t have and try to figure out what is the best approach for looking at tools that screens have that paper doesn’t.
L.W.: I have some follow-up questions about that in terms of the other two meta-analyses that you referred to earlier, that we have not read, so I’m super happy that your gonna tell us about them. In your study you describe using a very sterile environment where the screen was essentially kind of like a locked browser, and that there was no Internet access, there were no distractions, there were no other things going on other than reading the documents. But then one of the alternative meta-analysis says the distance between screens and paper is widening and then the other said the distance between screens and paper is narrowing. Were the devices controlled? Were people reading on their personal devices? Were they reading on multitasking tooled devices? Were they reading on academic assigned devices? Were they reading in public computer terminals? In those two different findings, was the access of device controlled?
V.C: You know I actually have not gone through their methodologies really carefully. I know that the one by Kong and all, that was published in “Computers and Education.” They just looked at inferential comprehension. First of all, so that’s a little different than my study. Their inclusion criteria are similar to mine. The other meta-analysis, they included quasi-experiments so not as controlled environments but…
M.R.: That’s a big deal. I think it’s a big deal.
V.C.: Yeah but they also separated them out, but I can’t remember to be honest.
L.W.: I’m interested for my part in this, I am a pretty anti-screen individual. I don’t know. The goal of the reading, what do I want the goal of the reading to be, and then choose the format that is best suited to my goals. That’s what I want to choose and paper ends up being the one that resonates with me for my pursued goals more often than not. I’m interested in how this plays out of course in the case of public education space when teachers are choosing to let their kids access and read from their own personal devices versus devices issued to them by the school versus public devices that they don’t have any ownership or emotional connection to and I want to see where those layout. If there is a closing of the gap between paper and screen, but that is not consistent among all accesses, then we need to make decisions conscious about how the different tools are supporting our goals.
V.C.: I mean one point that I’d like to make is that if they were given a book like I was given a book in K. through 12 school, that wasn’t their book and they weren’t able to write notes in it, or highlight or market up in any way, whereas a screen device, depending on the particular app you’re using and the software involved, even if you’re renting it, they’ll usually let you mark it up. That said, there’s limited research on whether or not marking it up makes a difference, But thats a whole other topic.
M.R.: Dr. Clinton, we really appreciate you spending your time talking with us and sharing your research and helping us understand what all this means for reading and education. If there are listeners who want to consume more of the things you’ve created or more of the research you’ve done or just what you write, where can they find you on the inter webs.
V.C.: Well I’m on Twitter. V_Clinton. I also have my faculty directory website where I keep my articles updated pretty regularly or you can certainly email me if you have any questions about anything I’ve written or if you want a full text copy something and you don’t have access to it so it’s [email protected].