Do fines make a difference in people's behaviour? This simple question leads to a labyrinth of research, the intersection of law and economics, and the importance of replication in the social sciences in a great conversation with the author of our Constitutional module in Law 201/701, Introduction to Canadian Law, Cherie Metcalf.

If you enjoy the podcast, take a moment to subscribe! All of our courses are rooted in legal research, but you can learn more from Cherie about Constitutional Law in Canada through her module in Law 201/701, Introduction to Canadian Law, at takelaw.ca.

Sign up for our mailing list on the Certificate in Law site, and subscribe to this show on any of the major podcast platforms: Apple, Stitcher, Spotify and Google Play. Search for "Fundamentals" in your app of choice!

Theme music for Fundamentals by Megan Hamilton. Art for the podcast by Valérie Desrochers.  

Transcript:

[music]

00:03 Matt Shepherd: Welcome to Fundamentals of Canadian Law. I'm Matt Shepherd and sometimes I run late. Sometimes I'm punished for running late: Bad marks, library fines, all kinds of things. We've all been late for things and most of us have been punished for it in one way or another. And that common life experience is our gateway into an amazing conversation about legal research with the creator of the constitutional law module in Law 201/701, Cherie Metcalf. Cherie is both an economist and a legal academic, and she's been looking into a kind of legendary legal study that examined the consequences of fines from lateness. The original study discovered something surprising. But when Cherie and her team looked at it, applying the principle of replication to a legendary piece of academic literature, the results were even more surprising.

00:57 MS: Legal research is fascinating stuff and it really does affect all of our lives. Keep listening to find out how and why. This podcast is not legal advice and is being presented for informational purposes only. Fundamentals of Canadian Law is brought to you by the Queen's Certificate in Law, the only online certificate in law offered by a law faculty in Canada. You can find out more at takelaw.ca.

[music]

01:29 MS: Most law students know faculty as teachers because from the first day of law school, you're in the classroom teaching, but they don't have as much exposure, I think, to faculty as researchers, which is actually on paper, when you look at the allocation of time, it's actually the bulk of your job.

01:45 Cherie Metcalf: Yeah, yeah. Research is a really big part of what we do at the faculty and, you know, most faculty doing research is a big part of the reason why they want to be legal academics.

01:58 MS: Right.

01:58 CM: So yeah.

02:00 MS: And your research in particular is inter-disciplinary. You came to law from an economics background.

02:05 CM: Yeah, that's right. So I... It was a little bit unusual because I actually did my PhD in economics first before I ever went to law school at all, just sort of bizarre personal circumstances but I ended up doing both my JD and Master's in law after having gone all the way through advanced training in economics.

02:27 MS: Right. And law and economics is a thing. Like, this is...

02:30 CM: Oh, yes.

02:30 MS: Sort of a sub-discipline of law is the research into law and economics intersecting.

02:34 CM: Yeah, yeah. So using sort of economic tools and theory to understand how law works and how it influences the way individuals make choices and to use the statistical tools that economists are familiar with to try and study problems about the law and how it works, this is a very common field of study in legal academia. It's... Probably I would say it's a little bit more prevalent in the United States, but even in Canada here, it's definitely a well-established research perspective in legal research.

03:13 MS: So I wanted to talk to you about what you're working on right now 'cause it's interesting for a number of reasons. First of all, it's just interesting. But second, it is... It talks about how we validate prior research and it also deals with something that has had... It's the kind... The study that this starts from is the kind of study that actually gets talked about and does have an influence in the end and sort of public opinion and policymaking. So what project are you just wrapping up right now?

03:41 CM: Right. So this is one of the research projects that I'm working on, and I'm working on it with some other colleagues at other universities, but it is trying to see if we could replicate the results from this study that you mentioned, this very famous study by Uri Gneezy and Aldo Rustichini called A Fine is a Price. So the initial study is really well-known. It was a study that was gonna investigate how fines work to deter sort of bad behavior and so this was what you referred to as a randomized field trial. So that means essentially they had... They did it in daycares in Israel. So they had this sort of sample of 10 different daycares that they had chosen because they thought they were pretty similar. They were in similar areas and then what they did was they sort of observed them for a period and then they introduced a fine in some but not all of them and then they kept on observing and they were measuring, well, how many parents were late, and then they took the fine away and kept observing.

04:51 MS: Right. So they were trying to see if...

04:52 CM: Yeah. Yeah, they were trying to... So that's a... It's called a... The reason why these kind of trials are so effective is because you're randomly varying this treatment, putting a fine in place and that actually allows you to make a statement about what the causal effect of the fine is, right?

05:10 MS: Right.

05:10 CM: You can isolate the effect of the fine and be confident that that's what's really driving changes in behavior.

05:17 MS: And I think one of the reasons that this initial study got so much interest is because the results were actually counterintuitive.

05:24 CM: Yeah, the result was really surprising. So they expected to sort of study how the fine would deter behavior and how effective it was, and instead, what happened was they put this fine in place and the parents started coming late more often. So they were a bit surprised about that, but then they sort of came up with two different theories about why that might be the case. So one was, well, maybe what's happening here is the fine is kind of signaling to people, okay, the consequences that you might have been afraid of, you might have been afraid something really bad was gonna happen, like your kid would get kicked out, and the fine tells you, well, that's not gonna happen. Okay? So now you're not as worried and now you might come late more often.

06:00 MS: Okay.

06:00 CM: That's one theory. And the other theory which I think has been maybe more influential was that before the fine was there, people really thought about this in a social way. It was really more about social norms and imposing on the daycare workers and it was more about social relations. And as soon as you introduced the fine, it changed the extra time and care to a commodity and then people thought about it differently and it just became... So that's the title of their paper, "A Fine is a Price."

06:37 MS: Right.

06:38 CM: And you can just decide how much extra daycare you wanna buy at the late fee. [chuckle]

06:42 MS: Right, okay.

06:43 CM: Yeah. So that was their other theory that is really influential because it's a lot about how, for empirical legal scholars, it's about how law interacts with social norms that control people's behavior.

06:55 MS: Right. And so they basically put these theories forward and that was kind of the end of that.

07:02 CM: Yeah, you know there's a little bit of follow-up after, and some people said, "Well, we think there's something a little strange about this." And they did some follow-up, but overall, that was sort of it, and the article has been really influential. It's been cited a ton, and it's a really interesting and intuitive theory, right?

07:25 MS: Right. So why... I guess we'd start with your work. Why is replication important?

07:33 CM: So this is really related to sort of general questions about research in the social science where people have started to question, "Well, how much weight should we put on the results, especially sort of surprising results, from one or two studies? How confident can we be that we should really be putting a lot of weight on those?". And so part of this is people have tried to replicate influential studies, and there's been a pretty mixed success rate, especially in sort of social psychology but even in economics and other sort of social science disciplines. People have become a lot more interested in thinking about replication as a way to check and see, "Well, how much weight and how sort of robust are these results?"

08:25 MS: Right. And I'm sort of familiar with the notion of replication from the hard sciences. Like someone says, "Hey, water boils at 100 degrees, we should check that." And a few other people do. "Yep, absolutely. That's what's happening." It seems like you're in a more difficult space to replicate things because the conditions are going to be different.

08:46 CM: Yeah. I think trying to figure out exactly how you would do sort of meaningful replication in social sciences, I think it can be a bit more challenging and especially for sort of empirical legal scholars because of the fact that the way that law operates. It is sort of highly contextual. So we have different laws, different... It's very hard to control sort of all the background factors that we think might be relevant even if we're using something like an experimental design in a field trial.

09:19 MS: Right.

09:19 CM: Yeah.

09:20 MS: So you and a number of colleagues set out to replicate "A Fine is a Price" just to see what happens?

09:25 CM: Yeah. So... I mean our project was part of a larger conference on replication that was held at Claremont McKenna College back in April. And so all the papers at that conference were looking at replicating different studies in empirical legal studies. And yeah. So, ours is looking at a couple of questions. So one is, it's taking more of a... In the paper, we talk about it as more of a robustness approach. So we're really trying to see if we can get the same result and also just sort of include some extra conditions to see if we could pin down which of their theories might be more influential from the original and also to think about whether the mechanism might operate differently in different settings. Yeah. And we're using a different, slightly different methodology. So it's not a field trial. So it's also kind of looking to see, "Well, how robust is this to different kinds of common strategies people use in empirical legal studies?"

10:31 MS: So the burning question is what happened?

10:32 CM: Yeah. Well, so for our study, we essentially use an experimental survey. So we sort of set this all up like you're at a daycare and here's how it works and how late would you be and... Right? So... And then we introduce the fine. So, you know, we kind of set it up so that it will mock the conditions in the original. And so we actually... We don't get the same result. We get the result that you would probably expect in that when we introduced the fine, people say they would be less late. And when we take it away again, people go back to being sort of the same lateness that they originally said they would be.

11:16 MS: Okay, so the intuitive response.

11:19 CM: It's the... Yeah. So it's more the sort of... It's more the intuitive response and also sort of maybe more of the kind of rational choice story about how fines work.

11:28 MS: Right. So is this... Is this a result of... Because you've changed a few things.

11:34 CM: Yeah, we changed a few things.

11:36 MS: So is there kind of a real world to survey differential that makes a difference here?

11:41 CM: So, you know that's a really good question, right? So one of the things people do say about surveys versus, say, a field trial is "Well, it can be different what people think. They would do versus what they would actually do." Okay?

11:54 MS: Yeah.

11:55 CM: I mean, I think that's less of a problem for us here because in some ways, these theories that the original authors put forward to explain their results, are really about ways the fine should change people's perceptions of the context. And the reason why they didn't change when the fine was removed is mostly because people really changed the way they thought about the problem. So I think in that kind of a setting, that's the theory you're trying to test that this difference between what you think you would do and what you might actually do when you're late and your car's parked far away or whatever, it's less of a sort of serious concern that the research method we've got doesn't match.

12:31 MS: Okay.

12:31 CM: Yeah.

12:31 MS: And you said you were working with colleagues on this?

12:42 CM: Mm-hmm.

12:42 MS: How does that happen? How do you find and connect with people that you wanna do these projects with?

12:47 CM: Yeah. So some of them I had started working with on a different project that uses lab experiments to look at the incentive effects of tort law, and I had met one of them at a conference and seeing their presentations and talked to them, and then another one I knew from different conferences, and... So a lot of it is you just meet people when you're sort of out and about talking about your research. And yeah. Then you can start collaborations, and...

13:19 MS: And what's the time frame for this kind of work? How long do these pieces of research usually take?

13:25 CM: I mean, when you're doing empirical work, it can tend to take a bit longer. So it's, you know, you... And also, this is work because it's a survey. It uses human subjects, right? So you have a process where you have to design the surveys and you have to get ethics approval and then you have to just sort of pilot it to make sure that you've got what you want and then you administer the thing. You can get all the data together and you have to analyze it, and so it takes a while. Yeah. Like it would be... You know it's over a year that I've been working on this project and it's still not totally finished. So...

14:02 MS: And what is the end game for this kind of research? What is it toward?

14:07 CM: Yeah.

14:08 MS: Is it publication in a journal?

14:09 CM: Oh, yes. Yeah, yeah. So you would definitely publish it in a journal, and the piece that we're doing was part of a conference, but the idea is it would be part of a special issue of a journal. So, yeah, for sure, we're hoping to publish the piece, and then it sort of becomes part of the larger body of work. You know. And... So our study, we did the replication of the daycare, but we also had another scenario where we looked at this same mechanism in a tax compliance setting, right?

14:42 MS: Okay.

14:43 CM: So then it sort of fits into different literatures and we can kind of get a sense of when, if this is a real effect, when might it show up and what conditions might trigger it and, right...

14:56 MS: So the tax compliance part of this, that adds to what you're describing as the robustness of the project.

15:00 CM: Exactly. Yeah, yeah. So for example, one of the things we were doing was trying to use our survey questions to see if we could figure out whether the fine operated by crowding out social norms or by just signaling consequences about enforcement. And it turns out that actually our results show that this is different across our two settings. So, in the daycare setting, we did see some changes in the way people thought about the daycare workers that suggested, "Yeah, it did kind of crowd out people's concern about making them wait." But in the tax setting, we didn't get any sort of crowding out about imposing on others or undermining tax morale by sort of tax cheating. But we did see that people changed their perceptions in terms of their fear of consequences. So you know that kind of tells us it's important to think about what setting we're in because these kinds of effects may work differently in different settings.

16:09 MS: And I guess this is the point where you can just sort of start theorizing of, "You know, I have a personal relationship with my daycare worker, but I don't have a personal relationship with an abstract tax person, somewhere doing tax things." But is that sort of a danger of this kind of thing is that you can easily jump into why you think something's happening?

16:29 CM: I mean I don't think it's necessarily a danger. I mean I think what's the sort of danger is when we take one or two studies and put too much weight on them and then build up a whole sort of theory based on one or two data points, right?

16:47 MS: Right.

16:47 CM: Because each study essentially really is like one observation.

16:51 MS: And that brings us back to replication.

16:52 CM: Yeah. Exactly, right? So I think, you know, one of the things that replication hopefully does and the sort of focus on it, is to remind us that any sort of empirical work is... You kind of... I think you best think of it as sort of contributions to a larger conversation where in order to really understand what's happening, we need a lot of different people to contribute and we have to sort of look at the whole landscape and not put sort of too much weight on any particular result, but maybe...

17:27 MS: But how...

17:27 CM: Especially if it's surprising.

17:29 MS: How do you know when to stop?

17:30 CM: How do you know when to stop? Well, you never stop. That's the beauty of being a legal academic. [laughter]

17:34 MS: Right. But I mean, theoretically... Like someone could try to take what you've done and make it more robust.

17:38 CM: Yes.

17:40 MS: Or try a different kind of thing.

17:40 CM: Yeah, yeah. I mean, you know obviously there's a certain amount of value added to doing extra studies. So you know as a researcher and as research communities, you have to decide where the best place is to spend your time. Finding tiny little improvements or things that don't change things a lot, you know that stops being worth spending the year on. So I think that's part of once you get a sufficient body of knowledge built up and you think you have a fairly robust understanding of the way that some mechanism works in different settings and contexts, then you can move on and look at something else.

18:20 MS: But in the research community, is your getting different results from the original study kind of a signal that we need to do more research?

18:28 CM: Maybe. Yeah. [chuckle] I don't know. I guess... You know like we'll see. Maybe people will think, "Oh, it's more about, you know, here it just shows that it was a different response, one we would expect, because maybe when you ask people and the fine's in front of them and it's all sort of compressed in time, it just was really noticeable. And so, of course, that's what they focused on." But in the real world, it might not be like that. So, you know, I think one of the things the study shows and the fact there's different results shows is that if we wanna be confident about the way a particular legal policy would work, say, we may need to have different kinds of research about it, right?

19:11 MS: And it comes down... I mean as you were saying before, it comes down to the end, it's people making decisions.

19:15 CM: It's people making decisions, right? And so I think, you know, this is part of the benefit of interdisciplinary research in the law is ultimately we're concerned with the way that law works in practice, right? And you know we have different theoretical perspectives in thinking about what do we think law is doing, how do we think people respond, how do we think people behave in reaction to it, and it's good for us to sort of get out there and have some sense of how those questions are answered in the practical world so that we can, as legal academics, can keep thinking about law in a way that makes sense and resonates with the people who are affected by it.

20:03 MS: And I think that's an important thing to bring up is this isn't stuff that just kind of exists in this rarefied theoretical space. It translates down into people making decisions about policy and people making decisions about what the law should be at the end of the day.

20:17 CM: Sure, right? So, you know, examples from this research, you could think, "Where is an application where we might think about this being interesting or relevant?" Well, let's say that we're trying to find ways to get people to curb their water use. How should we go about doing that? We could think, well, actually maybe we really just wanna appeal to people's social norms and values, or... And sort of just emphasize this is really important because it's a scarce resource and we need to do that and your neighbors are doing it. You're using more than your neighbors. Or maybe we should put in place fines or a price for using water so that it's not free anymore. And so, research like this goes to the question of, "Well, what are those different policies gonna do?" Like, which one is gonna be more effective in terms of actually reducing water use if that's what we wanna achieve?

21:15 MS: What's actually next for this paper? It's the conferences next?

21:19 CM: So, I'm going to the Conference on Empirical Legal Studies in just over a week. And so this project is in their poster session. So they have a poster session as part of their conference and everybody at the conference gets to come around and see your posters and you can talk about your research. It's actually a really fun session. I know a lot of people are in that session, so I'm looking forward to doing that 'cause I'll see what people think and hear people's feedback and...

21:49 MS: Is this kind of like a science fair?

21:51 CM: It's... Well, [chuckle] I guess it's like a little mini science fair. It's pretty small. I mean there's not that many people who would be in the poster session. But yeah, it's a little bit like that. Yeah. Yeah. It's actually... I think it's... Certainly in the sciences, using posters to present your research, is a way more common kind of thing, but it's common in empirical conferences, in social sciences too.

22:14 MS: So the outcomes of this is there's the conference. There's potential publication and then is that... When do you stop research? Like what is sort of the stopping point?

22:23 CM: What's the stopping point?

22:24 MS: Yeah.

22:26 CM: You know, I mean sometimes you can think about extending beyond the settings that you're doing or, you know... So ways you can think about extending this project could be to think about, "Well, is it sort of interesting enough that we should maybe try doing a field trial?" So going beyond this methodology to see, "Okay, well is it actually something about the survey approach we used here that explains the fact we got a different result? What would happen if we did it in a different setting?" You know again, it's kind of like a cost-benefit thing a little bit, because it depends what else you're involved with and how much time you have for research and whether there's other projects that you think... You know there's... It's more beneficial to start something new in terms of doing it yourself as opposed to working on the extra refinement or something that's just an extension of other research you've already done.

23:22 MS: Are your colleagues on this, are they also all law and economics researchers, or...

23:30 CM: So, one of them is actually just an economist. He's a straight-up economist. So he doesn't know anything about the law, so he has to let us do that part. Another one is... Yeah, I think he... You know he definitely does sort of law and social science, law and economics. I don't, he's not maybe not quite as technical, so his doctoral research was in law, not another discipline and then another one is... You know, she has pretty extensive training in economics but she's maybe a little bit more on the law side. So it's a good mix.

23:58 MS: Okay.

24:00 CM: Yeah.

24:00 MS: Thank you very much, Cherie.

24:01 CM: You are welcome Matt.

[music]

24:04 MS: Thanks to Cherie Metcalf. If you're studying, well, anything at all really, talk to your professor about their research. You'll be glad you did. Cherie is the Constitutional Law module author and instructor from LAW 201/701: Introduction to Canadian Law. You can also take our deep dive into Constitutional Law with LAW 205/705: Public & Constitutional Law which gives you a full course on division of powers, citizens rights, how our judiciary functions, and more taught by Jonathan Shanks. You can learn more about it at takelaw.ca. Fundamentals of Canadian Law is recorded at Queen's University situated on traditional Anishinaabe and Haudenosaunee territory. Our theme music is by Megan Hamilton who's also a staff member here at Queen's Law. You can find out more about her music at meganhamiltonmusic.wordpress.com. Original illustrations for this podcast are by Valerie Desrochers. You can find her work at vdesrochers.com. Thanks for listening.

[music]