Episode 2: How?

Transcript

Geoff Sayre-McCord: So I’m in college, and I’ve spent summers in Cambridge, getting sandwiches from Elsie’s Sandwich Shop. Luscious, big sandwiches, like egg salad and tuna that were — oh! — anyway.

Stephanie Tam: That’s Geoff Sayre-McCord, taking us on a trip down memory lane to his university days in the 1970s.

Geoff Sayre-McCord: And I’m a philosopher at the University of North Carolina at Chapel Hill.

Stephanie Tam: He’s also, importantly for this story, the son of Joan McCord, the first female president of the American Criminology Association, who passed away in 2004.

Geoff Sayre-McCord: It’s my great honor to introduce the best mother in the world, as well as an extraordinary sociologist criminologist.

Stephanie Tam: This is the story of the first large-scale, scientific trial of a crime prevention program, the Cambridge Somerville Youth Study, that took place back in the 1930s. And Geoff’s mom, Joan McCord, was the criminologist in charge of following up on just how much it had helped — thirty years later.

Geoff Sayre-McCord: I got to hang out with my mom, eat good sandwiches, be in a cool town, and help her. And we went into this — she went into this — convinced that this was a good program, and that she had, sort of, you know, an extraordinary opportunity to actually collect the data to see which aspects of it paid off how. That’s what excited my mom.

Sam Deere: And that question, of how we can help, once we’re already motivated to do so, is what we’ll be exploring in today’s episode — through the twists and turns of a story about some very smart people with some very good intentions, and some pretty radical ideas.

Stephanie Tam: We’ll take you through something of a detective story and its super-surprising findings, a whole lot of controversy, and finally, the beginning of a new way of doing good better.

Sam Deere: Throughout it all, we wrestle with how helping, even with the best of intentions, can do more harm than good, and just how hard it can be to figure out how to really help well. In this series, we explore the idea of effective altruism. It’s a way of combining the empathy and compassion that motivate us to do good things with the evidence and reasoning that helps us do those good things more effectively.

Stephanie Tam: Over three episodes, we take questions that keep us up at night, like — why should we care for distant strangers at the cost of ourselves and our own communities? Or, how can we make sure our donations are doing the most good? — and wrestle with just how hard it is to help in the face of suffering and uncertainty, and enlist experts — from philosophers and statisticians to social workers and practitioners — to find a way forward.

Sam Deere: In our second episode, we focus on how: How can we be sure what we’re doing is really helpful? How can we collect the evidence we need to figure that out? And how do we move forward in the face of uncertainty, while we’re trying to do good?

Stephanie Tam: We’ll take you from one of the very first large-scale crime prevention programs that used the scientific method on a social issue …

Geoff Sayre-McCord: And you could run data in ways that you just could never have done before. So her attitude, as I understand it, is she saw a great opportunity to follow up on the inspired commitment to doing social science as a controlled study.

… to the thorny philosophical and methodological controversies it raised …

Geoff Sayre-McCord: So, there basically two reactions, neither of which she thought were right-headed. So they took these results as an attack on the value of what they were doing.

… to the growing movement of evidence-based evaluation that it sparked in the world of social change.

Geoff Sayre-McCord: And there are now a fair number of longitudinal studies with control groups trying to figure out which interventions work. And that’s — that’ s a tide change.

Sam Deere: You’re listening to Doing Good Better, a production from the Centre for Effective Altruism. It’s a podcast that explores how to combine head and heart to wrestle with how to, well, do good …

Stephanie Tam: … better.

Sam Deere: I’m Sam Deere.

Stephanie Tam: And I’m Stephanie Tam. In this episode, we’re also telling today’s story with the help of a special guest, Radiolab producer Latif Nasser, who chatted to Geoff Sayre-McCord with me.

Latif Nasser: Hello hello hello?

Geoff Sayre-McCord: Hello! This is Geoff.

Sam Deere: Again, that’s Geoff Sayre-McCord, philosopher at UNC Chapel Hill and son of criminologist Joan McCord.

Stephanie Tam: This is great! So, I’m Stephanie.

Latif Nasser: And I’m Latif. Nice to meet you.

Geoff Sayre-McCord: Latif! Pleasure to meet you.

Sam Deere: So, before we dive into how Geoff and his mom get involved, we need to take a trip back to the 30s.

Geoff Sayre-McCord: The study  was started by a guy named Cabot.

Sam Deere: Richard Clarke Cabot. He was an American physician and professor at Harvard Medical School.

Geoff Sayre-McCord: This is in the 30s, in the Cambridge and Somerville area around Boston. Some cool old cars, but a lot of poverty, you know.

Sam Deere: Youth crime was a big problem at the time, and there was a growing concern about its connection to poverty and family dysfunction.

Geoff Sayre-McCord: And Cabot was a philanthropist concerned to help the poor, but also trained in research, and realised that he should figure out a good way to test his interventions.

Sam Deere: Back then, this idea of applying experimental medical methods to social programs was not only pretty new, but also pretty radical. But that’s exactly what he decided to do.

Geoff Sayre-McCord: So what he did was he went around and he identified roughly 250 at-risk kids — this is talking to, you know, preachers, teachers and parents — which kids seemed to be having trouble. And then he also identified 250 kids who seemed to be doing well. He then matched — within each of the two groups — pairs, for neighbourhood, family size and income; maybe some other things.

Sam Deere: So, for instance, if one kid came from a particular neighborhood, with working class divorced parents, he might be matched with another kid from the same neighborhood, also with divorced parents. The idea is that these pairs are sort of acting like sociodemographic twins.

Geoff Sayre-McCord: Exactly.

Sam Deere: And that’s all key to the logic of this particular experimental method, the randomized controlled trial. Because it means that any difference in outcome between the control and treatment conditions can be directly traced back to the treatment — because, at least in theory, the only difference between these two groups is that one of them received the treatment and the other didn’t.

Geoff Sayre-McCord: Right. And then the treatment took a broad range. It was basically counsellors who would visit the family; they offered tutoring, they would take the kids out into the country, they’d teach them to drive as they got to that age.

Latif Nasser: Ooooh this sounds like a good deal.

Stephanie Tam: Yeah!

Geoff Sayre-McCord: Yeah it was big brother/big sister, it was the kind of mentoring that still is celebrated.

Latif Nasser: Sure.

Geoff Sayre-McCord: So what you had at the end of the Cabot study was this voluminous set of records. Here’s this treasure trove of data, with controls! That’s almost non-existent in social interventions — that you have data on a control.

Sam Deere: Because, at least around the time that Geoff’s mom, Joan McCord, was a graduate student, that just wasn’t the way social work worked. So for instance —

Geoff Sayre-McCord: She had worked with the Maccobys at Stanford.

Sam Deere: — Leading social scientists at the time —

Geoff Sayre-McCord: She has a nice little autobiographical essay, and at one point she notes in that that she was sitting in on a meeting and all their data, which was about modes of parenting and child rearing, relied hugely on parental reports. And she was skeptical of their veracity.

Sam Deere: Joan worried that depending just on what the parents were saying about their own children wasn’t enough, and that we should also have independent measures outside of the family, too.

Geoff Sayre-McCord: So this is when she was a graduate student. And so early on, she thought that when we intervene socially, we should be looking for the same kind of reliable data that you use in medical contexts.

Sam Deere: In other words, the same kind of data that was being collected for the Cambridge Somerville Youth Study.

Geoff Sayre-McCord: In the seventies, all of sudden, you know, universities had these buildings full of giant mainframe computers you had access to them, and you could run data in ways that you just could never have done before. So, she saw a great opportunity to follow up on the inspired commitment to doing social science as a controlled study. That’s what excited my mom. Now I can collect data, do a follow up and figure out what were the conditions in the household that were precursors to which problems. Did mentoring help with this, but not with this, or? …

Latif Nasser: Oh, let me look at it closer, to figure out how — how exactly …

Geoff Sayre-McCord: It was so good!

Latif Nasser: It was so good. So on the one hand she was sort of looking around at all the other studies that were being done, and was thinking, “Oh, these are just, you know, people looking to the end of their nose.” Like, if you’re asking people about parenting, you’re just asking the parents of toddlers how it’s going. And then she looks at this study, and she’s like, “Oooh, here’s this juicy —  there’s tons of records here. I have different kinds of data here, and so here’s a juicy one to follow up on.” That kind of thing?

Geoff Sayre-McCord: That’s exactly right. She was clearly animated by thought that if you can find the causes, you’re in a better position to find treatments. And the vindication of your work is to do that. That’s what excited her.

Sam Deere: So, about thirty years after Cabot’s initial program, Joan McCord dives into a follow up study of the Cambridge Somerville boys — to see what happened to them, and if the intervention helped. And at the same time as his mum was looking into the effects of a mentoring program, Geoff decided to become a mentor himself.

Geoff Sayre-McCord: And it was in the grip of that that, when I got to college, I volunteered as a mentor.

Stephanie Tam: Yeah, you must have been excited too!

Geoff Sayre-McCord: Oh yeah, and I loved doing it. I love hanging out with kids. And I knew his mother, who worked at the college I went to. He was ten or twelve, he was right in that age, and his father had been in jail. It was mostly tutoring and hanging out and playing soccer. And at the same time, I was involved in trying to track down people when my mom started doing a follow-up of this. So I’m in college, and I’ve spent summers in Cambridge, getting sandwiches at Elsie’s Sandwich Shop, which sadly no longer exists.

Latif Nasser: Yeah, I was thinking I used to live in Cambridge but I don’t know that sandwich shop.

Geoff Sayre-McCord: Oh, you really missed out!

Latif Nasser: Oh man!

Geoff Sayre-McCord: They were these, just, these luscious, big sandwiches, like egg salad and tuna, that were — oh! —anyway, so, I travel a lot on my stomach. So, and we were tracking down people. This is in 70s so we didn’t have the web, but we were using old phone books, doing searches in the library, figuring out where they had lived. You’d be dialing the numbers and, you know, introduce yourself, say, “I’m trying to reach so-and-so, did you used to live in Cambridge at this address?” And having confirmed them, you know, and said, “I’m working for the Cambridge Somerville Youth Study; we’d like to contact you.”  You know, it was a little bit like a mystery, and you’re trying to track people down, and every time you found someone it felt like a little treasure hunt. And we went into this — she went into this — convinced that this was a good program and that she had, you know, sort of an extraordinary opportunity to actually collect the data to see which aspects of it paid off how.

Sam Deere: So …

Geoff Sayre-McCord: She found 98% of the people from the study.

Sam Deere: ~Actually, records were found for 94.9%, of the remaining participants, and they managed to interview 347 of them.~

Correction: Our correction is based on an earlier figure before Joan McCord finished the hunt and so Geoff's figure is correct. The audio has been amended to reflect this.

Geoff Sayre-McCord: Finding some of them was finding that they’re dead. And she got access to all sorts of amazing records: interviews and questionnaires. And there were a bunch of measures — seven in particular — that are worth mentioning: death records, criminal records, health records, mental health, alcoholism, job satisfaction and marital satisfaction.

Sam Deere: So they could find out if the treatment helped the group of boys be less likely to die early, commit crimes, suffer from mental health problems or be unhappy with their jobs or marriage.

Geoff Sayre-McCord: And I do remember vividly her running the data and coming home, at the time, with these huge stacks — and she’d go in with these boxes of punch cards and run them, and always sort of be excitedly waiting for the output.

Sam Deere: And then —

Geoff Sayre-McCord: On all seven measures —  we’re talking how long did you live, were you a criminal, were you mentally healthy, physically healthy, alcoholic, satisfied with your job, satisfied with your marriage — on all seven measures the treatment group did, statistically, significantly worse off than the control group

Latif Nasser: Oh wow, that’s not what I expected. Hold on, so … so she does a follow-up, looks at every nook and cranny of these people’s adult lives, including all this data that she usually couldn’t get access to, and then finds that it’s, that this has all backfired?

Geoff Sayre-McCord: Right, it’s made things worse, and to a statistically significant degree.

Latif Nasser: Wow.

Geoff Sayre-McCord: And the longer the treatment, the greater the damage. So there’s a dose effect.

Latif Nasser: Oh man, that is — brutal. What is happening here? What was her reaction, how did she…

Geoff Sayre-McCord: So her reaction at first was disbelief. For a long time, she would look at the data and think she’s not getting anything, this is really frustrating. And since programming was so sensitive — we were not talking graphical user interfaces, we’re talking, you know, a misplaced comma completely changes things. She re-ran data and re-ran data and isolated elements to confirm. And I remember her thinking, in effect,  “Oh no, I’ve screwed up. Something’s wrong!”

Stephanie Tam: Like, her fault?

Geoff Sayre-McCord: Her fault, exactly! And then she had an epiphany. She realised, no no no, everything that looks like just serious mistakes, you just turn it over, and you see: oh this is showing damage. It’s showing iatrogenic effects, that the attempted cure caused the harm. And it had just not been on her radar. And it’s — that’s one of the important things, is people who are engaged in social interventions really don’t spend much time thinking, “I may be screwing this person over.” They are self-conscious about, “Maybe this won’t work, but I’ve gotta try.” So she told me the results, and she knew I was serving as a mentor and that I was enjoying it. And she basically gave me all the reason to think that, to the extent I was doing this to help someone, I shouldn’t think I am.

Latif Nasser: Oohh…

Geoff Sayre-McCord: So that argument did weigh heavily, and you know, it’s nothing like the poke in the eye of a social worker who’s spent thirty years in social work. But I became convinced that there was a moral objection to continuing to do what I was doing. And then the challenge was to handle it as gracefully — considerately — as possible. It was a difficult mode of disengagement, something you could explain more to his mother than to him. Because she knew the whole point of this was to be helpful to him. But basically I had to withdraw by being too busy.

Latif Nasser: Oh, so that’s what you just said to him.

Geoff Sayre-McCord: Yeah, yeah. And, you know, that could itself be it’s own bad thing.

Stephanie Tam: Ugh, yeah

Latif Nasser: Like a self-fulfilling prophecy, kind of.

Geoff Sayre-McCord: Right exactly. So there basically two reactions, neither of which she thought were right-headed. So, one is the group of people who have dedicated themselves to social interventions, trying to help. I think the main venues for her interactions with them were professional meetings: the American Society of Criminology, American Sociological Association. Not just academics, but practitioners in various ways. And they took these results as an attack on the value of what they were doing, and their main response was, “Well yeah, maybe it didn’t work back then, but we’re better now, and we know what we’re doing now.” And, “Back then they were too influenced by Freud, and the weren’t enough influenced by this.” To which my mom’s response was, “That may well be true, but you don’t have any evidence that it’s true, any credible evidence. You just have short-term self-reports, which —the data from this study shows — are pretty unreliable.”

Sam Deere: Because, remember all those interviews that Joan McCord conducted in her follow up, talking to the men thirty years later?

Geoff Sayre-McCord: Of the people who were interviewed or filled out questionnaires, who had been part of the treatment group, their memories of the mentoring were overwhelmingly positive. So the subjective reports, you know, you’d get people saying oh it was one of the best things that happened in my life. Whatever happened to my mentor?…

Latif Nasser: Oh no, oh man.

Geoff Sayre-McCord: So you get this mismatch between subjective reports and these objective measures of consequences.

Latif Nasser: I — one of the things that is so difficult for me to wrap my mind around is the idea that the people themselves really thought — really enjoyed it — really thought that it was somehow edifying and nourishing and helpful for them, and then it turned out that they were wrong. I mean, to me it’s so difficult to know — I don’t know, I have this idea that being able to do good is — there’s so much of it that’s about listening, which is difficult to do. But when you do do that listening, and you find that, yeah, that it sort if is conflicting, or the person who is the receiver in all this isn’t the best person to speak on behalf of that. I just I get get thrown in this sort of epistemological black hole. I just don’t know what to trust any more.

Geoff Sayre-McCord: Good. So there’s a couple of things in what you said. I do — so I believe in epistemological black holes, and I think …

Latif Nasser: Oh man, you can’t trust anything.

Geoff Sayre-McCord: Well, or, I think I’m not that much of a pessimist. At the same time, this is compelling evidence that interventions make a difference. So it’s not an argument for doing …

Latif Nasser: Oh, man!

Geoff Sayre-McCord: … for doing nothing, it’s an argument for figuring out which drugs work.

Latif Nasser: It’s  still a big poke in the eye to tell people, “Hey, the stuff that you’ve been doing until now that you think has been helping, you could have been working at exactly cross-purposes to yourself the whole time.” Like, that’s a blow …

Geoff Sayre-McCord: Oh yeah.

Latif Nasser: … to someone who, you know, has been working in this field for years — if not decades — doing what they thought was the noble thing, and then all of a sudden, you know, this one person with this one study is like, “Uh, no. No. You’ve been doing the exact opposite.”

Geoff Sayre-McCord: I think that’s exactly right. It is a big blow. The thing that I think about regularly in this context is thalidomide.

Sam Deere: Thalidomide is a drug that was used to fight the effects of morning sickness.

Geoff Sayre-McCord: And in the late 50s it started to be quite widely used, and then suspicions started to emerge that it might be causing birth defects. And it turned out thalidomide dramatically causes birth defects.

Latif Nasser: Yeah.

Geoff Sayre-McCord: And it basically stops limbs from developing. And so it’s now been outlawed for this kind of use completely, and pregnant women can’t take it, but there’s no way any individual doctor would know it. But once it was discovered, there’s no way any morally responsible doctor would keep prescribing it. Now, you could say now, “That was a long time ago, they ate crappy food then, they weren’t as exposed to all sorts of chemicals that we’re now exposed to — maybe thalidomide’s fine, now. Let me give it to my partner who’s pregnant.”

Latif Nasser: Yeah, that’s a bad idea.

SD:Joan McCord never found out exactly why the intervention backfired.

Geoff Sayre-McCord: So there were the various candidate explanations. My mom’s favourite hypothesis — but she died before she could test it — was, she thought it may involve the transference of values. All the mentors were fairly well-off people who could afford to donate time to this process in the 30s, and all the kids were financially very poorly off; it was a very poor neighborhood. So one thought is, the kind of values — marry for love, get a job you find satisfying, travel the world — that would be in the ken of the mentors, were values that were not suited to the circumstances the kids were gonna find themselves in. And her thought was, perhaps the longer the mentoring, the warmer the relationship, the more the kids would acquire and embrace these values that then, as they led their lives, would lead to frustration; because, you know, their marriage was less for love and more for who got pregnant, or who was in the neighbourhood, or who they’d been dating in high school; and their job was the job they could get, not a career that was satisfying and something they’d love, and they didn't have the resources to travel.

Latif Nasser: That’s such a devastating explanation, because it’s like the thing itself — it’s like, the problem here is the thing you were trying to get in the first place.

Stephanie Tam: Yeah, like changing values expectations, kind of…

Geoff Sayre-McCord: Right, and I'm not sure they would have been consciously thinking, we’re teaching values. But we’re giving them a taste of luxury, without thinking, this is a luxury that’s gonna be unobtainable afterwards.

Sam Deere: So, there hasn’t been much conclusive support for that hypothesis. There was actually a criminology article published in 2016 looking into various explanations for why things went wrong, and some researchers have argued that “peer contagion”was at work. This is the idea that, if you put a bunch of at-risk kids together repeatedly, as some of the boys were going on summer camps in the treatment condition, the bad boys become the cool kids. Maybe they had a bit more street cred. And there has been more evidence backing this explanation, actually. But that’s a story for another time.

Geoff Sayre-McCord: Now you don’t have to understand how it works. You don’t have to be sure that things haven’t changed to make it not dangerous. Because you accumulated some serious evidence that it was dangerous. So you can say, maybe things have changed so that we’re not risking cutting short longevity by an average of five years. I mean, we’re talking a serious blow. You know, maybe things have changed so that they won’t have these effects. But the thing to do is to test whether that’s true given, the background evidence.

Stephanie Tam: Okay, yeah, can you tell us more about the reactions at the time?

Geoff Sayre-McCord: Okay. So let me mention the other group, too. The other group were social conservatives who were opposed to social intervention. Who pointed to this research and said things like, “See, you just can’t help people; it’s all on them.” And my mom’s reaction was, “No no no, this shows social interventions have an impact. It means things can change.” So you don’t study thalidomide and say, “Now we’re gonna give nobody drugs.” You realise, oh drugs are powerful things! Let’s figure out which ones work. The same thing with social interventions. They make a huge difference, it’s just we haven’t done the studies. We don’t put money into the studies to run controls to figure out what the effects will be, long term. So to sit back and say we should be abandoning social interventions is the wrong reaction. The right reaction is, we should start to be careful in trying to figure out which ones work.

Stephanie Tam: But to play the devil’s advocate, you could also say, well, maybe doing nothing is better than doing something we don’t know works. You know, like if you have limited time, limited resources, money — I mean, a huge difference between social work in general and drugs, is that pharmaceuticals have money. They have  kind of big resources. So, like, is it better to just do nothing, or to try and help a few people, or spend that money on pilots that really help no one until we know?

Geoff Sayre-McCord: So I think the devil’s advocate in you is important. I do think there really is a central ethical challenge here: how much is enough, where? And you know, it’s probably a different answer for disaster situations, where any evidence at all that it might be helping and not hurting is reason to try to do it, versus other things. Her idea was, we ought to be pouring money into these interventions; we ought to be requiring that they start small and have control groups; collect objective data; make sure that it’s measurement is valid. And she thought they’d better be longitudinal, because if you really want to help and not hurt you need to check longer term.

ST & Latif Nasser: Hmm!

Geoff Sayre-McCord: So there was an overwhelming resistance, and it took the form of, you know, oh well, “That was then, that was them. This is now, and this is us.” But she was not timid. The care and quality of the work and the clarity of the hypotheses and the evidence — that’s just unimpeachable.

Sam Deere: And gradually, the controversy turned into congratulation, as the importance of McCord’s work was increasingly recognized. In 1988, Joan McCord became the first female president of the American Society of Criminology.

Geoff Sayre-McCord: But there’s still whole schools of social work that don’t spend a lot of time on that. And there are now a fair number of longitudinal studies, with control groups, trying to figure out which interventions work. And that’s — that’s a tide change.

Sam Deere:  And there are now whole organizations dedicated to the evaluation of social interventions, and how to apply evidence to social policy, such as JPAL.

Rachel Glennerster: We are a network of researchers who use randomised control trials to evaluate what is the most effective way of addressing the needs of the poor. We have over a hundred and twenty researchers in our network, and we also have staff and research centres in universities all around the world.

Sam Deere: That’s Rachel Glennerster, executive director of the Abdul Latif Jameel Poverty Action Lab, or JPAL, at MIT. Their mission is to reduce poverty by ensuring that global policy is informed by the best available scientific evidence.

Rachel Glennerster: So, we achieve our mission through three lines of work. One is, we work to support our researchers doing randomised control trials of poverty issues — and when I say poverty I mean that very broadly, everything from education and health to financial programs for the poor, or women’s empowerment. Our second part of our mission is to ensure that that evidence gets used effectively. So we work with a range of, typically, governments in developing countries and NGOs, and others who are running poverty programs, to incorporate the lessons from these randomised trials into their programs. And the third thing we do is build the capacity of other people to do this kind of research. So that means a lot of training, a lot of building research capacity in developing countries. We’ve done a lot of work in helping change national government policy.

Stephanie Tam: So, take education, for instance.

Rachel Glennerster: Researchers who had been working with a small NGO in western Kenya; they’d been looking at a range of different things on education, they kept trying different ways of improving education, and none of them seemed to work.

Stephanie Tam: Things like providing textbooks, or adding computers.

Rachel Glennerster: And then they tried treating kids in schools for intestinal worms, and this had a very dramatic improvement in children turning up to school. Particularly dramatic when they tried all the obvious things — more textbooks, more teachers, adding computers — and so it was particularly dramatic that this health intervention proved much more cost-effective than all these other education interventions they tried before. And yet with these dramatic results, they went to the government of Kenya and, you know, talked about them, and started a conversation — which took many years— about how could we bring this into a national policy.

Sam Deere: Now, there’s still some controversy about exactly how effective deworming is at getting children to school.

Stephanie Tam: One might even say it’s opened a whole can of worms …

Sam Deere: … But it represents a critical shift in the way we think about social change and social issues, that can often be difficult to measure, and have  consequences that can be unpredictable.

Stephanie Tam: Right. In fact, the story of the Cambridge Somerville boys was part of how I became interested in this idea of applying the scientific method to social change. It stuck with me as a real warning about how helping can sometimes hurt, and as a hard lesson in humility.

Geoff Sayre-McCord: So my mom, she was perennially aware of the appropriateness of intellectual modesty. She thought the skeptics who emphasised how little we know were right to do it. What her dying words to me were — she’s on her deathbed; she had been for her life an agnostic, didn’t believe in God — she said, “This could be interesting.” You know, she didn’t think there was a god, but if there was, then there was more interesting things to come. And it’s basically the way she led her life was, she was always open to arguments and evidence. Always. And she would change her view in light of them, even if she couldn’t always change how she felt.

Sam Deere: So I think one of the reasons that the Cambridge study is so important is that it really shows us that evidence trumps intuitions. And it always comes back to this question of what do we care about, and what are we trying to do with our money and our time and our efforts to make the world a better place?

Stephanie Tam: Yeah.

Sam Deere: And is our aim to actually make the world a better place? Or is our aim just to be given a pat on the back for our good intentions? And if that's the only thing that we care about, then sure, I mean, we don't have to look at the evidence. But if we do care about actually achieving that good then we have to be prepared to be wrong. We have to be prepared to fail and to look at our mistakes and say, "Well, this is actually something that we need to remedy and try and do differently next time, because otherwise we are going to repeat it."

Stephanie Tam: Yeah, I think that when we talk about having humility — when you're going to help people, you should really listen to what they want and what their feelings are on the subject. And the idea that their experience of the study was this really positive point in their lives, where they were happy and they really loved their mentors, and they thought that the program was great. And what to kind of do with this tension — contradiction, I guess — between the subjective report and the objective measures, I think really raises, I guess, what the role of that subjective/objective measure is.

Sam Deere: Yeah. Well, I mean, if the value that you put on your present self is much higher than that that you put on your future self. And so this is why people do things that are risky — say for example, smoking — because you get the nicotine hit right up front. And even if you know intellectually that this could shorten your life, you don't have this intrinsically strong feeling that your future self is as valuable as your present self, and so you end up discounting how valuable that is. And so by the time that your present self becomes your future self, yeah, that becomes a problem. Because you're like, "Oh actually, now I would prefer to have another ten years of life, that I don't get because I've been smoking my whole life." And I think that's the same thing.

Stephanie Tam: So one of the things that can be a concern for people — and I think Geoff sort of touches on this — is when we are emphasising evidence, are there things that are harder to quantify, and so end up getting ignored, in this kind of rigorous, evidence-based search?

Sam Deere: One term for this is the streetlight effect, which is — let's say you're walking home and you're a bit tipsy, and you get home and you realize that you've lost your keys. And you're like, "Ah, where are my keys?" And you can't find them in your pockets and so you start backtracking. And it's a dark night, and so the only places that you can see anything are streetlights. So there's this sort of patch of light, where the streetlight is, and then a patch of darkness in between them. And if you're not thinking very clearly you might only restrict your search to the streetlights, because, you know, if your keys were there, they'd be easy to see. But of course you're ignoring the whole search-space between the streetlights, and so this kind of thing can come up when you're looking at different ways to help people. You don't just want to focus on the things that are easily measureable. You don't just want to focus on things where you can see that there's an easy answer. The way that we look for solutions has to be holistic and all-encompassing. You have to admit lots of different kinds of evidence. This isn't to say, though, that we lower the bar and that we're not rigorous; it's just to say, we need to be aware of our own biases and say, just because we have very strong evidence but it was easy to get, doesn't mean that something where we have weaker evidence — but that evidence was harder to generate — shouldn't be taken into account.

Sam Deere: This has been Doing Good Better, a production of the Centre for Effective Altruism.

Stephanie Tam: Make sure you check out our final episode, where we talk about what people are actually doing out there, right now.

Sam Deere: And if you're interested in learning more about how we can use the best available evidence to help others, check out effectivealtruism.org. You'll find resources on these topics and many more.

Stephanie Tam: Thanks for listening. That's all for now!

Doing Good Better is a podcast series co-hosted by Sam Deere and Stephanie Tam, exploring questions of why, how, and what of effective altruism.

Our producer is Stephanie Tam and our sound engineer is Dominic Apa; our production assistants are Sandrine Chausson, Jhansi Hoare, and Kiran Lloyd; with help from Sam Deere, Irene Tortajada, and Nikita Patel.