ANITA WILLIAMS WOOLLEY: Such a pleasure to be here. This is a cause that is near and dear to my heart. I teach at a business school. We're always trying to get more women to come into our programs. I'm on a few boards of directors. And so many of the themes that I've heard already here today really resonate with me. So I want to start off, recently we mark the anniversary of September 11th attacks. When those took place, I was actually a post-doc at Harvard working on a project that was funded by the intelligence community.

My advisor Richard Hackman, was a widely regarded expert in teams, and he was on an advisory board to then the director of the CIA. It became quickly apparent that one of the key issues underlying the attacks and the reason why they were not detected was not that the community lacked the expertise to know what was happening, but they lacked the ability to put that expertise together to really ferret out what was about to take place. And in retrospect, it was very clear that that could have been predicted. So we were called in to actually look at collaboration across agencies within the community and try to understand better what the issues were underlying the problems.

Among the studies that we conducted, we did an experimental simulation using people who were trained in different types of evidence analysis, and we put together a scenario, a terrorist scenario, based on a real-world case. And then we brought them into the simulation. We either composed the teams so that they actually either had diversity of expertise or they were all generalists, so highly skilled individuals, but generalists. And then we either started them off with their work by just having them go about it as they would normally. Or we had them think first about how they would apply their expertise to the scenario, and especially where they needed to integrate what different people knew.

And so what we found in some respects was rather surprising. So first I'll show you, when we have the generalists, so highly skilled, but not subject matter experts, if you will, it didn't really matter if we gave them this exercise. They performed about average in terms of solving the case. However, when we brought together experts, what we found was a bit more surprising. First, when we had people with high expertise, but we didn't force them to figure out how to integrate it, they actually performed significantly worse than the generalists. But if we had the experts, and we forced them to first take 10 minutes, literally, and think about how to combine what they knew, they performed significantly better.

And so it really suggested that there was a key process step that the community needed to integrate in order to make use of the experts that they already had on hand. This led to later interest in my lab at Carnegie Mellon in collective intelligence. So for decades, actually, collective intelligence has been studied, but in the context usually of animals, other species, not humans. For good reason, because there are some pretty impressive examples. So ant colonies, for example, are made up of ants that are very simple creatures individually. They don't have a whole lot of memory, or capacity, or problem solving capacity. But collectively, they accomplish pretty impressive things, things that are well beyond the capabilities of any individual member.

But these ideas had not been explored in human teams. Many of us are very familiar with the concept of general intelligence, and the notion that there is this underlying factor that drives performance across many different domains. We wanted to know if the same thing might be true for teams. Is there an underlying collective intelligence factor that enables certain teams to perform well, consistently, across many different scenarios? And is it something that's really separate from individual ability? Up to this point, when researchers had talked about collective intelligence and teams, if they even used that term, it was as a function of the intelligence of the individual members. How smart are the people, and that's how smart the team should be.

But we wondered if there was maybe a separate kind of measure that we could come up with about how well the team works together. So we brought teams into our lab. They spent the better part of a day with us. They worked on all kinds of different tasks. And what we found was that first, there was strong evidence that teams that did well on one kind of tasks, say a mathematical task, also did well on a creativity task, also did well on a verbal task, et cetera. And when we use their performance scores on these tasks to calculate a collective intelligence score, we could predict with a pretty high level of accuracy how that same team would perform in the future when we brought them back to do a video game simulation.

Furthermore, collective intelligence was a much stronger predictor of future performance then the average IQ of the individual members on the team. So we've replicated this finding several times now. And consistently what we find is that collective intelligence is a strong predictor of future performance, much more so than the average or even the maximum intelligence of the individual team members. So even the smartest person on the team doesn't exceed the capability of the team. So this really suggested to us that this is a factor that is independent of individual ability and worth exploring more deeply.

And it led us to ask ourselves the question, well, what leads to smart groups? If it's not smart people, what is it? We looked at some of the usual suspects for an answer, things like group satisfaction, cohesion, and motivation. In our studies, these are never consistently correlated with collective intelligence. So some of the ideas around team building that are common may not actually lead directly to collective intelligence, if in fact they are intended to build cohesion and the relationships among the members. But we did find three buckets of things that they do consistently lead to collective intelligence.

The first has to do with some attributes of the individuals on the team, but not individual intelligence, as I've already shown you, but a couple other things that have surfaced. The second has to do with the kinds of goals that they're pursuing. And the third has to do with how they're working together. So I'm going to review some of our research in each of these areas. First, right people-- one thing that we didn't set out to explore initially was what the effect of the gender composition of the group was on collective intelligence. And in fact, there was a graduate student in our research group who initially was saying, you know what, it seems like there's this relationship with proportion of women. And at first, we're like, I don't know, we shouldn't make too much of that until we see it a few times.

After the second, and the third, and so on, it's like, OK, we need to pay attention to this. So this is an aggregation of data across a dozen studies now. It represents several hundred groups. We have classified them with respect to how many women are in the team. So on your far left are the all-male teams. And then next to that, if there is just one woman. Next to that, if there is more than one woman, but it's still majority male, et cetera. And the yellow line is average collective intelligence. And what you can see is that it's not until the teams become majority female that they are consistently above average collective Intelligence. And they peak at the point where there is just one guy.

We don't have an explanation for that one yet. And then it trails off down to average again when it's all women. We have some hypotheses about that, but no data yet. But what it strongly suggests is that gender diversity is a benefit to collective intelligence, gender diversity with a tilt toward women. And one of the reasons for that relates to a second finding related to this trait we call social perceptiveness. So many of the participants in our studies take a measure known as the reading the mind in the eyes test. This is a measure that was originally developed by researchers of autism and autism spectrum disorder, because this is the kind of ability that people who are on the autism spectrum struggle with. It's picking up on subtle nonverbal cues and making inferences about what somebody else is thinking or feeling.

So participant's see 36 pictures. They have to pick from among the four options that are listed above. It turns out that teams that are higher on this trait are more collectively intelligent. Women, on average, score higher on these tests than men. That's true in our sample, and is also true in the population. And that explains a large proportion of the effect of having more women in the team, is that you raise the average social perceptiveness. So the first attribute of right people has to do with the social perceptiveness, or probably more generally, some social collaboration skills that enhance the work of the team.

A second piece and pertinent to the theme of this session is diversity. Just to be clear with our terms, there's a lot of literature on diversity in the psychology and in organizational behavioral literature. People sometimes focus on things that are surface level, that lead you to characterize people as different from the self. A lot of both the problems and the benefits come from what we know as deep level diversity, or underlying differences in perspectives, opinions, knowledge, information, et cetera, that can be brought to bear. So one form of deep level diversity that we've been looking at is cognitive style diversity.

Many of you mau be familiar with this notion of right brain versus left brain thinkers, or you might have heard somewhere in your education about people who are verbalizers or verbal learners versus visual learners. Well, more recent research in neuroscience suggests that it's really not right brain/left brain, it's actually top brain/bottom brain. And that there is another distinction within verbalizer/visualizer that turns out to be important. There are two different kinds of visualizers. The first kind of visualizer is a spatial visualizer. This is probably the kind of visualizer that most of us would think about when we think about visualization.

These are people who are good at rotating objects in three dimensional space. They also tend to be very good with mathematical kinds of thinking. They're analytical thinkers, meaning they break things down into parts and arrange them in space or over time. And these people are heavily represented in quantitative oriented professions. By contrast, are object visualizers. These folks think holistically and notice the properties of, the global properties of shapes. So for example, a strong object visualizer would notice if you've just gotten a haircut. My husband's not an object visualizer, so I have to drop a lot of hints.

But because they pick up on these kind of holistic properties, these folks tend to be more represented in the arts and design. And then there are the verbalizers. They're also analytical thinkers and break things down into parts, but they represent them as words instead of images. So these distinctions have been identified in neuroscientific research, showing that people use different areas of the brain to engage in this processing, individuals tend to favor one of these styles over another, and that they're slightly negatively correlated with one another. In other studies, we find that there are also, as I've mentioned, represented in different proportions in certain professions.

So in law, in humanities oriented professions, you have a much higher level of verbalisation. In engineering and the sciences, spatial visualization, and in the arts and design, object visualization. So we were curious what this does when you put these people together in teams. What are the issues that surface? Or what are the benefits? So just a general finding, when we do nothing to intervene to help people work together, what we find is this curvilinear relationship with cognitive style diversity in collective intelligence, where you do need a requisite amount of diversity for teams to reach the highest levels of collective intelligence. But it is possible to have too much. It can create certain issues, which I'll get into next.

So this is more evidence for the benefits of diversity. Again, we didn't do anything, like our integration exercise I mentioned earlier, to help these teams make use of these skills. So this is just what they did on their own. So cognitive diversity is important. And now-- let me go back-- so now, I'm going to continue by talking a little bit about goals. And this relates back to the cognitive diversity. One of the most prevalent findings in all of organizational psychology was the benefit of having a clear goal. So I think many of us probably experience that, and so I'm not going to discuss that research, but discuss and nuance that we've added with some of our research about the type of goal.

So if you think about a goal, in any task that you're doing, you can think about, OK, is it clear what I'm trying to do, or how I'm going to do it? And so, that kind of creates this conceptual two by two of whether or not the ends are specified or the means are specified. So I think we would all agree that having nothing specified is not the quadrant you want to be in. And if you've experienced that, I'm sorry for you. It's pretty chaotic. The opposite quadrant is also kind of problematic in that it can rob people of the autonomy to use their judgment for how to do things. And so, if you've had a micromanaging boss, for example, somebody who told you exactly what you're supposed to do and exactly how you're supposed to do it, and you just follow the instructions. And so we call it, pejoratively, factory work.

But it's the other diagonal that I'm going to focus on. And we refer to the situation where you know the process you're going to follow, but you don't exactly know the outcome you're seeking as process focus, and the opposite as an outcome focus. So let me go into these in a little more detail. Process focused teams are those who will emphasize the schedule, the tasks, the roles first. And then what they achieve is subordinated to those processes. Whereas, an outcome focused team is more what some of us think of as a prototypical entrepreneurial team, for example, emphasizing the desired outcome first, what I'm trying to attain without regard to the resources that I might control currently. And so the process gets subordinated to those outcomes.

So it turns out that these are beneficial for different kinds of situations. And so in one study, we were looking at creative team performance. These were consulting teams coming up with a strategy for a small company. And we evaluated early on in their work if they were outcome or process focused. And then looked at the evaluation of their work when it was delivered. And so what we found is that, as you might expect, the unfocused teams didn't do very well. The process focused teams did significantly better, but best of all was the outcome focused teams. And in interviewing these teams, what we found is that they spent a lot of time early in their work really thinking about what are going to be the key things to achieve for this to really be successful and useful for this client, so, before they got into outlining the slide deck, or the final report, and divvying up who was going to do what, et cetera.

So that was in a creative task. In another study, we brought teams into work on tasks where commission of errors was going to be very costly, really hurt performance. And what we found in that case was that a high process focus was good in those cases. These are errors, so lower is better. Highly processed focused teams committed fewer errors, whereas teams that were not as process focused committed far more. So putting this all together, what it suggests is that a process focus is useful when a task requires impartiality, error prevention, comprehensiveness, so things like conducting an audit. You don't want to be outcome focused, I imagine, when you're doing an audit. You want to follow the process and attain the outcome that is the right outcome if you've followed that process.

Idealized versions of science are process focused. You're supposed to follow your procedure and you get the results you get. Jury trials, also process focused. By contrast, when a task requires innovation, insight, identifying priorities, that's when you want more of an outcome focus. So crafting a new strategy, developing a new product, things that really require creativity. I want to emphasize, however, that it's not that planning is not important. Planning is important for both of these. It's really the nature of the planning that tends to change.

So relating this back to the cognitive styles I was describing earlier, it turns out that cognitive diversity has an effect on whether or not a team becomes outcome or process focused. So in one study, we found that the more spatial visualizers we included in a team, the more process focused they tended to be. And that again is because spatial visualizers break things into parts and arrange them over time in three dimensional space. And so that really lends itself to coming up with a very detailed process for how you do work. Another piece that we have found that turns out to be influential-- this goes back to the research in the intelligence community-- we've looked at teams that are looking at a problem from an offense perspective, meaning that they're going to carry out some operations against another entity, or from a defensive perspective, to keep bad things from happening.

And it turns out that this is also a precursor to developing an outcome or a process focus. We find that teams playing defense are significantly more process focused than teams playing offense. And this is because most defensive teams are trying to cover the waterfront, if you will, be very comprehensive and systematic. But we have found in another study in intelligence teams that it really hampers their ability to gain insight, to be creative, to identify priority targets, et cetera. So it's something that-- a nuance that's important for enhancing the work of those teams.

OK, so we've talked about the right people. I've talked about some things about the right goals. And now, I'm going to discuss briefly good collaboration. This is a big topic. We can go on and on with this. But I'm going to share with you a few key findings in this regard. So in our studies, when participants come into the lab, we often record things about their communication. And so, in some of our studies, they've worn these gizmos known as sociometric badges, which pick up all kinds of information about spoken communication, about people's non-verbals, how close in proximity they are to one another, et cetera.

We focused though on patterns of spoken communication. And what we find is that in teams where there is a more even distribution of communication, the team is much more collectively intelligent. Or in other words, if you have one or few people dominating the conversation, it really hurts collective intelligence. Relating this back to having the right people on the team, we also find that when you have more women on the team, and when you have people with higher social perceptiveness on the team, it's much more likely the team has an equal distribution of communication. So this is another reason why the team composition affects collective intelligence.

These findings, it turns out, also generalize to teams that are collaborating remotely. So we've looked at this in teams that are working on the internet and only communicating via text chat. And we find the same patterns. First, that actually having more women in the team, interestingly, predicts collective intelligence, even when the other members aren't sure who's male or female. And also that teams that participate more equally are more collectively intelligent. So a high level of equality in communication is important. But I'm going to get back to this study at the beginning, where I show you the expert teams that didn't perform well and the lack of integration.

When we dug into the reason for that, what we found was that in the teams where we didn't force them to spend time planning how they were going to work together, they essentially worked in parallel. They just divvied up the information, the visual expert looked at the pictures, the aerial photographs of the site, et cetera, et cetera. They just did their own thing independently. So there is a framework that we use for thinking about how people work together. And we label different levels of inter-dependence. And so the lowest level of interdependence we refer to as pooled interdependence, because everybody is working independently and just feeding into some central repository. It could be a report, whatever it is, chapters of a book, an edited book, et cetera.

The second level is what we know as sequential interdependence, where I work on something and I pass it along to Scott, and he adds to it and passes it on to Mark, et cetera. And so there is a little bit more integration, but not as much as at the highest level, which is reciprocal inter-dependence, where we go back and forth and iterate on things. So what we found was that the experts were working with a pooled interdependence. And this is actually fairly common in other studies we've done in consulting firms, in investment banking firms, et cetera, where people are very time pressured. It's much more efficient to work at this lower level of interdependence. And so people tend to fall into that. I just also relate these two different sports, because sometimes it also helps people understand the differences.

So pooled interdependence is a lot like being on a swim team, not because it's "pooled" interdependence, but because everybody does their own individual event and it gets added together. Sequential interdependence is more like American football, where the offense affects the defense, the defense affects the offense, but they don't interact in real time. And then reciprocal interdependence is like what the rest of the world calls football, or basketball, where there's much more give and take. And so pooled interdependence is very good for efficiency. Sequential interdependence is good for accuracy. But integration is really fostered by reciprocal interdependence.

However, many teams, again, fall into favoring efficiency. And so they undermine their ability to really integrate the diverse expertise they've brought together, and ultimately, the collective intelligence of the team. But it turns out that it's not that hard to foster integration. I described the very simple exercise we did at the beginning of the simulation with the intelligence analysts. Another example comes from the medical community with the safe surgery checklist. This was an innovation that came out several years ago, has radically brought down the rates of post-surgical infection. And it turns out to be a very simple thing, where they go through a number of steps, really to check on details, but it also gets the team communicating-- what's everybody's name? What is the surgery we're conducting? Which arm are we chopping off? You don't want them to get that wrong, et cetera. Who's taking care of what?

And it turns out that it really does foster a higher level of teamwork. So thinking about the simple tools that you can bring into your teams to launch the pattern and the norms for fostering a higher level of integration can be a huge benefit, especially if you're integrating diverse expertise. So I've mentioned a number of things that come out of the studies we've done. You might be asking yourself whether this matters. And so we've looked at a few situations in which we examine the degree to which all of these things are present and if it predicts performance.

It turns out, as you might expect, that it does have a great effect. So in the first study, we looked at 64 analytic teams and six intelligence agencies, evaluating the degree to which they had these conditions in place, and looking at the performance evaluations of those receiving their work. And we found that together, it controlled over 70% of the variance in their rated performance. In another study, looking at 120 senior leadership teams, what we found was that the degree to which they had these things in place predicted over 50% of the variance in their performance.

And interestingly, the senior leadership teams, the thing that they struggled with the most was having the clear goals and the appropriately focused goals in our study. So in summary, the pool question at the beginning set up some of these as potential predictors of diverse team performance. It turns out all of them are strong predictors, in combination with maybe a slight tilt toward having the right people when we compare the relative proportion of variance controlled by the different factors. But certainly, you can't simply put the right people in the room without thinking about how they're going to work together, as our simulation in the intelligence community demonstrated.

So just by way of wrapping up, what our research strongly suggests is that collectives are characterized by a stable level of collective intelligence that is measurable and that can be used to predict future performance. Cognitive diversity can enhance collective intelligence if it's managed well. So to really create the conditions for collective intelligence to develop, it's important to A, recruit the right mix of people. Here, our findings definitely point to both cognitive style diversity, gender, and social skills. To clearly and correctly specify the goals that they need to be working toward in the context, given the outcomes that they're trying to achieve. And then for leaders of those teams to focus on setting the structure and the norms early on, so that the team can achieve a high level of integration and make use of that expertise.

SCOTT PAGE: My name is Scott Page. It's great to be here. And it's actually just wonderful to present with Anita. She mentioned-- Anita does a lot of work in collective intelligence, and what has it been, like six years? Seven years, there's a collective intelligence society. And it's a group of people who are-- some people study ants, some people study people, some people are like me and just write math on the board all the time, and that sort of thing. And what we try to do is we try to understand what is it that makes a collective of people better than an individual.

So one of the sort of charges put before us in coming here was not to make the business case-- in fact, it says in the introductory paragraphs for this conference that we understand the business case. And I'm going to take a little bit of exception to that. And I think that if we really want to see the extra margin returns, if we really want to see the benefits of diversity, we have to understand how and why it works. And so a lot of what academics do, what Anita does, what I do, is we're splitters. So we take sort of big statements, and we divide them up into component parts, and try to understand how they all work, right people, right, process, that sort of thing.

So I'm going to start with this Di Vinci quote, and I'm going to go through about 97 slides in 26 minutes, that if you love practice without theory, you're really sort of going out there without a rudder and a compass. And it's absolutely true. So what I want to do is I want to give you not a single case for diversity, what I want to do is I want to unpack multiple logics. There's all sorts of very distinct logics that explain where diversity bonuses come from.

And so, this is a different narrative. Then I'm going to talk briefly and how that relates to gender diversity. And then going to do two specific cases in some detail. I'm going to do prediction and problem solving. And then once we've got some understanding, I'll talk about how do we lead in? How do we change practices? What do we do? Anita hinted at of some of these. I'll echo some of her statements. But what practical steps do we take within organizations and society to enable everyone to bring their whole selves and to make better outcomes?

OK, so when we think about the case for diversity, it originally, whether this is race, gender, sexual orientation, it's often phrased in terms of social justice. It's increasingly framed as a demographic imperative, in the sense that American society-- the people who are 85 years old are 85% white. The people who are four year olds, are less than 50% white. So we are just becoming a much more diverse society, we're become more global. It's a democratic imperative. The problem with the social justice argument, and the problem with the democratic comparative argument, a problem with them, is that they often balance off, or trade off, diversity with excellence.

It's sort of like we have to be diverse, but somehow that's going to cost us. And the affirmative action debate is often framed in that way. Justice Scalia, using legal language that I didn't quite fully understand in describing the University of Michigan's case to try and let people in on the basis of diversity, said that we were sacrificing-- here's the legal term-- being a "super-duper university" by being more diverse. What I'm arguing is the exact opposite is true. If you want to be a "super-duper" place, to use the legal jargon, you actually need to be diverse.

But the logic isn't a single logic, it's multiple logics. And in terms of how to think about this, I'm going to start with a quote from Katherine Phillips, who's a good friend of mine, who's the Dean at Columbia, and she says, in some ways, intuitively it seems obvious to be the case may, if I take a bunch of people who think differently and are really smart, they should do better. What's less obvious, If I take people who come from different identity groups, that they do better. But yet if you look at tons of data-- and Katherine's one of these people who's read like 1,000 studies, so whenever I want to know about a study, I just call Katherine, and she tells me what it's about-- Is that socially diverse groups, identity diverse, differences in who we are, effects how innovative we are, and affects the quality of our outcomes.

So why is that? Well, one thing we can do is we can just say it's magical thinking. And there's a lot of people in this space. They'll just say two heads are better than one. Now the problem with saying two heads are better than one is you can also say too many cooks spoil the broth. And so we don't want to get into sort of a battle of analogies. And the problem with analogies is they don't give any conditional logic. So the reason we write down models, the reasons that it's actually worthwhile for society, for Anita and I to do what we do-- sometimes we question that-- but the reason it seems to be worthwhile is we sort of flesh out the condition of when something's true.

So when is it the case that he who hesitates is lost? And when is that the case that a stitch in time saves nine? You want to identify those conditions. So what happens is people will sort of invoke analogies. To my good friends have recently written books on analogies. And one of the great things about analogies is we often use really crappy analogies. So one things people will say is like, a group of people is like a portfolio of assets and that's why you want a diverse group, bad analogy. Here's why it's a bad analogy.

When I invest in stocks, I get the mean of all those stocks. So I invest in airlines and oil companies. So when the oil prices are good, the oil companies do well, the airlines do bad. When oil prices are low, the airlines do good, the oil companies do bad. And it balances out, but the thing is, I get the mean. When I'm putting together a group of graduate students to sit around and decide something, or a group of co-authors to work with, or I'm being put on a team of co-authors, we don't get the mean. We just get the best. We get the best idea that comes from the group.

And actually, and Anita's done a lot of work and this, you get better than the best, because what happens is you get re-combinations of ideas. So people who've done in-depth studies of groups, Kevin Dunbar of pharmaceutical groups, what you find is people come in, they present ideas ideas, idea get recombined. So you're not getting some averaging minimized value at risk, that sort of stuff. You're actually getting the best of the best. So the case for diversity as some sort of loose analogy as a portfolio is the wrong way to think about it.

So what you want to think about, I think, is you want to think about cognitive diversity somehow leading to better outcomes. But this can't be real science, because it's not even a two by two box. It's just like circles with arrows. We've got to really do better than that. So if we can just think, and say what do we mean by cognitive diversity? We mean things things, as Anita said, we mean information, we mean knowledge, we mean mental models, we mean representations and perspectives. We mean the analytical tools or heuristics that people apply given a problem, like the things you learned in graduate school, the things you learned on your paper route.

Then what happens is those repertoires get mapped to specific tasks. So the role that diversity plays in innovating is very different than the role diversity plays in let's say truth verification. And let me be precise with this. One of things that Katherine talks about a lot, Katherine Phillips talks about a lot, when you put identity diverse people in a room, you get more ideas. So diversity means more. When you put diverse people on a jury, or diverse people trying to figure out the truth of the matter, you actually get rid of stuff. You find errors in things. So you actually get less. So depending on the task, diversity could give you more, or diversity can give you less.

And in each case, in one case you want more, and in the other case you want less. So it's not just one simple story. It's also not a simple story where I can draw one arrow. It kind of looks like this. That's why I've got 92 slides. Because it's complicated. I'm a professor of complex systems, so I try to make everything as complicated as possible. The point is these things, if I want to think what matters for predictions, it's things like mental models, it's representations. If I want to think about what matters for problem solving, it's things like heuristics and it's things like representations.

So when we think about that complicated picture, but all these things inside our heads that enable us to collectively be better, than we want to ask, how does the other stuff matter? How does the diversity stuff matter? And so within a university, we think a lot about things like gender and race, and how to think about it. And there's only three predominant frames. So I'm just going to very quickly go through these three frames.

The first frame-- I'm going to borrow the analogy from my Mia Sen and Omar Wasow, called "The Bundle of Sticks." And the idea here is when you think of someone's identity, there's all these things. There's genes, dialect, gender, neighborhood, social class, that sort of stuff. It's not one thing, it's a giant bundle. And the reason they use the word bundle is because these things are all interconnected. The second metaphor, and Anita actually use this, as "The Iceberg." So when we think about identity diversity, there's the stuff we see. So you look at me and you think Scott's a male, he's a white guy, he seems to have a nice Midwestern accent, that sort of stuff.

You see physical qualities, but you don't necessarily see things like family status, social class, that sort of stuff. Now what's interesting about "The Iceberg" metaphor is that iceberg that we see day to day, minute to minute, is different than the iceberg that you see as financial analysts, and is different than the iceberg that Anita sees when she does her studies. Because a lot of times when they actually pull data from the census, we're seeing stuff that's below this sort of visual water line, because we know things like social class, zipcode, neighborhood, that sort of stuff. So "The Iceberg" analogy is actually much more subtle. There's the experienced iceberg, and then there's the sort of like data iceberg.

The third analogy is kind of the money shop from biology. So people pay a lot of tuition dollars to go to universities, and this is biology departments money shop. Opening lecture, Bio 101, at a lot of universities, the professor gets up and she says there is no great blue heron. And the students are all sitting there going what are you talking about, I've seen one. She's like here's Audubon's picture of a great blue heron, and there's no such thing. What do I mean? I mean is when we define a species, we mean a population of things.

So there's a population of things that we call great blue herons, and while I've been talking, a great blue heron has died. And while I've been talking a great blue heron has been born. So what it means to be a great blue heron has actually changed during the course of this brief conversation. So the great blue heron this population of things. There is no one. So if you look at people, or if you say we need a woman on our board, or we'd like a more female representation, that sort of thing, it's not like there's such a thing as a woman's perspective or something like that.

Women are like great blue herons, so are men. We're a population of things, and our population is incredibly diverse. And so when you think about how this works, this sort of causal story-- so you think of the amazing data that Julia was showing us earlier-- there has to be some sort of causal link in some ways, between this identity diversity, cognitive diversity, and better outcomes. Now what people tried to do-- 15 years ago, 20 years ago, 25 years ago-- was this. They tried to say, oh, let's look for how gender affects information, gender affects knowledge, gender affects networks. Let's look at how race affects knowledge, race effects information, that sort of thing.

And that kind of worked. But then, we realized that was wrong. It was wrong because everything affects everything. You are not just a woman, or a man, or an African-American, or an old person, as my kids call me. What you are is you're all those things. You're the bundle of sticks. And it's that bundle of sticks that gets mapped into all these ways of thinking and knowing, and that sort of stuff. And that's what gives you collective advantage.

And so when people talk about-- this is the fancy buzz word in the Academy now-- intersectionality, it's the idea that you can't just pull off one component of my identity, or Anita's identity, or anybody else's identity, we're all this messy bundle of things, and that messy bundle of things gets mapped to how we think. And differences in how we think have value. So how do I prove that? Or how do we think about that? And why is if we think about that, we actually get to more interesting ways to run our organizations or better ways to run our organizations?

So let's slow down. Let's talk about prediction for a second. So I'm going to predict something. If I want to know is this stock going to pay off, or if I want to know if the Tigers are going to make the playoffs, I need information, and I need knowledge. So what's the difference between information and knowledge? So data is the raw stuff that's out there. Information is taking that raw stuff that's out there and putting it in categories in some way. Knowledge is understanding causal and correlative relationships between that knowledge, like force equals mass times acceleration, that sort of thing.

And that's where the mental models come in. So I've got information, I embed that in some sort of knowledge, and then I have mental models to think about, OK, how do I map that into some sort of prediction. So here's an interesting sort of fact. If I have a bunch of people make predictions, the crowds' error-- and I'll put this more formally in just a second-- equals the average error, minus the diversity. So if I ask how good is a crowd of people at predicting something, It turns out it depends on two things. It depends on average error and it depends on diversity.

And when I put up an equation like this, because I'm a mathematician, it's like a real equation. So it's not on of those equations like team equals thoroughness, plus effort, plus ambition, plus meaning, or something. It's not one of those. It's like math. And what the math says, which is sort of interesting, is that this first term on the far left, C is what the collective predicts. That's sort of collective intelligence, thing. And theta, which is t is, the truth. So how far is the collection from the truth, that's the squared error, that's how far off the crowd is. So C is just the average of what people think. T's the truth. That's how far off the crowd is.

So the first term is just that big E, is just summing up each side, that each person's prediction, that's person one's prediction, person two's prediction, and so on, divided by n, the number of people. So you might think a crowd is as good as the average person in it. And it turns out that's not true. A crowd is better than the average person in it. And the amount by which a crowd is better than the average person in it is equal to the diversity of the crowd. Here's the funky thing, which in the diversity, just how different are people from one another in their predictions, not in what they look like.

But the thing is differences in what we look like tends to lead to differences in how we predict. So what's kind of weird about this is if I said to you, you want to make a good prediction, should you have smart people, or should you have diverse people, the answer is yes. You want both. And you want them in equal measure. I mean, I could have done the math, and other people could have done the math, we all could have done the math on this. And it could have been it's 100 times ability, plus a little angel dust of diversity. It's not. They're equal, they come out equal.

Now the interesting thing is, I give an online course called model thinking, that hopefully, this winter, I'll get over my millionth student in this class. And people will send me data, and they'll say Scott-- the email always starts like this-- Scott, I hope you're sitting down, but I took data from our firm, or this thing we did, plugged it into that equation, and it works. And I send them this little email back with a right triangle on it, where one edge is 5, and one edge is 12, and the other edge is 13. And I say I hope you're sitting down, check this out. 5 squared, plus 12 squared, equals 13 square. The Pythagorean theorem works again.

So the point is this is always true. This isn't some loose idea that somebody wrote up on a napkin. This is an algebraic identity. The management challenge is to think about how do we get smart people who think differently into the room. So let me show you some data that really drives home how this works. This is some wonderful work by Jack Sole, he's a friend of mine, [INAUDIBLE], 28,000 forecasts by economists for six economic indicators.

Now, the red dot you see at the top, which is the one that's the baseline performance of a single economist. The blue line you see at 80% is the median. And the orange line, at 79%, which means basically 21% better, because these are errors, is just taking an average. So if you just take the average of all these economists, you do 21% better than taking a person, even though the crowd is exactly equal on average to an average person. It's just that because they're diverse, they do 21% better. Now the thing I've circled here, if you actually take the three, four, five, six, or seven best individuals, like a small crowd of people who are all good, they do substantially better. They do 27% better than an average person. And they do 12% better than the crowd.

So the idea is not that you want-- and this is one of the things we've learned from collective intelligence-- you don't necessarily want to take bring everybody in. If you bring everybody in, you'll get well my undergrads call a dog's breakfast. I don't quite know what that means, but that's what they always say. So I'm just going to call it a dog's breakfast. But if you can bring in like seven or eight people who are really bright and are different, you're going to end up doing a lot better. Now, how do I know they're different? I know they're different because, look, the best one, the very best person, if you knew at the end of the time who that person is, she's only 15% better. That's that green triangle at 85%.

But when I bring in the second best person, who's not as good as her, and link him with her, they're now at 78%. When I bring in the third person, who's not as good as the first person or the second person, they're down to 76%. So the point is diversity is ability. So Preston McAfee, who's one of my-- good friend of mine. He's chief economist at Microsoft. He has this wonderful quote. If you do the math on this-- he and I both did it, making sort of different assumptions. You have to make some assumptions about correlation. It turns out a good rule of thumb is as long as the person isn't 50% dumber than you, you should listen to them.

So this is really comforting, because you can still walk around and think I'm 40% smarter than everybody else and I'm a good listener. So you can feel that, but just don't think you're 51% smarter. So Christina Romer knew this stuff, and she was advising Obama on the economy. She was his chief economic adviser. So for fun, and this is a case of tapping into diversity with an organization, if you sit on the FOMC, you get these three books. You get the Blue Book, the Beige Book, and the Green Book, and then you decide what policy should be. So to have some fun with the FOMC, because she was separate from that, she just took the Blue Book, which has all the forecasts from the economists. And she says, I'm just going to compare averaging the 600 economists to what the FOMC says.

And so she ran a simple regression. And just for fun, let's look at unemployment. So this is just a simple linear regression. The weight you'd get on the staff forecast would be 97%. And the weight you'd get in the FOMC was minus 0.3. So what you're getting here is you do not way better, but like, oh, my gosh, categorically better, by just averaging the predictions of these people in-house, than having a committee read this log book and then said what did you read, what did you read. The numbers are all there. Just average them up.

Now, actually, on growth it turns out the FOMC does a little bit better than they do-- then the staff forecasts. But the point is information is being lost. And this was her point. You have all this stuff in the Blue Book. You should be using it better. Now, if you look at things like investment houses, and this is some work-- I don't think his paper's published yet, but it's been approved-- 25 years ago, 75% of mutual funds were run by individual people. Now, 75% of mutual funds are run by teams. It's been a complete reversal. The reason why is 60 basis points.

So if the proof is in the pudding here, teams are just better at managing funds than individuals. Now, if you look at-- this as a Morningstar report that came out recently. And there's also data on women only firms, but there's not many funds. If they look at men only funds and women and men funds, women and men funds have lower expense ratios and 10 year versus the mean, they're twice as high-- not surprising. Let me step away from prediction before I get to we lean in on this.

Another place you can look is you can look in the Academy. Now the Academy is a funky place, because of the fact that we sort of get optimal teams. So in Anita's papers and in my papers, you'll see like sometimes I'm writing with Lu Hong, sometimes I'm writing with PJ Lamberson. I'm writing with different people all the time. And she's writing with different people all the time. We pick our teams to get the right people. So we collectively decide who gets to work with one another. And so as a result, we get almost ideal teams. So we're like the best case scenario.

Well, this is a chart that's incredibly hard to read, because it's basically a National Research Council report, and they're made to be boring. They're like sort of sleep aid devices. But on the left-hand side, you have a log scale. So it goes 1, 10, 100, 1,000, 10,000. The thing to notice here is the orange line that starts out between 100 and 1,000. So this is the number of papers per year coauthored by between 6 and 10 authors. So in 1960, that was approximately 240 papers. In 2015, that was approximately 240,000 papers. And that is now the model paper. So if you opened up a paper in an academic journal, the most likely combination you would have is 6 to 11 authors, more than one author, more than two authors.

In fact, in science, 90% of papers are co-authored now. And all those that aren't co-authored, the single authored papers, most of those are like survey articles, or articles where people write let's talk about my work and how good it was, like some sort of introductory thing. Not my work, personally, whoever is writing the article about themselves. So what you get is-- we've found that there's been this massive shift, far exceeding what we see in running mutual funds, towards co-authored work. The reason why is these are hard problems.

So when you think about the business case, for diversity, the business case for diversity really only makes sense now. It didn't make sense when people were like chopping down trees and laying railroad ties. If you're hiring people to chop down trees, you line them up, you see you can chop down the trees the fastest, and you hire those people. It's an additive, separable task. If you're trying to cure cancer, if you're trying to figure out how the brain works, it's not an additive, separable task. It's incredibly hard to do, and you need teams. Now you could say, OK, well, maybe these people just like working with each other. Maybe this is risk aversion. Well, it turns out, co-authored papers have four and a half times the likelihood of being high impact, where high impact is measured by getting 100 citations.

So if you take an average-- this is a small sample set, by the way, it's only 21 million papers. It's only every paper ever published. So if you get every paper ever published, and you put in one pile the papers written by one person, and you put in the other pile the papers written by more than one person, and then you say what are the odds that this thing gets 100 citations? The one in the pile written by multiple people is four and a half times as likely to, four and a half times, not even close. If you unpack it, and this is work by Brian Uzi, what you find is they call it atypical combinations.

So what they find is that the yellow dot, the yellow bar, is sort of like an average paper. The red bar are papers that sort of go deeply into a particular literature. The blue bar or papers that sort of make unique combinations between literatures. And the green bar are papers that go both deep and make unique combinations. So for example, the kind of work that Anita's doing, where they're doing things on group performance, like team performance, and cognitive psychology. So they're addressing two deep literatures that typically aren't put together.

So Brian will sometimes describe this as Reese's peanut butter cup kind of stuff. Deep theory of chocolate, deep theory of peanut butter, put them together, and it's like magic for the month part. So Melissa Schilling took a big collection of the high impact papers, and said let's really trying to unpack this by doing much more careful study. And it turns out-- this is an odds ration, so 14 means 14% increase-- so every year you get older, you get a 14% increased chance of writing one of these papers, which bodes well for me. Prior cites double your chance. So if you've written more papers in the past, that helps. Author count actually hurts you. So this is really strange.

Because I just told you you're four and a half times as likely to have a home run if you have multiple authors. And now, if you actually run the regression, you find out you have a 20% less chance of writing one of these every time you add another author. What's going on? What's going on is the last two things. So depth, I put HHI-- and this is the Herfindahl indice-- What she does is she breaks different areas using like the Dewey decimal system. So there's cognitive psychology, there's economics, there's group psychology, that sort of stuff. The more different disciplines you cite-- you cite two disciplines as opposed to one, you're three and a half times as likely.

And if you connect papers that nobody's connected before, you're 15 times as likely. So when Anita's talking, and I'm talking, about diversity really mattering, there it is, there it is. What's happening, what makes for high impact in academic research, is combining different ideas. Now, Melissa and I joke about this a lot. The author count thing is like a theory of marriage, and this is why you get the best. It's a hassle to be married because you disagree a lot. You think about parenting, it can be a pain. You'd rather just do things your way. But because the other partner usually has other ideas and knows different things, you end up making a lot better decisions. You don't get the average.

This is once you're not getting the average, you're getting the best. So if you think about evaluating an action-- so let's now think about what you do. And let's think about a firm's thinking about adding another company, thinking about adding a new product line, approving a drug, this is some work by Juliet Bourke from Deloitte, she says, look, one way we do this is we sort of parse this down. We say what's the effective outcomes, options, processes, people, evidence, risk. This is their rubric. If you go to McKinsey, they have a different rubric. But they're fairly similar.

So in working with Juliet, one of the things we do is we can link that then to the thing that I showed you. And you can think, well, how do you think about the actions effecting outcomes? Well, to do that, you have some sort of causal model. When you think about what are the possible options, what are the things we could do? Well, the set of options you can think of depend on how you represent things. So Steven Johnson, who has written extensively on innovation, calls this adjacent possible. If I present some idea to you, you encode that, you represent that in some way. And given how you represent it, there's some set of adjacent possible.

So let's take a particular case. Chipolte-- Chipotle's great, McDonald's bought a big stake in Chipotle. And this turned out to be a good thing because McDonald's made a lot of money. I mean, even Obama went to Chipotle, which was good, right? But what happened was there was a people issue. When you look at this action there was a big people issue. And the people issue basically stemmed from the fact that McDonald's is very process oriented, very efficiency minded, very cost sensitive. And Steve Ellis who is the CEO of Chipotle, he was about the food. This is an organic thing.

The reason McDonald's bought it is because they wanted to say can we actually make these supply chains work in a sort of a free range organic space for-- they're not called fast food anymore, they're convenience food, whatever the category's called. And this seemed-- so McDonald's ends up bailing out because there's this big fight. And after they bailed out, Chipotle had a huge year. And McDonald's had a bad year. And people were like, boy, McDonald's got out at the wrong time. The thing is though, McDonald's was bringing a lot of process knowledge.

And it turns out you can't spell Chipotle without your E. Coli. So this is like there was a process issue. And so when you think about how you do you work, your implicitly coming at this in all these diverse ways. You've got all these diverse frames. And within each diverse frames, you're also exploiting diversity. So what do we do? How do we sort of lead in here? What sort of leadership patterns do we have in order to create the sort of inclusivity, the set of interactions that are going to make this work? So one way we do this, I think, is to recognize the water-- and Juliet talked about this film in terms of culture.

There's a wonderful graduation speech by David Foster Wallace, where he talks about two fish swimming in the water. And this older fish comes up to him and he says, "Hey, guys how's the water?" And they like look at each other, they're confused. The older fish swims away. And then one younger fish looks at the other and says, "What the hell is water." Here's water. And the interesting thing about these guys, and they're all guys, is they only went to school with guys. One of these guys once said to me, who works for a company I won't name, for obvious reasons, "Boy, Scott, I was surprised how many smart ladies there were." I was like horrified at first, until he told me his story.

And then once he told me his story, I realized he'd had gone to Exeter, went to Princeton. He never had a class with a girl after at age 11. And he never worked with a woman ever until 1994. These people are still in the pipeline. They're about to leave, but they're still in the pipeline. So we have to think about though they've made the water we're in, and how do we change that water? One way we change that water-- and this was talked about this morning, in Terrance's, I think, just absolutely fabulous survey talk-- is getting these hours right.

So this is Michelle Peluso, who's a friend of mine. She ran Travelocity. She walks out every day at 5:00. She may go back home and work for seven hours, but she walks out at 5:00. And when she does, she is very loud about it. I'm walking out, it's 5:00, I'm walking out. Because she wants to change the water. The other thing we've talked a lot about, you guys did this morning and it comes all the time, is these small biases. And one of the things that is so true-- and here's where a little math goes a long way, is small biases are big biases.

And so if you look at Marianne Bertrand's stuff, or [INAUDIBLE] stuff, where they do these sort of blind things and change the label from a man to a woman, or an African-American sounding name to a Latino name, to an Asian name, to a white name, you see these little tiny biases. I mean 5%, 6%. You don't see 80% biases. They're always 5%, 10%. Well, here's the interesting thing, if there's a series of promotions you have to get, and you're slicing 5% at each level-- this is just a little mathematical exercise. If I start out with even numbers of men and women, then I'm at 57%, then I'm at 58%, then I'm at 80%, then 83%. [INAUDIBLE] like all men.

So one of the interesting things is like the Academy prides itself on having all these women leaders. One reason the Academy has a lot of women leaders, is we've only got four levels. You go-- you know, Anita's going to become tenured, then they're going to make her dean, and she's going to be president. You don't have-- there's not many ways to go. If you go to Boeing, I was talking to Boeing, they've got like 14 levels. And if you fit their data, if you just assume like 5% discrimination at every level, it just works perfectly. Here's a major brewer, 31, 22 , 21, 14, 7. You can just watch it slide off. So there's no such thing as a small bias in a hierarchical organization. You're slicing an edge off that cube every time and it's a big bias.

You also run into this all the time. Whenever I'm at a firm, you'll see someone say, look, I'm all for this, but I'm a quotas. I think one pushback on that has to be you've got to be against performance discontinuities. And so-- Anita hinted at this some-- what do I mean? If you look at the percentage of women, percentage of people of color, percentage of differentially abled people, there are these sort of threshold effects that once you fall below a certain number, you have no role models, no mentors, it's harder to recruit, it's harder to retain, you feel like you're a token.

When you think about that intersectionality, you feel like you're supposed to-- instead of bringing your whole self, you go into every meeting saying, oh, I'm supposed to be the guy here, or I'm supposed to be the woman here, I'm supposed to be the person of color here, as opposed to I'm supposed to be myself with my expertise. And so, it was talked about this morning, are these things like linear, are they convex, are they concave? There really seems to be, looking across a lot of literatures, critical thresholds, that if you don't reach those, you're just making the work-- somebody's well-being isn't going to be very high. And so no one's-- people want to be against quotas, but they also should be for well-being.

When I think about this, and again, I come at this from someone who's like a mathematician trying to think about why is it on complex problems diversity so important, that if diversity is the same thing as talent, we need to act like it. So what we do in a lot of companies is we evaluate each person, and we promote the individuals who do well. Some companies, like IDEO, do it quite differently. They evaluate the teams, and your performance as an individual at the end of the year is how well your teams did. Because you care about people who make teams good, not themselves good.

So there's all sorts of little things. So this is Howard Ross, he's a friend of mine from Cook Ross Consulting. He's found that extroverts do really, really well if you give them information at the point of the meeting. Introverts do horribly. Introverts do better if you give them information before. And so what you should do is get agendas out in advance, so both introverts and extroverts have something to share. David Thomas, who's now the Dean at Georgetown, he's found, with Robin Ely, who's at Harvard, really compelling evidence that if you put a group of people together and they're diverse in any sort of way, and you don't explain why the people are there, both to everyone else, and to the people themselves, the team doesn't function well, because people think, oh, this team is diverse because politically we have to be diverse.

So it's super important when you're running a meeting, just take a moment and say here's why the five of you are here, and here's what-- this is why we chose these people. And then once you've got the people there, you won't need to oblige everyone to participate. Anita showed fairly compelling evidence of this. It doesn't do any good if I'm in the room and I don't share anything. Some places actually make it mandatory. So Ford, Boeing, once you get above a certain level, if you're in a meeting, you kind of have to talk. Now the thing is you don't want to create a culture of fear, so there's different ways to allow people to participate. But it's very important that everyone feels like there's a reason I'm here, because I'm bringing something of value, and I'm going to share that in some way.

And one of the really tricky things, and this is some work by my colleagues, Daniel Romero, and I'm going to steal his phrase, "Turtle Up." This is with a financial services company, and he's got the email-- he's got all the emails. He doesn't have the content. He's just got who emailed who. So what's happening is everything goes along fine, everybody's emailing everything. And then you can look at volatility in the market. When the market gets volatile, this is what happens. People "Turtle Up." They only talk to the person they know best. That's exactly the opposite-- Anita's like shaking her head-- that's exactly the opposite of what you want to do.

When things get volatile, and it's really complex, and you don't know what's going on, you should be talking to more people, not your buddy who you went to Princeton with or something. So what you want to do is you actually want to encourage people, when things are hard, open up. So Genentech in one of their financial units, a couple of years back, did the following on their key performance indicators. At the end of the year, one of the questions was, what was the most important decision you made this year for Genetech's bottom line? The question below it was, who from outside your silo, because they use that term internally, who from outside your silo did you ask for help? So basically forcing you to not "Turtle Up." Because there's a natural tendency, when you feel like, oh, my gosh, I don't quite know what I'm doing, to not go out and ask. And what you need to do is you need to put in place some sort of procedures so people do that.

Two last things. One, is you sort of have to acknowledge the discomfort from this. This is a paper by Maggie Neal and Catherine Phillips, who I showed before. So they did the following-- I call this shooting fish in a barrel experiment. They had a bunch of people read a who done it, and then what they did is they put groups of people who all thought it was the same answer on teams. And they put teams who thought it was different people on teams. And so what happened is the diverse group did better. Well, we have that mathematical theorem showing that has to be true, unless by talking, they really screwed up. So we know this true.

So I'm reading this paper and I'm thinking why did you do this, but then the second part of the paper is just brilliant. Here's what they did. They let people bet money on if their groups were right. And it flips. The reason why, if you're in a group with a bunch of smart people-- and these are all Northwestern undergraduates, so they're smart people-- and you all agree, you feel really confident. You feel like we're good. If you're in a group where people disagree, then that means somebody is in the minority opinion and their decision doesn't win. And then it's less fun, and then you have less confidence in the group.

So I sit on the university's-- the way it works in Michigan is when someone like Anita goes to a department, and the department votes her for tenure, then she goes to the college, and the college has some like star chamber of secret people who then decide did the department-- was this an inside job, and what happens is sometimes I get out voted on that committee. And I come home mad as hell. And my wife shows me this slide. Because the point is if I were all knowing, it would just be a committee of Scott. I'm not all knowing, and because I'm not all knowing, because we're a diverse team, we're going to get some stuff wrong.

So last thing, and this relates, I think, more to race than gender, but it's super important. Kim Scott has a wonderful new book out called Radical Candor. And when you think about allowing people to move on in their careers, you need to both care about them personally and you need to challenge them directly. Because if you only care about people, and you're like, oh, it's so good to have you here, but you're not constantly challenging them, you're not constantly pushing them, you end up in this box of ruinous empathy. And to really allow people to succeed, you've got to both-- you've got to challenge them. You've got to call them out, and you've got to help them improve, and let them build the repertoires that make them contribute to your firm and your organization. All right, let me stop there. Thanks.

MARGARET FRANKLIN: You won't be surprised. We have lots of great questions. And they don't relate to the data. It's the how do we take a look at this data and do something. So let me start with one for both of you. And there's a couple of parts to it that I think touch on leadership, because what you've talked about is groups, but not necessarily the impact or influence of leadership, and particularly some of the last stuff that you put up really. I mean your work is in the lab, but the last stuff you've put up really has elements of leadership.

So what is the tipping point or optimal mix to ensure a diverse group? And then, I love-- whoever put this out, put a little story-- I participated in many meetings where if one attendee has offered a contrarian point of view, there's a pause, moment of discomfort, than a dismissal and the previous conversation just moves forward. What does it take to get the group to acknowledge the point of view and discuss it before dismissing? And maybe, in that, talk about the impact or role of leadership.

ANITA WILLIAMS WOOLLEY: So the first part of the question was optimal mix of diversity?

MARGARET FRANKLIN: Yes.

ANITA WILLIAMS WOOLLEY: OK, well, I think Scott touched on this. It's a little bit evident when we looked at the gender diversity. But certainly, in looking at integrating a different group into an organization, what they find is that when the proportion of those people approach 40% is when the salience of their group membership dissipates and suddenly, they're individuals again, versus representatives of that group. And so when thinking about composing groups, as Scott even mentioned, if you're just the woman, or the whatever, and you're representing that group, then it doesn't allow you to necessarily share your expertise or the group to receive it in quite the same way.

And so I'd say in composing groups with regard to ideal diversity, keeping some of those proportions in mind and trying to integrate key perspectives, but maybe at least give people a partner, so there is other work showing, or even having a duo versus a solo token is better. So that's the ideal diversity piece. I would say that the piece with picking up on different points of view, I view leadership as something that team members can also contribute to, in addition to the specified leader. But certainly, in that case, a specified leader should have his or her ears open to pick up and not allow the group to just gloss over those points.

And so, if somebody brought up, especially a useful point of view, and it got just glossed over, in a way that is a failure of the person leading that meeting. It's also a failure of the other members to not have their ears open for these different points of view. And so that's definitely a process issue that the group would need to address.

MARGARET FRANKLIN: Scott, do you have anything to add

SCOTT PAGE: No, I think you're good.

ANITA WILLIAMS WOOLLEY: He can write the equation for that.

MARGARET FRANKLIN: How do you-- both of you have identified key elements, and I would say sort of matrixes or checklists of things, but when you look at in your lab, or otherwise in your research, how do we learn to recognize good people? Is it sort of an equation? Can you just apply this and say we're going to go out and seek this because this is important, we're going to find those people, and then were to articulate it to the group? Is it that simple?

SCOTT PAGE: No, I think this is a really interesting question. Jon Kleinberg, who's a computer scientist at Cornell and is certified by the MacArthur Foundation as a genius, so he's probably pretty smart, he has this recent paper where he shows that if the task is sort of non-separable. So you're not chopping down trees. So it's a question of [INAUDIBLE] that there is no test of individuals, such that if you give a whole group of people the test, that you find the optimal team by picking the best people on the test. It doesn't exist. you cannot write it. And the reason why is it depends on-- you need this-- what I'm laying out-- you need this diversity of representations, you need diversity of heuristics, you need diversity of models, you need diversity of knowledge. And so this is-- I sometimes call this the John Milton problem.

So when Milton was alive, there were 30,000 books in the British Library. He read 12 languages, he wrote in seven languages. He pretty much knew everything, close. If you figure he read two books a day, there was no TV. He read two books a day, he pretty much got through everything anybody could know. Now, there's actually more content produced in the field of neuroscience alone in a year than existed at that time. So no one can know anything. And so the thing is that this idea-- talent just doesn't make sense anymore. Everybody's got talent.

What you want to think about is the right mix of talent. And the beautiful stuff about their work-- and this is where-- I can't wait for that paper to publish so I can use that slide all the time-- the single peak thing, it's not all diversity all the time. You've got to have the right amount diversity, the right type of persons. And that's why when you sit in that meeting, and you say here's why everyone's here, if you can't think of a reason why the person is here, maybe they shouldn't be there. I think Ray Dalio, who's Ray Dalio, just leave it at that-- he's always asking who should be in the room, who should I have here. I think that's the big question.

ANITA WILLIAMS WOOLLEY: The combination, yeah, absolutely. If there was one skill you were going to screen for, I think there's our data, there's a lot of other data that really points to this these social skills, the collaboration skills. There's been some economic analyzes showing that all the stuff that doesn't require collaboration is going to the robots. And what's going to remain are the jobs that require these collaboration skills. So if there was one sort of universal thing you might look for, it would be that. But otherwise, you're thinking about combinations.

MARGARET FRANKLIN: So when we think about the endurance of you get a successful group of people or a process, it sounds like the endurance or longevity of that actually is a bit fragile, because it is person dependent, whether it's the leader or your group compositions. It's not just a process that you can kind of pick up and run to the next group. Or do you think it is more enduring?

ANITA WILLIAMS WOOLLEY: You're saying if you have, say, some sort of a process that you're using, you can generalize it from team to team, and it should--

MARGARET FRANKLIN: I'm actually thinking about if you took the principles that you have, and you kind of said, OK, we're going to put them into work, we're going to do it for investment teams. We're going to really put some conscious effort into it. Have you taken a look at the difference of that actual philosophy and process in team construction? Is it something that you have people coming and going, but that thinking endures? Or is it too new to even know?

SCOTT PAGE: I think there's processes within firms that have worked that do two things that they have endured. One is just recognizing that you don't need diversity an easy problems. The checklist thing is fascinating. I've done a couple of things, like [INAUDIBLE] has spoken first, and I've spoken second. And then they sort of say, OK, you decide when to listen to each one of them. Hospital emergency rooms are a great example. If you come in with a broken ankle, they know what to do, and there's a checklist. So they hand you to a resident at Mass General, and they go down the list. And then somebody looks and says, yep, she taped your ankle fine. And you go out the door.

If you have something really strange, and they don't know what it is, then they take the tools checklist and sometimes throw it out the window. And they have you go see seven different people, and everybody tries to make sense of what's going on. And then they have a conference, and they make a decision. It's complex. So that sort of procedure, once it's put in place, and it is in place at most emergency rooms, it's sustaining, and it's leveraging diversity. There's other particular things, like within Ford, [INAUDIBLE] instituted this thing where when somebody comes in and says how's this project going, every single action of that project gets coded as green, yellow, or red.

And if it's red, it means I'm having trouble with this. And then what happens is then they sit and discuss the red things, because the whole point is put the-- when you're giving these tests, it's not an easy problems. They're not having diverse teams add five and three. These are freakishly hard problems. And that's why diversity can get give value. So I think you can put processes in play, in place, and maintain them that are sort of thinking about the way you'd allocate assets in any other setting, you're allocating your diverse talent, probably.

ANITA WILLIAMS WOOLLEY: The thing that I was thinking of too as you asked the question was so you create a framework for a group to be collectively intelligent. We have data we haven't published yet that shows that remains relatively stable over time and tends to be robust to some group membership change. And so if you're thinking of a leader who's putting a system in place, I guess take away one is actually keeping teams intact can be a huge source of value. If you put together a good team, they're going to remain good over time. And maybe some of the rapid change we see in some organizations is actually destroying value, because that team could continue to perform. But too, when that team has a structure in place that works, they can absorb a certain amount of membership change and still remain an intact team, suggesting that the processes are fairly robust.

MARGARET FRANKLIN: Releveant for our business is short term versus long term. And does the changing dimension of time alter whether you are process oriented or outcome oriented? Or maybe you're in a space where you have to deal with both, and how do you think about it?

ANITA WILLIAMS WOOLLEY: I mean, the ideal is to be balanced and actually that's where diversity could help. I would say though it's not really about the short term versus long term but the nature of the outcome that you're seeking. And so if it's, as I mentioned, something where real innovation and insight is paramount, then an outcome focus would be better. But putting it together with cognitive diversity, maybe have some spatial visualizers in there to keep you out of the red zone with overlooking things. And then you might have a chance of really capitalizing on both.

SCOTT PAGE: Outside my pay grade.

MARGARET FRANKLIN: If you think, and again, maybe not-- just the headline, the first thing that comes to mind, But let's say you've kind of got your teams properly constructed, what's the downside of diversity?

SCOTT PAGE: What type of diversity?

MARGARET FRANKLIN: You've got your right people. You've got-- it's not necessarily a pure thing. Even when you've got the teams working, is there a downside to having a diverse team? You may have talked about it just a little bit. You have to emotionally be mature to get over the conflict that occurs out of diversity.

SCOTT PAGE: You've probably look at this more.

ANITA WILLIAMS WOOLLEY: Actually, earlier, you were saying you don't want a diverse team to work on an easy problem. Diverse teams are going to be less efficient, if you will. It's going to take a little bit longer to have a ramp up time to work on a problem. And so you really want to save them for your tough problems. They're not going to be the best team to implement a known solution, where efficiency is paramount. If there's a downside it would be using them for the wrong kind of outcome. And I think sometimes that's why they get a bad rap in organizations is because we put together this diverse team because that seemed like the politically correct thing to do. But then we put them on a problem that couldn't benefit from it, and they took too long to even figure out how to work together. I think it's a mismatch of task.

SCOTT PAGE: And I think, the other thing I'd add to that is let's think about identity diversity as the diversity we're thinking about. And that correlates-- you can causally [INAUDIBLE] why it works, but like with cognitive diversity in particular ways. There seem to be differences. And my personal view on that, a lot of that's the sort of experiential, because the lives we construct, the situations are just different, [INAUDIBLE] gender. But it also leads to differences in value diversity. So among African-Americans, Asian-Americans, Latinos, I mean just there's differences in values. And so you this benefit of cognitive diversity, but you also have this cost of value diversity. People want different things. And so this gets to her ends and process. But the ends are perfectly well defined. If we all know what we're doing and values can't enter in, you're cool.

But if it's a process focused thing, and it's also-- this is like, Congress has a lot of diversity, but they all want different stuff. So that's where the bed and breakfast analogy probably works great. It's a mess. So I think you've got to think about sometimes in order to get into the identity diversity, you're going to bring in value diversity. And this relates to-- let me let me talk about-- just quickly, on this point, this is a big issue in the nonprofit sector.

There's been this big move towards being more inclusive. But oftentimes, it's like some leaders sets their vision. And then they say, OK, now we're going to be inclusive and I want these diverse people to come fulfill my mission. And that's a problem, because the thing is that when you bring people in, you have to let them decide on what the problems are as well. You've got to be inclusive towards the task, and that can, I think, be one of the big [INAUDIBLE].

MARGARET FRANKLIN: I'm a Canadian. We have a new prime minister. And he said because it's 2016, we're going to have 50% women on his cabinet. And when you actually read, when you looked through the cabinet member my member, every member on that cabinet was appropriate for the portfolio, the portfolio they had. But that I headline around the world was because it's 2016, that's why we're having 50% women. And so as we think about team construction, it's a room largely filled with women who would say, yes, 50% is a pretty good number to hit, why not? In a blunt and sort of crude manner, is really gender diversity where you start with, and then say we can back bill some other qualities, values that people bring, but we're going to just really go hard at gender and explain it better?

ANITA WILLIAMS WOOLLEY: Well, I would say in a team environment, our data suggests that the quality of the collaboration is higher. And so if it's a task that really is going to benefit from a high level of collaboration, and you don't know anything else about the people initially, stacking it with at least 50% women would suggest that you'll have a higher quality of collaboration. If you would do that at the expense of the expertise you need, we could debate. But I would say that given what we know today about where these skills tend to lie, placing your bet on having at least 50% women would be a good place to put it.

SCOTT PAGE: A cabinet position, it's not like the cabinet sits around, at least the US cabinet, sits around and jointly makes a lot of decisions. People have their own portfolios and deal with it. So I think here, it's more a case of just optics. It matters a ton. What's interesting is, being in this space, I was trained as a mathematician. And now I spend some of my time in this very weird space. As a mathematician, the fact that I'm a white guy doesn't matter, because I'm just sending out equations over the ethernet or something. In this space, the packaging I come in matters a lot. It makes me less able to do things and more able to do other things. It's a mixed bag, and that's true of everyone I think.

And so I think about something as focal as cabinet positions, absolutely. Because, I think if you're saying, oh, it's 80-20, or 20% is enough, that is just sending us an incredibly strong message to every young woman out there, and every man out there, that men are better and we have to have at least 20% women, no way. I'd actually be happy if it was more.

MARGARET FRANKLIN: I like the way you think. OK, so we're going we have just two minutes left, and I want to wrap up on this because I think it's-- again, when we talk about the how, how do you actually improve investor outcomes? How do we do better by clients by having diverse teams which start with women? You guys have both shared incredible frameworks, data, ways to think about it. So I think you've done just a spectacular job on the how. So somebody asked how often do you share your insights to groups of senior business leaders, largely male groups, and what types of industry? And I would follow on to say how do we follow you? How do we get access to what you're doing on a regular basis?

SCOTT PAGE: We both tweet all the time.

ANITA WILLIAMS WOOLLEY: I think probably, well, I've gotten an increasing level of interest in the topics that I'm studying from all kinds of industries, which is nice because I do frequently find myself one of the only women in a group largely male talking about these issues. I really enjoy putting up my graph to show them that we need at least one guy.

SCOTT PAGE: That's why I got to come.

ANITA WILLIAMS WOOLLEY: So that's fun. And so, I'm on the web, I have a website. And then also the Tepper School at Carnegie Mellon also posts a lot of information about me. And so I would love to continue to hear how things are going out there for all of you.

MARGARET FRANKLIN: And I will add one thing about Anita, when we first met in New York, she had presented her stuff. And I said, so are you consulting? She goes Marg, I'm just trying to become a tenured professor. Let me explain how this works. She's going to become a tenured professor and then she can do other stuff.

SCOTT PAGE: I don't know how to answer that. I try to-- I work for a public university and I'm limited in what I'm allowed to do and not do. So the first question I ask is am I going to learn a lot, like is this going to be something-- I'm going to learn a lot-- and then I ask is it going have a lot of impact. And then I ask would my dean and president hold my ear to a stove if I didn't do it. Once I gave a talk to 2,000 high school superintendents. They made me like basically hold up an M flag. But last week I spoke to [INAUDIBLE] Gilead, this afternoon, in an hour and a half, I'm with AB InBev. So I probably do 50 to 100 talks a year. But a lot in Michigan with foundations, a lot with Michigan businesses. a lot just on campus.

Michigan's got a big diversity initiative. We have 110 departments, and the medical school has 60 departments. I've probably spoken to 40 of them in the last three months. But that's because I can find my way to work. It's always a question of can I get in and out and not-- the nice thing is I can make my kids lunch, pack them off to school, go to metro airport and be back by 5:00, and they don't know I went anywhere.

MARGARET FRANKLIN: Wow, I would just like to clone you.

SCOTT PAGE: But it's always a question. So typically, what happens is I sit down, and my wife decides, to be honest. I'll say here's the opportunity and then she says--

MARGARET FRANKLIN: Wow, OK.

SCOTT PAGE: By the way, I was a spousal hire. So my wife was hired by the University of Michigan and I was like the husband they had to take. So she's kind of the star in the family. So she gets to--

MARGARET FRANKLIN: OK, I like that. Thank you very, very, very much. I was most excited about this, and you guys delivered.

ANNOUNCER: Copyright 2017, CFA Institute, all rights reserved. This program is designed to give accurate and authoritative information in regards to the subject matter covered. It is distributed with the understanding that CFA Institute is not engaged in rendering legal, accounting, tax, investment, or other expert advise If legal advice or other expert assistance is required, the services of a competent professional should be sought.