How to Assess the Validity of Data Sources

“I try to understand temporal patterns. I’m looking for things that are repetitive that I’ve missed. If things aren’t repetitive, I try to find out their dependencies that actually create relationships. If it’s truly ad-hoc, I want to know why are these things completely random?”

– Ray Wang

What you will learn

• What is the vital difference between push and pull channels (02:55)
• Where to find emerging data value chains (05:38)
• What is the commonwealth of self-interest (06:58)
• Why you have to get good at finding patterns (07:50)
• What are the different kinds of relationships to look for (09:29)
• What is the process from your understanding to communicating so people who think differently would also understand (11:14)
• Why you need metaphor and analogies to convey emotion, imagination and passion (14:32)
• What are three important information trends to keep your eye on (20:22)
• How to assess the validity of a data source (23:42)
• What to do when the signal doesn’t fit the model (24:33)

Episode Resources

• Post-it Notes
• SWOT Analysis
• LISTSERV
• Dion Hinchcliffe

Transcript

Ray Wang: Ross, it’s been way too long, I have not had a chance to get down to Australia. I’d love to, but thanks for having me here.

Ross: You are a prime example of someone who thrives on an enormous amount of information, making sense of, amongst many other things, the edge of enterprise tech and where everything is all going. How do you keep on top of massive information?

Ray: You know what it is? It’s constant streams of information. We are constantly looking for data points, but what we’ve gotten really good at is building ontologies, building filters, building ways of framing, and I think that has helped to be able to handle that type of information overload. Every piece of information that comes to me has a purpose, has a time, has a level of urgency. I categorize everything that way when it comes in. If I get an email and it says, Hey, let’s catch up later this weekend; I’ll probably look at that three days later from now. But if it’s something that’s an urgent deal, like, Let’s go? that’s something I’ll take on right away. If I’m reading something about a futuristic trend that’s happening, like space transportation, I’ll follow it away, saying, Hey, this is going to be related to this other research I’m looking at. It’s constant filing, constant curation, and a constantly sensitive understanding of time and urgency.

Ross: So much to dig into what you’ve said already, but part of it is the distinction between the Push and the Pull. As you get a whole bunch of stuff, which is pushed to you, and you got to set up your filters to be able to assess that. The other is the Pull; you’ll go out and find the things which haven’t landed in your inbox. When we start with the Push, how do you filter things which come into you? You’ve said, you’ve got various ways of assessing those as they come in? Is that all in your email? Or through social filters? What are your incoming channels?

Ray: It’s crazy. There are many channels, Ross. Email is still primary for me; the social networks, whether you’re on Twitter, or whether you’re on LinkedIn, are big feeder sources as well. Then it’s all the private chat groups I’m in right now. I’m in a lot of private chat groups, whether they’re on Signal, WeChat, even on private social networks that are going on. I think it’s those signals that are actually proliferating more than anything else, the public social networks are definitely dying, the private networks are actually growing, and it’s much higher quality information. That’s coming from the Push side.
The Pull side is a little bit different. I think the Pull side is if you know what you’re asking for, you know what to curate. It’s so easy to be able to set up the meetings or set up the conversations, but what you’re missing is that level of serendipity, which you used to get, when you could move around, bumped into someone at a conference, bumped into someone on the way to a conference, have that conversation after someone had spoken, that I’m missing, and that’s been a big piece for the last 18 months, that has hampered my ability to get the signals faster.

Ross: In those private channels, how do you get into the right groups? What are the things you’re finding there that you’re not finding in other places?

Ray: Honestly, I think it’s authenticity, like honest conversations, the ability to go deeper with someone about something, the level of context around a problem, understanding where an exception is occurring, I think I get a lot more fidelity out of those private conversations, or the social networks that are more aligned. They might be affinity based on places you’ve worked, they might be roles or job titles, they might be a topic that people are interested in, they might even be geographic, but I think those are playing a much bigger role than they were about five years ago.

Ross: So not so much on the public social networks anymore?

Ray: There are signals there, but it’s so noisy that sometimes it’s not worth the time.

Ross: Coming back to the Pull, how do you go out and find what’s relevant in that sea of information every day?

Ray: For me, I spend a lot of time talking about business transformation, looking at where the trends are headed, looking at technologies, and understanding policy. For that Pull, it’s really your network. The bigger your network, the more likely you can find someone within that network or at least a couple of degrees of separation. Pull for me, let’s say we were talking about a topic like Hey, what’s going to happen with space transportation, space tourism, or let’s figure out what’s happening with the future of transit like autonomous vehicles, I could probably reach out to about 40, 50 people to have that conversation with. I’d set up a meeting; I’d say, Hey, let’s catch up, what are you seeing? What’s going on? Ideally, what would happen is we’d all be part of some similar type of private networks, where someone would just ping and you get a conversation or return back, like the old Listservs.
That’s actually what it’s like in these private networks today. People just add you to a network or they add you to a group. For example, we just had a healthcare summit, we had 40 top CIOs participating, we just set up a Discord server. That’s as simple as that, and you’re done.

Ross: In this case, it’s knowing who is the right person to reach out to, and they’re the ones that’d be able to point you to the right sources, be able to make sense of that. Of course, this is based on some reciprocity. There’s some reason why they’ll agree to take your call. This is part of that building those people networks.

Ray: Commonwealth of self-interest is popping up there somewhere.

Ross: Describe what do you mean by that?

Ray: It’s in everyone’s best interest. In case you have a question, or you want to know something, or whether it’s a job hunt, or whether it’s a tip, or whether something’s coming your way, there’s enough self-interest for you to be wanting to be part of one of those groups. The value exchange is there.

Ross: Yes. People are building these networks of reciprocity.

Ray: Exactly.

Ross: Latent reciprocity in any case.

Ray: You’re paying it forward at some point.

Ross: Yes, totally. One of the things you said just at the outset was around ontologies and framing. When I build my visual frameworks, a lot of what I’m trying to do is to build some structure. How do you go about that? What’s the process for being able to either build these frameworks or how do you apply that when you’re seeing new information?

Ray: For me, it’s always been visual. I’ve got posted pads everywhere. It’s really about finding patterns. Is it groupings that are the same? Is it groupings that are different? Is it groupings that have a specific context? I just do that mentally. Then when you lay things out to see how they fit, that’s where you start applying other types of models on top; for example, what are the political, environmental, technological, and societal implications? Is there a legislative angle to it? What’s the economic impact? What’s the cost-benefit analysis? All these frameworks just start popping in over time. Let’s run a SWOT analysis. Every one of those techniques that people use gets filtered and applied until you find a pattern or you find no pattern, which is many times the case, like, Oh, there’s no pattern. We see more of that than we see patterns but we got to get good at finding patterns as well.

Ross: Do you literally just jot things down on Post-it notes, go to whiteboards and move them around?

Ray: Whiteboard, Post-it notes, sometimes it’s just like doodling, drawing entity diagrams, all those things help.

Ross: In terms of building some of these structures what are the types of relationships you might have? Hierarchical, or causal, or time relationship? What are the different types of relationships that you look for, or you tend to see?

Ray: I tend to try to understand temporal patterns. I think that’s important. What I’m really looking for are the things that are repetitive, that I’ve missed. If things aren’t repetitive, I try to find out their dependencies that actually create relationships again. If it’s truly ad-hoc, I want to know why they’re ad-hoc, why are these things completely random? That’s one line of reasoning.
Also, the other way I look at things is I try to understand an automation angle, as I’m looking at stuff. When do you fully intelligently automate something? Do you have enough capability to do that? Or are you still trying to learn? Like when do you augment a machine with a human, and find all the false positives, false negatives, try to figure out why people break rules? Why are protocols broken? Why are there exceptions? And then when can you actually augment the human with the machine to give them more insights and information, so they can actually work faster? Or when I’m in a situation where it has to be human touch, you can’t automate that, human’s got to be in charge.
That’s the other way, sometimes I look at some of these angles, to understand the impact. If it’s fully automated, that means it’s digitized, it can scale, which is going to operate very differently. If there’s a human involved in this, then we actually have to treat things a little bit differently because not everything’s being modeled, plus humans are random, they don’t follow rules, they do whatever they like. Do you get the idea? We’re not predictable. Those create another nuance that you should be looking at.

Ross: You’re building frameworks in your own mind, or your Whiteboard, or a visual framework, this is your understanding, the idea is particularly for you, but your organization is communicating that to others, and they’re trying to get some sense of this distillation, or the relationships, or the framework, which you’re creating. What’s that process of taking it from that understanding in your mind through to communicating in ways which not just somebody who thinks as you will get, but also people who may think differently would also get?

Ray: The rule that I always use is if you can explain it to a third-grader, you’re in good shape. That simplicity, the relationships, the ability to understand the causal impacts, if you can communicate it to a third-grade level, you’re usually doing fine. The trick is communicating at a third-grade level and communicating at multiple levels, that’s the art if you’re talking in multiple levels at the same time, and that requires a lot of work. You basically have to understand how your subtle intonations or anything implied carries through the context of that conversation about what it means. You really have to understand that you’re communicating at multiple levels and not creating confusion. That’s an art. That takes a lot of time to actually figure out how to do it.

Ross: You use visuals in your presentations, your books, your research, and so on. How much do you send to that on a visual framework?

Ray: One of my colleagues does that awesomely, Dion Hinchcliffe. If you remember Dion, his visuals are amazing. I’ve learned not to do visuals because I’m not at that level of trying to distill something highly complicated into a picture, but what I do try to do is simplify and break down examples so that people can relate to them, and not use industry-specific examples, use something a little bit more generic, so everybody can see how it applies to their industry. Mostly because if you get down to industry-specific examples, what typically happens is people who are experts in their industry will tell you well, that can’t possibly happen that way. That’s why I simplify things.
For example, people like to talk about, let’s take a complicated conversation about journeys and APIs and micro-services, I’ll boil it down to a peanut butter jelly sandwich. I’ll explain, hey this one software vendor’s APIs and microservices look like a simple peanut butter jelly sandwich; here’s a piece of bread, here’s another piece of bread, I spread peanut butter on one, I put jelly on the other, put them together and cut in half; but this other company’s APIs and micro-services are a little bit more complicated and a little bit deeper, you get the bread, you find out if it’s gluten-free, if it’s organically certified, you decide whether you want to toast the bread, how much do you toast it? Is that organic peanut butter? Is that non-organic peanut butter? How thick do you want to paste it on? And suddenly people get the idea. You can understand the difference between simple models and complex models, and trying to put them all together, you get that conversation.

Ross: Using metaphors essentially?

Ray: Using metaphors, using storytelling, but none of that had anything to do with a technical conversation.

Ross: To what degree do you use metaphors in terms of your own understanding? It is one of those interesting points where arguably there is a bit of a debate around first principles; thinking, coming down, and saying, Okay, you’re just pulling down things that are elemental pieces. There are dangers and values of metaphors in terms of being able to pull out analogies, and being able to find useful things; but of course, no metaphor is perfect, and they can be misleading as well. To what degree in your own understanding do you look for or find metaphors and how do you see structures to how things are working?

Ray: The thing is, you have to reason from first principles, that doesn’t go away. Everybody knows that if you’re really trying to win an argument, that’s a very logical approach but to capture the emotion of the argument, to capture the imagination, and the passion, that’s where the analogies come into play. That’s balancing the art and the science. The science is first principles, here’s what it is, here’s a logical reason, here’s how we get there, the analogy and the storytelling behind is the passion, the emotion, and getting people excited about having them relate at a human level. I think the best communicators actually do both.

Ross: Do you structure your day in terms of your information or habits? How do you interact with information?

Ray: I get about five hours of sleep, so there’s a little natural advantage here. I need about five hours to function, six hours to be at peak performance, and four, I could probably getaway for a couple of days. I start my day, and I literally scan all my input sites, whether it’s my Twitter feeds, or my news sites, I just scan information in the morning. The first thing I do in the morning is I see what the heck is going on, at least get a baseline, look at stock charts, look at what’s happening in different trend lines, whatever reports are coming out, and I leave business TV on, just to have that too in the background, just to get a grounding. I scan the day and then start that way.

Ross: Do you take any notes while you’re doing that?

Ray: No, I don’t take any notes. I’m scanning for broad trends just to get a feel. If it’s really important stuff like work deliverables that have to be done, that require time, some things you can just do in context, switch super-fast, that’s not an issue. But when you really have to take the time to deeply think, like if you’re writing a book, or if you’re trying to take a complex idea, if you’re trying to get a work deliverable out, you got to block time for yourself. For me, that’s like blocking a couple of hours to actually think, to actually get through something, to actually get work done. That’s a little bit different.
Other than that, if you’re on calls, and they’re not video calls, you can get a lot of work done. There are a lot of manual tasks, email tracking, things you can do in multi-task. I know people don’t believe in multi-tasking but it works for me. The only thing that doesn’t work in multi-task is when you’re stuck on a zoom call, you have to look at people, the video just kills you, you’re single-threaded for a while, and you’re very unproductive.

Ross: Do you have times when you’re pulling together like the synthesis process? Okay, you’re doing the scanning, you’re making sense of things; is that all just in your focus blocks? Is that when you’re distilling and making sense of things, or the other times when you are spending time to read an article or a book, or when you’re not aware of a problem?

Ray: Some problems just can’t be solved right away. They require some depth, they require a little bit of perspective, they require seeing other things happen, and they’re constantly in the background. For those types of deep thinking that you’re trying to do, or you’re trying to build a brand new model, those take months; that’s not something you would do in a day or in a week, they take months to get to that level of substance. But if you’re trying to crank out a consulting deliverable, or help a client with the project, then you can probably do it in a couple of days.

Ross: With the long-term thing, do you spend a slice of time on it every day or every little while just to be able to see the way you’re thinking about it?

Ray: In the post-COVID world, yes, because everything is scheduled, nothing is serendipitous, you have no free time to yourself. Before COVID, it would be a flight, you just get on a flight, Sydney to LA, Sydney to New York, Sydney to San Francisco. You get on the plane, you eat something, you check some emails, you watch a movie, you come back, you think about something, that was thinking time, that was personal time, travel was amazing because it gave me my own personal time, I lost all that just like you. You lose the time to actually bump into things, some random things might occur. These days, it’s all forced. I got to block three hours to go think, that’s not fun. It doesn’t work like that. But if I’m traveling, and I happen to bump into something, or I happen to be thinking of an idea, I’ve got my own time, now I can actually do something with it. I think everybody works differently but for me, that travel time, that transit time, was my thinking time.

Ross: Coming back to thinking around what’s relevant and what’s not relevant, you’re very broad-ranging in what you cover, and it seems to be getting broader over time as well. Do you set some kind of a frame for what you perceive, like, this is relevant to my consulting, to my company, to books you may be thinking about; the nature of you can’t keep across everything so doesn’t, how do you set those filters or that frame?

Ray: I think probably three things that are important. The first one is that we focus on the business impact; it has to have a business impact. Whatever I look for, I’m just trying to understand, is that a new business model? Is that technology going to change this? Is that a marketplace move? Is that something that will change the balance of power in an area? I always look at the business impact. The second thing that I look for is, is that a new technology trend? There are Uber trends, macro-trends, micro-trends, all the way down. You’re trying to gauge if a technology will change what happens in the marketplace. The third one’s a little bit different. I spend a lot of time looking at capital flows, and trying to understand where the money is being bet, where are you betting? Why I look at capital flows is because it’s a good indication of not only psychology but also people’s perception of where markets are heading. Those three things pretty much drive everything, so business, technology, and capital flows.

Ross: Right, so anything which relates to that. Do you have any focus industries? Or do you tend to think everything fits in the world of business, and it all is relevant?

Ray: I used to look at industries, but the challenge of looking at industries today, as you’ll probably know, in the way that I’ve written recently, it’s about these data value chains that are converging. For example, communications, media, entertainment, and telecom to me are really the same industry, whether you’re selling a video game, or enterprise software, or a movie, or music, or a phone plan, it’s all the same to me. I don’t say that lightly, because it’s the interaction between the content, which is those media types, the distribution channels that are actually happening, what’s happening from the technology platforms, and then, of course, the customer networks, those interactions allowing this digital monetization to occur, whether it’s ads, or goods, or subscriptions, or memberships, or search, they’re all working in the same way.
We see data value chains emerging, so that’s why I don’t really look at industries in the traditional sense. Retail, manufacturing, distribution, we see that convergence already. Hospitality, healthcare, and retail are converging. You see those kinds of things happen. I start to look at where data value chains are playing a role, just like you look at capital flows, I want to understand who takes the upstream, and the downstream implications of that data, and what do those insights create, and how do those insights create new monetization models or risk mitigation models.

Ross: Yes, absolutely. It’s a long time since we’re seeing the convergence of industries, but in a way that provides a frame for the information; if you’re looking for these data flows, and particularly the ones which relate across industries, that’s where that is a signal that it is relevant and interesting.

Ray: You do this all the time, I do this all the time, we’re trying to extract signal from the noise, and that’s been the challenge. The world has gotten noisy, you just got to get better at figuring out which data sources are more valid and which data sources are more trusted? There’s a little bit of having to do that as well.

Ross: Tell me about that. How do you assess the validity of data sources?

Ray: It’s completely temporal. Over time, what’s the track record of the source? What’s the track record of that information and insight? Has it been tampered? Has it been altered since you’ve last seen it? We’re dealing in a world of deep fakes; we’ve got to be very careful.

Ross: One of the interesting things about signals is what’s surprising. What do you do when you see something that surprises you? What’s actually you’re going to say, Oh, that doesn’t fit model? This is something that is a counterexample. How do you deal with that?

Ray: There’s the logical aspect of it, and then there’s the reality. I’m going to take a very controversial example, just to make a point; it might get me in trouble, but let’s take the issue of police deaths of unarmed individuals. In the US at one point in time, there might have been 42 of those in the year of 2019, maybe 2020; and of those, five of them were African American, 35 of them were Hispanic or White. Who would have thought that one incident like that would trigger a massive set of riots, and chaos in the United States throughout the summer, with massive amounts of property damage, massive amounts of social unrest, all happening at once? That doesn’t make any logical sense.
If you were just to look at the numbers, it is like, oh, people should be more upset about violence in cities where thousands of people are being killed, thousands of people are not getting their justice, and people don’t feel safe, they should be angry about that, that’s the logical conclusion. But then the emotional piece kicks in, one video can change the way people view things, use that as the analogy versus your first principles argument.
What I’ve learned over the last 24 months is that humans are not very good at making logical rational decisions in their heads. They don’t understand how to assess a proportional response, the level of risk, and probabilities. They let analogies, emotions, and imagery take over their logical minds. If you understand that, then you understand the art of these trends that are happening because the data is going to say one thing, like, who would care about 40 people who are unarmed and shot versus tens of thousands of people in the cities being killed every day. That would be more important in an issue. I think that’s an important lesson to be learned.
When we’re talking about COVID-19, one would say this is crazy. If you look at the numbers, and if you look at what’s going on, you would think that the rational response would be to give every country the manufacturing capacity to have vaccines and get that done all at once, so that there wouldn’t be variants. Because the numbers say that the variants are operating at a thousand times the vaccine production, you’d want to solve that as an emotional issue; but here we are, 18 months later, and we’ve got countries like the US, where we’re wasting vaccines and talking about third boosters, and the other countries, which are barely getting their first set of vaccines. That’s not logical. It doesn’t make any sense. You see these things happen. You’re just like, Okay, I guess humans aren’t very logical.

Ross: It continues to be a surprise, I suppose.

Ray: It’s not going to surprise, but you get it, right? You’re just like, how can we be this? Would you trust the machine to then make that decision is the next logical conversation point. The answer is like, I don’t know, we still want a human in charge.

Ross: It is also around then being able to say, Okay, what are the signals? Part of it is saying, Okay, you can’t analyze this rationally, you can’t have a logical structure here to this, but what are the senses or the structures in being able to assess particularly those group, or social, or emotional responses, which are still very much shaping our world?

Ray: This goes back to your point. This is really about framing. Our ability to actually frame these conversations is important. Give people the context so that they can get to the right decisions.

Ross: In summary, for anyone who is looking to thrive on massive amounts of information we live in today, what’s the advice you would give?

Ray: I think what you have to figure out is the purpose of why you want to even get there. What are you trying to do with that type of information? There is a lot of noise. For example, I don’t spend a lot of time looking at gaming, I love it, it’s a fun spot, but that’s not for me. Gaming and e-sports, that’s not an area I typically go deep and cover. You have to figure out why you’re actually looking at that information and those information sets. Then you also have to figure out what information is useful to you. It’s easy to get overloaded. It’s massively easy to get overload if you’re not processing very rapidly, or if the data is not in a way that allows you to minimize context switching.

Ross: Fantastic. Thank you so much for your time. It has been some really deep insights into what you do. You’re obviously right on the edge of making sense of lots of information. Thank you so much, and have a wonderful day.

Ray: Thanks a lot, Ross. Thanks for having me here.