Episode 14: Sarah DeAtley, Microsoft - Data Science and Content Analytics

Kristina: Hello again. Welcome to The Content Strategy Podcast. I’m your host, Kristina Halvorson. This podcast is brought to you by ContentStrategy.com and Brain Traffic, a content strategy consultancy. Find out more about Brain Traffic at BrainTraffic.com.

---

Hello, and thanks once again for joining me on The Content Strategy Podcast; very happy to have you here. I’m also happy to have here with me Sarah DeAtley, who is joining us from lovely Seattle. Sarah is a customer experience analytics expert, driving strategies for major corporations. Right now, she’s currently focused on marketing optimization across content and media at Microsoft. Previously, she’s driven content analytics and optimization at Dell, Apollo Education Group, ooh, Egencia? Is that right?

Sarah: Yeah, that’s correct.

Kristina: Egencia, which is actually Expedia, is this right? And CDK Global …

Sarah: Yeah.

Kristina: … Which is actually ADP. Are those the fronts for those organizations?

Sarah: Well, for Egencia, it’s one of the brands within Expedia group for the B to B corporate travel …

Kristina: I see.

Sarah: And then CDK Global was acquired by ADP.

Kristina: This is one of the things I love about this podcast is that I’m always learning. Always learning. The last podcast interview that I did, I found out that Zagat and Frommers are both owned by Google. Did you know that?

Sarah: I did not, no.

Kristina: Yeah, see? Google, everywhere. You didn’t know that. Now you’ve come on to the podcast and you’ve learned something. It’s mutually beneficial all the way around.

Sarah, thank you so much for being here today. The reason that I have brought you here today is that I am super excited to talk to you about your work in analytics, and that you came to analytics because of your excitement about and your background in data science. Is that right?

Sarah: Kind of the opposite. It’s more …

Kristina: Dammit.

Sarah: I have ended up in the more data science side of thing just because that’s where analytics has gone, but I have …

Kristina: Start at the beginning for me.

Sarah: Yeah, I was ... It’s very random and pretty unique and I think it’s why I’ve always ended up looking at content analytics because I was an English major in undergrad and I didn’t want to be a lawyer or a teacher. So I thought, “Oh, well, be a librarian. That could be cool to help people research things.”

So then I moved to Ireland to study Irish history so I could specialize in academic librarianship and then I came back to Seattle to University of Washington to focus on library science and after one quarter I realized this is not for me. I need something a little faster paced. Fortunately the program I was in was really broad and so they let me take classes in user experience design, user-centered research, database design, and interaction design and …

Kristina: Within a library science program, really?

Sarah: Yeah. It was more they called it information science …

Kristina: I see.

Sarah: … But it was a big enough school that they were like, “Oh, sure just don’t drop out, take other classes and stay with us.” I was working at a museum doing web development and design as a side job and they said, “Oh, can you take a look at our Google Analytics account?” I said, “Sure, I don’t know what that is but okay.” I started to get into analytics because I thought, oh, well, this is a really great way to prove if people are having a good or bad experience versus just doing usability test or something.

Then in my first job out of grad school I was applying for a UX design job and they said, “No, you should probably go into analytics. You could go really far in that.” I was like, “I don’t know what that is, but fine, I’ll keep doing this thing.” I kind of stumbled into analytics but I think because I had this weird hybrid kind of social sciences background plus knowing analytics tools, I’ve always been asked to look at the things that no one knows how to measure like content analytics. Since I’ve been doing it for 10 years now, I realized over time the industry was shifting more toward data science, so I went back to school to get more educated in data science and start incorporating that into what I do. In another life I could’ve been the librarian living in Ireland, but instead I’m doing content analytics.

Kristina: How do you feel about that? I’m sitting here listening to you. I mean, two things. One, I just want to say that every single person I have on the podcast at some point says something like, “It was a weird and winding road.” Or, “I have such a weird hybrid background.” It’s part of why I love working in content strategy because people just have the craziest paths to doing what they’re doing now. Secondly, I’m feeling like a total idiot going starting with like, “You started in data science.” In my own defense, I think that’s where we landed before we started the interview and I ... Before we started recording and my brain just ... It’s just before the holidays for those of us now joining.

Sarah: Yeah, no worries.

Kristina: Yeah, exactly, so and this will go on after the holidays when my brain is cleared and I’ll just be like, “Oh, what was I thinking?” Anyway, so you got an extraordinary background. I’m really interested now to hear about how this is all coming to bear in your work that you’re doing or that you have been doing over the last several months at Microsoft. Can you talk about that just a little bit?

Sarah: Yeah, sure. I think what I’m doing now is a bit broader, so I’ll touch briefly on what I did first at Microsoft, and then what I’m doing now. What I’m doing now is helping to ... Basically we’ve built a giant data set that incorporates all of our different marketing silos’ data. We finally found a way to connect everything and it’s a big deal because Microsoft has so much data. Now it’s my job to help look at that data and build attribution models to understand the impact of different media touchpoints and then also to start to look at content that no one has really looked at before because we didn’t have the data to do so or we couldn’t compare it as directly to other data points. My job is really focused on big data and analysis and trying to make sure we do something with the data we have.

When I first started at Microsoft, I was helping a team that was building a ton of B2B content and that was really their focus. They had no reporting, no measurement, and definitely no data science. It was my job to come in and kind of help them see out of all the content they had what was worth keeping or killing, how should they measure this, and then how do we scale these insights across many teams and many countries?

Kristina: I guarantee that a large portion of the audience that’s listening right now is just salivating at all of these things you’re talking about. Like, I’m sorry, bringing meaning to big data and analytics? What? Yeah.

Can you talk about just like on a very small scale, for example, where you would start if you ... If a marketing team ... Let’s say one of these siloed marketing teams came to you and they were just like, “Look, we got all this data and we got all this content and we don’t know what to make of any of it.” Where would you start with them? How do you begin the conversation?

Sarah: Yeah. I think one of the things that makes the analytics more impactful, and luckily some of the teams I’ve worked with have already had this, is having a really great sense of what your content is for and how much of it you have, because if I’m going to come in and say, “Oh, here’s the best KPI to measure this content,” really that’s going to vary depending on who the content is for and what it’s trying to do and what we hope to get out of that.

Otherwise, I basically won’t ... I’ll just be guessing and I’ll have to backtrack and say, “Well, my thought was is that you want this type of article to help people select a product. Is that what you want it to be for?” It really doesn’t require the marketing team to know everything about analytics, but it’s better if they do have a strong opinion on here’s what we want content to do and not do, because then it helps me come in and say like, “Let’s now apply KPIs to those goals and then we can start the process of a really comprehensive content audit.”

Kristina: What are some sample goals?

Sarah: Yeah. I feel like I’ve seen pretty good goals where it’s very granular and focused, like, “We think this piece of content is meant for people that are just learning about a product. They may have heard of our company, they may not have. We want them to basically know enough about us that they’re willing to fill out a form so that we can continue the discussion with them.” Essentially the more you can tie a goal to a hope people will either know more about us or interact with us more or complete a sale, that’s really helpful, but also the more I know about who is this content for helps, because if it’s for a certain part of the life cycle or if they’re in segments of people, like this is for executives or this is for IT professionals or something, then it helps me know in the data whether your content is getting to the right people or not and if it’s doing what we want it to do.

Kristina: What are some examples of some bad goals? If people come to you and they say, “We’re interested in these things”, and you’re just like, “That doesn’t matter”?

Sarah: Right. Yeah, I see that a lot because a pretty easy trap to fall into is that the goal of content is just coverage. Especially when people are building up a content program and they’re just checking off boxes like, “Oh, well, we don’t have this eBook localized in France yet. We need to get one in France.” Or, “We don’t have this video for this product yet. We need a video for this product.” That’s kind of their goal in that sense is just having something there for someone without really knowing if there’s a demand for it. That’s one example where it’s a bad goal from a business perspective. It’s an easily achievable goal, but it’s really difficult to tie to business success down the road because then you’ve just created a bunch of content without knowing what it’s supposed to accomplish.

The other thing I’ll see sometimes is one-size-fits-all goals. Like, “Oh, this content is meant to help with decision-making.” Okay, but what is that decision? Hopefully all content helps somebody make some kind of decision or understand something new. You’ll find this content that’s trying to do too many things at once and you’re like, well there’s no way this content will be all things to all people. You need to either decide what it’s for or break it apart into something more specific so that it can do something instead of just failing at everything.

Kristina: That’s a common complaint of content strategists everywhere. Let’s try to get this content to do something instead of just failing at everything. That’s our baseline goal here as content strategists.

What’ll happen to me often times is I’ll walk into a room of let’s say content marketers, and they will kind of be tearing their hair out saying, “Hey, we have all this data. We’re not sure, though, if we are measuring the right things. We’re not sure if what we’re measuring is meaningful.” What I’ll find is that ... I mean, what you say makes sense, right? They need to understand what the business purpose is or they need to understand what they’re trying to help people do and who it’s for. But a lot of times they don’t have that information to begin with. I mean, do you find that in some teams? That like people are just sort of operating with, “Well, we’ve been tasked to create content and that’s what we’re doing and leadership wants clicks.” I mean, do you see that in teams? ... Not necessarily Microsoft, but just in general?

Sarah: Yeah, I mean, definitely I’ve seen it everywhere I’ve worked at when I’ve worked on content, just because there’s a disconnect I feel between the different ways they’re always told to measure content like, “See how far people scroll,” or, “See how many people click on that video,” or “See how long they watch the video,” things like that. The actual bottom line business metrics ... Even if they’re a content team that has a really clear idea of, “Well, this content should be helping with people buying this specific product,” and then the only ways that they’re told to measure it are like, “How many people got to this page?” or something. Well, how do I know if that was enough? How do I know that those page views or that engagement had any connection to these bottom line business metrics?

I think it’s a common problem, just because it’s super easy to measure all those engagement things and then it’s really easy to have no idea what they mean.

Kristina: What I usually will point to is you’ve got to have some kind of qualitative research to kind of tie into that in order to make ... The last team that I worked with kept talking about the narrative. “We need more of a narrative. We need to be able to tell a story about what content’s working and what’s not so we can bring it back to our partners in the organization to talk about what content we should be creating and what content is worthless.” I sort of was saying, “I think you need to actually talk to some end users about what’s meaningful to them.” I mean, is that part of what you are bringing to the table? Or do you find that there are other ways to collect more quantitative data that can help talent shape that story?

Sarah: Yeah, I mean, I feel like they’re two sides of the same coin, so I’ve definitely ... Like for example, I’ve had cases where we’ve seen ... like we’ve implemented a new content module, some third party that said, “Oh, we’re going to provide these recommendations of content and people will just naturally consume more content and that’ll be great.” And then we saw that no one was engaging with this module even though we thought it was in a pretty prominent part of the page.

When there’s an absence of activity, there’s not really a way to add in more quantitative data because all it’s going to show is just that no one’s clicking on it. It’s not going to tell you the why. In those cases where we basically don’t have enough data, you can incorporate qualitative things like surveys or in those cases sometimes I’ll do user testing so I can understand, what are the first things they’re noticing on the page? Why are they not noticing this? What did they think the purpose of this was? They’re not going to get that from any type of behavioral data.

Kristina: Right. When you ... Do you ever get the opportunity to sort of design a measurement program or analytics program from scratch with the marketing team? Or is it more that you are often times are walking in and they’re like, “We have this thing and we need you to measure it and what should we do?”

Sarah: I would say most frequently I come in and I’m designing the measurement strategy from scratch because there might be a measurement strategy but not for content specifically. It could be just for a certain business group overall or a certain website or something. Other times I ... If I do actually come in and there’s something there, it’s usually not what I need it to be, so I’ll come in and kind of blow it up and then figure out something different where it is so we can actually act on and tie it to some business KPIs.

Kristina: Tell me a little bit about the work that you’ve done, because you’re worked in a lot of different places and obviously you have seen analytics programs and capabilities shift and change radically over even just the last few years. Tell me something that you’ve worked on recently that you were super excited about, that you were just like, “I am blowing my own mind right now.”

Sarah: I guess one of the things I was able to do from ... With one of the content teams I was working with that I just don’t even think I had seen it before, or at least I had never thought to do it before, was essentially we had various teams that had all been in this kind of building up content program phase where they were generating more content, creating lots of emails, trying to set up marketing automation programs. They were like, “We just need to get the content there and we’ll set up these email programs and essentially people will always read these emails, always open them and click on them, and it’ll be great.”

What we found was that essentially the top 10% of content was driving 90% of engagement, and that in itself is not uncommon. There’s always going to be a small percentage of content driving a lot of the engagement, but we took it a step further, especially with email to say what is that sweet spot in terms of how much content should we be creating? How much email should we be delivering before we start to hit essentially the efficient frontier of that content and people are less and less likely to engage with it?

We did some modeling to understand as people were consuming more content and getting additional emails, we could see where their likelihood to engage was dropping off a cliff. We could see with one group, for example, that they had this program where they would send an email every week. It felt like for all time, but it was a cut-off at 30 emails or something. After the eighth, ninth, and tenth emails, engagement dropped off. The likelihood of engaging never got any higher, and so we could kind of see essentially if you haven’t started to engage people right away, it’s going to be harder and harder to get them to consume even up to that tenth email, let alone these 30 emails. It gave the team kind of a number they could work back because then they could say, “Oh, well, what if we shifted these emails to slots one, two, and three and these to four, five, and six?” Trying to make sure they weren’t wasting effort.

That was something where I’ve seen that applied in media analysis at times, but I’ve never seen it applied to content and how much content should we really have.

Kristina: That’s amazing. That must have blown their minds, too, because that’s a huge question that I hear, which is, “We’re creating all this content. Is this too much? Do we need more? Or how can we tell?” Being able to like identify those patterns and really be able to point and say, “This is when it stops mattering,” is pretty amazing.

Sarah: Yeah, it’s helpful for them just to have an actual target based in data, and then also they could then go to all their stakeholders and be like, “Well, here’s why we’re cutting out content and we actually know how much we should be cutting out.” It just made all the other content teams feel like, “Well, why don’t we have that? How can we scale this?” Made me popular for a while.

Kristina: Do you find that people bring you in when they are looking to make a business case for more content or less content? I mean, is that a pretty common reason that you get called?

Sarah: Yeah, I’d say generally with just being in the analytics, you’re called in to help with ... I would say most frequently it’s the time when things are not doing as well or they’re more mature, and so people are like, “Okay, well, we need to start growing up and measuring what we’re doing and be smart about it.” We’re usually not called in in the very beginning when you’re just coming up with something. I would say more frequently we’re called in to help get rid of content than to help generate more, though I have been asked like, “Oh, can’t you build a model that tells me exactly what type of content to create?” “Well, maybe not exactly,” but there are things now where ... I’ve seen this from Adobe and some other companies, too, where they’re able to use natural language processing to basically break down all the components of your content to see how much engagement it drives.

They can compare it to something new you’ve created to see how similar those are and say, “Well, now this content has a high chance of being very engaging. Or maybe this content has a very low chance because you’ve created content that’s similar to this underperforming content.” I think it’s getting to that point where people are thinking more about, “Well, can we use the data we have to know what type of content exactly to create,” versus using the data to start hacking away at content.

Kristina: Well, that’s exactly what marketers everywhere want to hear. That’s the best 2019 news you could have possible delivered.

Sarah, we’re about out of time, so I wanted to thank you so much for taking the time to chat with me today. If people are looking for you online, where can they find you?

Sarah: Yeah, so it’s probably easiest to connect with me on LinkedIn if people have questions. Yeah, I’d say that’s probably the easiest.

Kristina: Great. All right. Well, thank you so much.

Sarah: Yeah, you’re welcome. Thanks for having me.

---

Kristina: You’ve been listening to The Content Strategy Podcast. I’m your host, Kristina Halvorson. This podcast is brought to you by ContentStrategy.com and Brain Traffic, a content strategy consultancy. Find out more about Brain Traffic at, of course, BrainTraffic.com. Thanks, and we’ll see you next time.