I believe democratising content curation is essential, if you want to ensure a learning culture that is going to stay, that's going to thrive. - Protima Talapatra
Protima Talapatra is the Head of Learning Delivery Transformation at Ericsson, and oversees L&D for a large, globally distributed employee population. Ericsson recognises the importance of empowering their learners to curate content, which they were able to achieve with an LXP. However, they quickly recognised that this practice can lead to overloaded LXPs.
One of the biggest problems they wanted to solve was cleaning content. But to do this manually could never keep pace with the rate new content was entering the system.
When Ericsson approached Filtered, they had two objectives:
- Enhancing the discoverability of their learning content in Degreed
- Tagging content to a set of critical skills which they had already defined, beginning with three: security, business storytelling and design thinking
The solution was getting human experts and algorithms to work together to achieve accurate skill tagging at scale. Here’s how it worked:
Stage v0: Casting a wide net
At first, a report was generated from all the content available within Ericsson’s libraries.
Of the 140,000 + courses that were within their content libraries, there were 1904 learning assets that were identified as being well-aligned to the critical skill sets.
Ericsson then brought these assets to their Subject Matter Experts (SMEs) to validate the algorithmic tags. The results were as follows:
- Security - 66%
- Business storytelling - 72%
- Design thinking - 40%
With this data in hand, they wanted to understand the disparity in accuracy. They asked their SMEs to look at the assets that were least relevant.
Some notable examples were:
- a patch management course aligned to security
- a book on data visualisation for business storytelling
- an agile meets design thinking course
Stage v1: Iteratively configuring skills for better accuracy
With this new input, the skills tagging in Content Intelligence was reconfigured and the net was cast again. This time, there were 1067 assets returned (76% fewer assets than in stage v0), which were shared with the SMEs to validate again.
The learning content accuracy results were noticeably higher:
- Security - 80% from 66% (14% increase)
- Business storytelling - 90% from 72% (18% increase)
- Design thinking - 67% from 40% (27% increase)
There was still some irrelevant content, but with this new AI-assisted workflow, Ericsson is able to iterate the process to achieve even higher accuracy rates.
Skills level tagging
The next step was to tag skill levels, a process which requires more nuance. As with stage v0 and v1, the results were shared with the SMEs on the accuracy of the skills levels.
Between themselves, SMEs had an agreement rate of 50% on what level of skills an asset was. The results from Content Intelligence were remarkably close, if not better. Validated by SMEs, security skill levels tagging accuracy was at 52%, and for business storytelling and design thinking, it was at 35% and 63%, respectively.
Key takeaways
With Content Intelligence, Ericsson was able to retrieve a highly relevant set of content assets aligned with their critical skills swiftly and iteratively, with humans feeding back on the tagging to achieve an accuracy increase of up to 27% between stages.
We learned that in terms of tagging, learning assets to skill, the Content Intelligence tool within Filtered does a marvellous job, it has almost human-to-human accuracy there. - Protima Talapatra, Ericsson
Tagging skills levels is a task which is difficult even for SMEs to agree upon. With Content Intelligence, they were able to garner results which were comparable to human SME accuracy, and at a level that would greatly accelerate curation across the skills framework for their team.
If you'd like to understand how to reach the same results as Ericsson, schedule a 30-minute Content Clinic with our internal experts to get a fresh perspective on your learning content strategy.
Watch Protima’s full presentation below, along with the transcript.
Kat Cornwell: We will pivot over to my friend Protima. David gave us a very high level view on content and skills and now we're going to see it in application. When you do try to boil the ocean, so to speak, what pieces of technology are out there to help you be really efficient and effective with content curation? Because usually you key in leadership and you're gonna get multiple, 1000s of pieces of assets coming back at you. So trying to figure out as a learner, what is the right one and as a curator, what is the most applicable piece for that pathway? Right.
So, Protima has got a really great case study to walk us through with Ericsson using Filtered and Degreed, she is the Head of Learning and Delivery Transformation. She is based in India. And like most of us in L&D, she had a very winding path here, she started out as a math and physics teacher before she pivoted into learning design and instructional design, which is always interesting. She's got a love for romantic movies. And I think kind of the connective tissue in her background is this data piece, she is definitely a Trekkie. So if you love Star Trek, throw some questions in. I asked her who her favourite Star Trek character is, and it's Spock, which is probably not a shock - she's into data. However, she does have a lot of emotions and feelings which Spock really does not. So without further ado, Protima, let's, let's throw it over to you.
Protima Talapatra: Thank you so much, Kathryn, thank you for that introduction. And I'm so happy to be here with everybody, super excited about this. And once again, if I can share my screen, let me know. And if at any point of time, my slides refuse to move, do let me know that Kathryn and maybe I'll try and fix that.
I hope you can see my screen now and see the slides. First of all, I believe what David just took us through holds such a great connection. And I was just going through what I have to talk about and what David just said, when he talked about looking at the content differently, looking at what is there, focusing on the skills and empowering your employees and empowering the people to look at content and work through content. But how do you do that? When you have a large employee population, which is globally distributed - not that easy to do, but I believe that's the next problem that we’ll talk about. Let me quickly run you through some of the key points that I wanted to talk about. I hope you all can see my slides.
I believe democratising content curation is essential, if you want to ensure a learning culture that is going to stay, that's going to thrive. It's very, very important that we democratise learning content curation because David just talked about the influencer culture. Everybody wants to know and understand that you've arrived and you know what you do. You put in so much time and effort and why not? Why shouldn't it be so that others don't get the benefit of your knowledge?
That's where learning experience platforms come in. They enable content curation, they enable the democratisation of content curation. A flip side to it is the fact that democratising content curation leads to overloaded LXPs.
Most importantly, content cleaning, as I saw somebody put in the comments, it just refuses to end. There is no way that you can clean up content as fast as people curate content. So that's the biggest dilemma. And I think that's the problem that we need to solve. And that's the problem that we're facing when we went out to Filtered and we said, we need a solution from you. And can you help us here?
And we really had two questions to ask them. So we said, how can you support us in ensuring that when people search on Degreed they get the right options - to enhance the discoverability of the content? And how can you tag the content to our critical skills? At Ericsson, we have a defined set of critical skills that will take us towards 2025 full steam. We said can you look at our critical skills and can you tag the content that we have in our content libraries, to these critical skills, and if somebody searches for content on Degreed for this group.
How do you ensure that the right content pops up? That's a challenge that we, at Ericsson, took up. And we had the support from Filtered to help us take that forward. What did we begin by doing? And how did Filtered work that through? When we started off with this proof of concept, we worked with three key skill areas. So we talked about security, we said, let's talk through security. Let's talk through business storytelling and design thinking. So those were the three key skill focus areas. And we obviously had definitions for each of these skill areas. When Filtered started work with it, what they did was, they just cast a wide net. And they said, let's pull out all the content that is tagged to these three critical skills. Let's see what there is in the market, what is available. And out of the 140,000+ courses that were within our content libraries, aligned to these three critical skill areas, we got about 90-100+ learning assets that Filtered said were well aligned to the critical skill sets that we were talking about as comp, software incompetence, security, business, storytelling and design thinking.
This information that we received as a first-time input from Filtered, we took it to our subject matter experts. And we said, this is what Filtered says, do you agree with it? The responses were good from the SMEs and for example, for security, 66% of learning assets were aligned to security by Filtered. For design thinking, it was a little bleak, about 40%. For business storytelling, we had about 72% match according to the SMEs to what Filtered's Content Intelligence was saying that matches.
We went through the content and we said let's look at where the disparity lies. With this assumption, we asked the subject matter experts. So what do you think, which are the courses that really don't make sense to be part of this? There were courses, which were, let's say, a patch management course aligned to security, from Pluralsight. There was a course around data visualisation for business storytelling and, at least for Ericsson, it did not make sense. There was a post on agile meets design thinking from Coursera on design thinking which our subject matter experts felt did not make sense.
This was the top, this was the input that Filtered's Content Intelligence received. And with this, they did the v1. What they essentially did was, they reconfigured, the skills tagging, and they tried to cast a net again, but now with the reconfigured, skill tagging, they said "Ok, we need to ensure that these things don't fall into this." And then once again, whatever inputs that we received, those were shared back with our SMEs.
This time round, we got about 1000+ courses, almost half, 1067 courses and we saw the accuracy has really improved. For security, we we were at about 80%. For business storytelling now we were at 90%. For design thinking we were somewhere around 67%, there was almost an increase of 20-23%. In terms of accuracy, from where we started off at version zero. I believe, as we keep moving forward with this, it is only going to grow and we will also move forward to expand on its accuracy further.
Now, when we looked at out of 1000s of courses, we still had some courses which the SMEs said that they don't really match. So what are these courses? This time around, we felt that, maybe they were courses like hardware security for business storytelling, courses like elevator pitches and presentations, which for Ericsson did not make sense or for our target audience does not make sense. For designing design thinking we had courses like design thinking for product management team, again, possibly at that point in time with the focus that the subject matter experts were looking at did not really make sense. So this is what we got as a residual set of tags.
In the next interesting bit, now that we had already tagged the courses to the skills, we asked Filtered: why don't you try the skill levels. Now, this is more of a difficult game. If you look at the slide out here and if you look at the last yellow-coloured bar, you will see that between the subject matter experts themselves, the SME to SME ratio of agreement lies at about 50%. So that's where it stands. So we saw that security, we were at about 52%. For business storytelling, the accuracy was about 35%. And for design thinking, we were at an accuracy of 63%. So that is what we achieved in terms of skill level..
What have we really learned from the entire exercise? We learned that in terms of tagging, learning assets to skill, the Content Intelligence tool within Filtered does a marvellous job, it has almost human-to-human accuracy there. In terms of tagging, I believe, levels, it's a more difficult task. Even at the SMEs level, the accuracy is not that good. But I believe you know, we have great scope to move forward. And I think we are at the right juncture and we'll be able to move forward with that. I think it's a great start. That's what I can say.
Very happy to answer any questions that you have. Kat mentioned to me I have 7 minutes, I don't want to overshoot my 7 minutes so I couldn't look at the chat and at the questions that were coming in, but very happy to address any questions that might be there. Thank you so much for your time.
Kat Cornwell: I don't think you're over, I think you're you're right on track and lots of emojis in the chat really lit up. Well, what I love about this Protima is that your proof of concept really did start to prove out that the AI technology is pretty on target with your human component of SMEs and being able to tag and curate the content. So with that extra time, if you let the technology do that lifting for you, where else can you start to add value in the organisation, what else is on your to do list as a learning professional that you want to get to that you just can't get to because you're so stuck in the weeds with this work, right? So that's pretty exciting for me.
About Protima Talapatra
Head of Learning Delivery Transformation, Ericsson
Protima Talapatra is a Learning Professional with 20+ years of experience in the learning domain, working across sectors like telecom (Ericsson), IT (IBM), core production space (Tata Steel), and hospitality (Taj Hotels). Her expertise lies in developing & deploying solutions and designing the architecture for their implementation. She is the Head of Learning Delivery Transformation for Global L&D at Ericsson.
Her role requires her to engage stakeholders, participants, and senior leaders to create more efficient and automated solutions that improve the employee experience of learning delivery. Other roles she inhabits include that of a mother, an amateur cook, and the proud owner of a green thumb. She spends most of her time talking about the possibilities of 5G to her children, her plants, and any (somewhat) willing listener. The rest is spent avoiding Black Mirror episodes that her daughters choose to watch instead of just listening to her.
