'If you must play, decide on three things at the start: the rules of the game, the stakes and when it’s time to quit.’
- Chinese proverb
When it comes to implementing new learning technologies, it’s become increasingly difficult for organizations to understand the rules of the game, the stakes, or when it’s time to quit. CFOs are placing L&D under greater scrutiny to ensure that any investment will deliver value, making long-term contracts with vendors harder to justify. On the other hand, urgent business imperatives and stakeholder pressure can force L&D teams to roll out enterprise solutions without spending time interrogating their potential impact, leaving organizations saddled with solutions that just don’t work.
All of this means that L&D teams need to adopt more agile, data-driven approaches and work backwards from business results. When looking to implement new learning software, conducting Proof of Concepts (POCs) is a great way of doing this.
At Filtered, we’ve identified four key steps to run a successful POC:
Running short, focussed POCs is beneficial for both organizations and vendors. Both parties gain an understanding of whether the solution is the right fit, get an opportunity to work with each other to see whether their cultures align, and can gather useful data to inform the next steps. Equally, if you don’t gather any evidence that a solution will deliver value, you can walk away before the sunk cost effect impels you to carry on regardless.
Emily Ricco, Learning and Development Manager at HubSpot, has written about how to buy business software, which you can read about here, so we spoke to her about the dos and don’ts of running POCs.
What do you see as the main benefits of conducting a POC for a SaaS product?
As an L&D team, one of our most important roles is to be a strong partner to the business. To me, this means that we need to meet our business partners where they are. Our business partners may not understand L&D terms like “the forgetting curve” or “performance support” but they definitely understand data that shows the impact of an initiative. We need to speak their language and also make sure we’re adding to and not taking away from the bottom line. A POC or pilot allows us to gather data about whether an initiative will succeed within our company’s ecosystem and culture before investing too much time or money in a full launch. With this data in hand, we can more effectively seek buy-in and create champions within the business for our initiatives.
As a buyer, what are your thoughts on paid POCs?
Most L&D teams are considered cost centers or overhead for their company. That means they’re not directly generating revenue and are instead spending money. As a leader of an L&D team, I want to make sure most of the money we spend goes to indirectly helping generate revenue (for example: influencing employee retention or improving employee skills).
If we’re going to spend money on something we’re not sure is going to do those things, I’m not often willing to take the risk. However, a paid POC is usually okay if I’ve used the software at another company successfully and I want to make sure we can get buy in at my current company. Alternatively, if the cost will go toward services from the vendor that ensure a successful implementation or plan for assessing effectiveness, it’s probably worth spending some budget.
What are the biggest mistakes companies can make when buying enterprise software?
One of the biggest mistakes is not looping in your IT team early enough in your buying process and also not leveraging their expertise around vendor management enough. At our company, our IT team is a source of great knowledge about how to choose the right vendor, the types of questions you need to ask before signing a contract, and how to create an implementation plan. Make sure to work with your Legal and security teams to protect yourself from difficult scenarios that result from unclear contracts.
How do you got about identifying success criteria for a software implementation?
It’s important to understand the problem you’re trying to solve with the software. You need to know what your goal is in order to define what success looks like. For example, if you’re trying out a library of off-the-shelf e-learning content, why are you trying it out? If it’s to offer your company more curated options for learning, you might want to choose an adoption metric as success criteria. If, on the other hand, you’re trying to decrease the amount of requests your internal L&D team receives by directing the requesters to the library, you could choose “number of redirected requests” as your measure of success.
What are some of the biggest risks that a POC overcomes?
A pilot or proof of concept can help you avoid spending budget on solutions that don’t solve your problems. They can also help you maintain credibility in your organization because you’re not investing in software that doesn’t work the way you were told it would, doesn’t deliver on customer support, or has a lot of bugs.
What are the factors that stop POCs from being useful?
If you don’t have a plan for driving adoption or incentivizing participation in the proof of concept, you won’t get enough data to be able to make a decision. Sometimes software requires a lot of setup and integration work in order to be used. These candidates generally aren’t great to do a POC with because a good portion of the time you have for testing will be spent with setup. Other factors include not setting the right success criteria and/or not creating a test environment that is similar to the actual environment you would roll out the software in.
What kind of proof do you expect to see before purchasing a piece of enterprise software?
Ideally, I’d like to be able to compare two options before purchasing a piece of software, even if it means that Option A is using the software and the results it brings and Option B is not using the software and identifying the opportunity cost. By identifying a goal and then comparing two options to reach that goal, I can be more confident in the purchasing decision.
What impact have POCs had on de-risking projects or gaining internal buy-in?
Generally the pilots we’ve engaged in have ended up with us not choosing to work with the vendor. If I’ve felt confident enough about a solution to forego a pilot, it’s usually because I already have buy-in or I know I can more easily get buy-in. On the other hand, if I have questions or concerns, that means I’m not sure how I’ll get buy-in or whether it will do what we need. In these cases, I’m looking for the vendor to help me build my case. Historically with pilots, the vendors we worked with didn’t try to understand the mindset of our stakeholders or fully understand our use case during the sales process. This ultimately led to the failure of the proof of concept. A proof-of-concept really needs to be a partnership between the prospective customer and the vendor in order to be a success.
How important a factor is the culture of the vendors you work with when making buying decisions?
It’s really important. We generally prefer to work with companies that have similar values to HubSpot, like customer-centricity and transparency. It can be hard to gauge the culture of the vendor through the sales process, but by speaking with multiple people from the company, you can usually tell if there’s alignment in your values. For instance, if a sales rep isn’t connected to their product team’s roadmap or doesn’t bring in a solutions architect to speak with you, you can assume that their organization isn’t very transparent or collaborative. Or if the sales rep isn’t able to articulate your use case or customize their conversation to your needs as a customer, this shows a possible lack of empathy or adaptability, which are both really important values at HubSpot.
What are your red flags when it comes to working with vendors?
It’s important that they have a reliable customer support model and a dedicated customer success manager. It’s also important to look at the company size of the vendor and make sure they have enough employees to support you in the way you need. For instance, if they only have 5 employees, are they going to be able to prioritize your feedback during the pilot? Will they have anyone who can answer your questions in a timely manner? A small company sometimes (but not always) indicates a lack of a plan for their software’s ongoing development. Do they have a transparent roadmap for their product that you can take a look at? This is definitely not always the case. We’ve worked with small vendors who were incredibly customer-centric and forward-thinking, but it’s important to ask the right questions. I never thought these things were red flags until my IT team brought them to my attention. They’ve helped me avoid some undesirable partnerships.
In my opinion, the vendor should provide training (or access to extensive help documentation) and an offer of office hours to answer any questions. I’d also expect them to assist with any integrations or at least answer my IT team’s questions in a timely manner. I’d also expect them to familiarize themselves with my use case and play a trusted advisor role. I’d much rather work with a sales rep who tells me they think their solution is not the right fit for me than with someone who just wants to make the sale.
Not a lot of vendors that I’ve worked with offer phone or chat support. This makes a huge difference for my team and me. When we’re first getting started with new software, it’s much easier and more desirable to be able to explain the situation to a real person in real time and get a faster solution to the problem. Having to wait a long time for a response or having to go back and forth explaining what we’re trying to do via email is a huge waste of my team’s resources and time.
In my organization and experience, the best way to have a successful proof of concept is to collect and use data to show we met our goal (or are at least trending in the right direction).
Can you give an example of a changing approach based on unexpected results of a POC?
Last year, we piloted a software that we thought would solve a need for our team — to make our e-learning design process more streamlined and enable others at HubSpot to more easily create learning content. An unexpected result of the pilot was that the software wasn’t as intuitive for users who didn’t have a learning background. We ended up using another authoring platform that was more user-friendly after we ended our partnership with the vendor.
What is your advice to others who might consider running a POC?
Running a proof of concept is an entire project in itself. You’ll need to dedicate the right amount of time to it in order for it to be useful. You’ll need to think about goal setting, marketing, influencing, troubleshooting…. Don’t take the project on lightly! But if done right, it can make a big difference in helping you make a data-driven decision.
Want to bring all this expertise to bear and run a Filtered POC in your organisation? Get in touch with the team here.