With so much industry attention on digital transformation at the moment, it’s worth paying a little attention to the way we assess new technology offerings. Is there such a thing as the perfect proof of concept (POC) and how does your firm currently go about it?
Thankfully, I’ve had some recent input to help me answer the question. Last week, I was part of a panel discussing best practices in POCs in the context of data management technology implementations for the hedge fund community. Though the crowd was primarily buy-side and the focus was data technology, the discussions were equally applicable to technology POCs of all kinds at different types of financial institution. People in technology and operations roles have also complained to me a great deal over the years about trying to get management attention on POCs, so I have my own views on why things go awry.
So, what does a good POC look like?
To begin with, it should be treated like a full-blown project. This includes setting timelines, responsibilities, governance and oversight considerations. Too often, POCs aren’t given enough internal time and attention, which results in suboptimal decision-making. If the business user community that is expected to ultimately benefit from this technology isn’t actively engaged in the POC, how on earth can the best option be selected?
POCs often go off the rails because of the lack of internal engagement from the right user communities. For example, a free trial of a data set may expire before the data can be properly assessed by the relevant business groups. The opportunity to benefit from something free is therefore either missed or the data sourcing team (in this instance) needs to spend time that could be best spent doing other things, negotiating with the vendor. The latter option can also negatively impact the vendor/client relationship and affect pricing negotiations further down the line. This could be a free trial of data, it could be a short-term opportunity to trial a vendor platform, it could be a full-blown POC with the vendor custom building a mock-up of the final solution.
There are a lot of moving parts and effective communication is at the heart of a lot of things. Hence the need to treat it all like a project. This means internal communication before, during and after the POC and communication with the vendor or consulting partner.
To the latter point, the business needs to establish exactly what it wants from the POC and communicate with the relevant vendor before the project kicks off. Setting realistic expectations before the process begins is vital, and that means setting expectations on both sides. The vendor needs to understand its role and obligations and commit to both of these. The business side needs to do the same.
The POC also needs to match the maturity level of the firm. There’s no sense in deploying an advanced analytics platform if you don’t have the clean enough data to feed into it and the right user community to actually use the platform. If you need specialised skills to operate the technology offering, the right users need to be involved at the outset. Too many useless hours have been wasted by vendors and IT teams on trialling technology that isn’t compatible with the environment in which it is to be deployed. The context is all-important.
Non-functional requirements can be as important as your technical checklist. It is easy to focus on the most obvious requirements for the POC such as the tech specs, but understanding the licensing restrictions and ongoing vendor support (to name just two criteria) is just as vital. Regulatory considerations could also sit as part of this bucket of things to evaluate, given the industry-wide focus on operational resilience. How does the vendor handle error resolution? How do they communicate with their clients? What type of cloud environment do they use (if at all)? What can you build into a service level agreement in future?
We all know that most firms operate in silos, so another important consideration for all of you is: has another part of your firm done a POC with this vendor before? Moreover, do you have some of this technology or data in use already somewhere within the firm? All important things to check ahead of time.
Most importantly and my parting sentiment, make sure that the technology solution, data service, managed service, whatever it is, actually solves a business requirement and the user community is ready to adopt it.