An Evaluation Model for GenAI Initiatives
So, let me give you a short background on what you’re dealing with here. My evaluation model – and I think this is actually quite important – represents a methodical approach to assess whether specific measures and pilot projects in the Generative AI space make sense or not. What I’m trying to do here is find a balance between, let’s say, emerging ideas for using Generative AI, the current capabilities of the models, the business impact of these ideas, and – thisis crucial – how well they fit culturally within the organization.
Through my model, I can quickly identify which measures should get priority and which ones – and this is where it gets interesting – need to be put on the back burner for now. And more importantly, I explain the reasons behind these decisions.
A Tool for Selecting
Gen-AI Measures
Look, I’ll be honest with you – in a world that’s characterized by complexity and, let’s face it, complete chaos, the tangible adoption of generative AI in companies remains a challenge. That’s just the reality I’m dealing with.
But here’s the thing:
keeping track of everything and aligning all factors toward a common course requires wisdom and foresight. That’s why I’ve reduced the various perspectives to the three well-known aspects of „Desirability,“ „Feasibility,“ and „Viability.“ This reduction – and I think this is really smart – should enable clearer discourse about the suitability of specific measures.
My goal is to create deep understanding for the organization. And what’s absolutely essential here is that technical feasibility, symbolized by the intelligence of the models, is just as important as business relevance and employee support. Only when these three levels are united in a measure does it lead to success.
A Concrete Example
In many companies, ideas that come from high-status individuals or were copied from competitors get implemented. But this approach can be misleading because – and this is something I always tell my clients – companies are unique, and an idea might not fit existing processes.
Consider a company that sells a complex technical product about which there’s little publicly available information and no internal documents or resources exist. In this case, developing intelligent chatbots isn’t a sensible measure because the technical feasibility simply isn’t there. No LLM can meaningfully advise on specific topics without extensive fine-tuning, and customers often don’t have the necessary background knowledge to use such tools effectively.
Here's Where It Gets Interesting
If you flip this consideration around, it shows which measures can improve company performance in the short term. Is the product consultation difficult and the sales team overloaded while marketing lacks detailed knowledge? Then targeted Gen-AI applications can help. Internal chatbots could transcribe meetings and make knowledge accessible. Fact-checker tools could review marketing content and point out technical accuracy issues. Tools for simplifying technical jargon could serve as learning aids for employees.
These measures directly address the company’s bottlenecks, require little to no model expertise, and support employees in their daily work. They’re probably faster to implement, will be more easily accepted by employees, and thus lead to positive economic effects for the company more quickly.
The Evaluation Model in Detail
The evaluation model consists of three overlapping circles – what I call a Venn diagram. These three circles symbolically represent the three aspects: Desirability, Viability, and Feasibility. Areas where these three aspects overlap naturally lead to adoption. So measures that fulfill all three aspects simultaneously will have a very high success rate.
Areas where only two of the three circles overlap represent a sub-optimal compromise. This is where adoption difficulties occur with measures.
The 3 Aspects
Feasibility
This aspect deals with the implementability of specific use cases. I’m talking about questions like: „Are there models that can cover the work steps?“, „Are there ready-made solutions for this, or can you build the infrastructure?“, or „Is it legally permitted to implement this idea?“ Feasibility primarily relates to content-related questions that become important during solution implementation. If this aspect isn’t fulfilled, implementation becomes lengthy, difficult, or even impossible.
DesirabilityÂ
This aspect focuses on the softer, more cultural parts of measure evaluation and puts people’s needs at the center. It’s about questions like: „Do your customers want to use this and does it solve their problems?“, „Do your employees want to use this and does it make their work easier?“, or „Is it ethically and morally acceptable if you implement this idea?“ If this aspect isn’t fulfilled, implementation will encounter push-back in the company, be ignored by end customers, or even be publicly criticized.
ViabilityÂ
This aspect puts economic viability at the center and deals with the benefit of measures. It’s primarily about having an economically positive effect for the company overall. I’m talking about questions like: „Can the operational costs be sustained?“, „How much does this change the Customer Lifetime Value?“, „Is the use case significant enough compared to other processes?“, or „Does it solve one of your growth problems?“ If this aspect isn’t fulfilled, implementation won’t „survive“ long after completion or will be abandoned during implementation.
My personal Take
The evaluation model – and I think this is actually quite profound – is a tool for reflection and self-awareness that critically questions existing measures for Gen-AI adoption. Companies must have the courage to rethink these approaches and expand their capacity for action. This enables not only conscious and efficient resource allocation but also focusing on the really effective generative AI measures.
Only through this differentiated yet simple perspective can you master the challenges of digital transformation and look toward an inspiring future. It’s time to become active and shape change pragmatically and solution-oriented.
More stuff to read
Â