How to evaluate ideas

By Jeffrey Baumgartner Organizational innovation is not just about generating creative business ideas. It is also about reviewing ideas in order to identify those which are most likely to become successful innovations. Unfortunately, many organizations make mistakes in their idea review processes that result in rejecting the most potentially innovative ideas in favor of less innovative ones. In some instances, the idea review process is a simple matter of a manager reading through a batch of ideas and selecting those she believes will work best for her firm. This is most often the case in smaller firms run by a single owner and manager. In most medium to large businesses, however, a structured evaluation process is necessary in order to:

Identify the ideas that are most likely to succeed as innovations for the company. Ensure that complex ideas are reviewed by people with the appropriate expertise necessary to understand what would be necessary to implement the idea - and what might go wrong. Enable a middle manager to defend the idea to senior management, stakeholders, and financial officers who may need to grant budgetary approval of the idea. Make it possible to review a large number of ideas in a resource efficient manner. Improve the idea by identifying potential implementation problems and preparing suitable actions to overcome those problems. Sadly, this last aspect is often lost in formal idea review procedures.

Methods
There are all kinds of idea review methods. We will look at three methods that we use, pass-fail evaluation, evaluation matrices and SWOT analysis.

Pass-fail evaluation
If there are a large number of ideas that need to be reviewed -- for instance, you have run a company-wide ideas campaign on a popular issue and have generated hundreds of ideas - a simple pass-fail evaluation is often essential to bring the idea pool down to manageable levels.

It is best to start with a simple criterion for determining whether an idea will go on to a more in-depth evaluation. This criterion might be related to budget, time-frames, fit with company culture or just practical viability. Whatever the criterion, it should be made clear. If a colleague later asks what happened to her idea, you can explain why it did not pass this initial stage. Knowing that her idea did not get implemented for a pragmatic reason - such as being too expensive to implement - is more reassuring than having the idea rejected for no apparent reason. It is also important to be careful that you are not too quick to reject ideas which, with modification, might meet the pass-fail criterion. For example: a very creative idea that does not meet your budget criterion ought nevertheless to pass. You may be able to determine a means of implementing the idea at lower cost. If there are few ideas, the pass-fail evaluation is often not necessary. It is easier to move on to the more sophisticated evaluation matrix.

Evaluation matrix
The evaluation matrix is a simple array in which experts compare an idea with a set of criteria. In our experience, five criteria is best as it allows for a rounded review without bogging the evaluators down in unnecessary detail. The evaluator ranks how well the idea meets each criterion (we use a scale of 0-5 points for each criterion). Evaluators are also encouraged to provide comments elaborating on their ratings and, in particular, suggesting how the idea might be improved to overcome weaknesses. The evaluation matrix provides a criterion by criterion score as well as an overall score for each idea. Assuming several ideas, focusing on a particular problem or business issue, are being evaluated at the same time, these scores can be compared and the highest scoring ideas can be selected for further review. However, it is important to look at the evaluators' comments. An idea with a low score might be vastly improved following minor changes. We favor the evaluation matrix as the primary idea review approach because it is simple to set up, requires a minimum amount of time for review, enables comparative idea review and makes it easy to identify the most promising ideas in a large collection of ideas. That said, the evaluation matrix in itself is not usually sufficient for making a final decision on an idea that may cost millions of Euro to implement. But it helps you select ideas for more detailed review, thus making the review process more efficient.

SWOT analysis
An analysis of Strengths, Weaknesses, Opportunities and Threats (SWOT) is an old marketing stand-by and as such is a useful follow up to an evaluation matrix. In the unlikely event you are unfamiliar with SWOT analysis, it is a simple form in which reviewers indicate the potential strengths, weaknesses, opportunities and threats of an idea. Because the SWOT analysis looks at an idea from different perspectives, it provides a more rounded review of an idea than some methods.

Our SWOT analysis approach includes a scoring system in which reviewers give 0 to 5 points each for strengths and opportunities and takes away 0 to 5 points each for weaknesses and threats. This provides a SWOT metric which can be handy for comparing large numbers of ideas. We also ask evaluators to suggest methods to overcome weaknesses and threats.

Idea development
Once an idea passes these initial hurdles, it may be ready for implementation or it may require more detailed testing. We call this phase of idea review: "idea development" as it is no longer a process of evaluating an idea so much as a method to develop it for implementation. Idea development may include business case preparation, prototype development, project management initiation or test marketing. There is not room enough in this issue of Report 103 to go into idea development in detail - so we'll save it for another issue. Moreover, how a firm develops an idea depends on the nature of the idea, the nature of the firm and existing processes for implementing ideas.

Criticism versus improvement

Over the years, I have noticed that business analysts tend to be overly critical of new ideas. This is understandable, they are tasked with managing and minimising risk. And creative ideas tend to be the riskiest. As a result, many evaluators stress weaknesses and threats. On one hand, this is understandable. Your company should not be implementing ideas that will prove to be costly failures. But, many weaknesses can readily be improved. An idea that would be very expensive to implement may, with minor changes, be implemented at far lower cost. And by improving a creative idea's weaknesses, you may be turning a costly failure into a profitable success!

Evaluation teams
Evaluations should be performed by a team of people with relevant expertise. Ideally, that expertise should be varied. For instance, if you are evaluating new product ideas for an electronic gadget, your experts might include engineers, marketing people, retailers (who would sell the product) and one or more people representative of the consumers expected to buy the new products.

Evaluator agendas and prejudices

A particular benefit to having teams review ideas is that while individual evaluators are prejudiced, a varied team is likely to cancel such prejudices out.

For example: an engineer trained in an older technology may well be reluctant to give a high evaluation score to an idea that uses a new technology with which she is not familiar. The success of such an idea might well threaten her job! A jealous manager might not like the fact that her subordinates are more creative than her and so might give poor evaluation scores to creative ideas. At the other end of the spectrum, creativity and innovation people like you and I are often too enthusiastic about the most creative ideas and so give overly high scores for creativity. Sometimes, a less creative idea might prove to be the more innovative (in terms of being profitable). Jeffrey Baumgartner is the founder of Bwiti bvba, a Belgian-based company that helps organizations to become more innovative and more creative. He writes and edits Report103, a weekly newsletter on creativity, ideas, innovation and invention in business, and operates the JPB.com website. Published on 5/18/2008