The definition of business intelligence has evolved over the past 10-15 years. At that time, the 'intelligent organisation' was one that valued collaborative sharing of knowledge supported by intranets, blogs and content management systems. More about the history of BI Today, business intelligence is a much more integrated, highly strategic, management tool that supports every day decisions on how to operate the business to better achieve corporate objectives. Business Intelligence may therefore be defined as "the process of analyzing large amounts of corporate data, usually stored in largedatabases such as the Data Warehouse, tracking business performance, detecting patterns and trends, and helping enterprise business users make better decisions".
Benefits
identify market opportunities know where market share is unsatisfactory better understand their profitability drivers identify unacceptable cost areas gain an accurate view of sale and/or distribution costs, per channel, per customer, per transaction, per day recognise business areas of high performance identify the key performance indicators [KPI's] to use to measure capability calculate sales commissions, number of sales closed, highlight good and poor performers track whether strategies for certain markets or customers are working and driving business value get instant insight to the exact profit by company of each sale
The most common of these include: 1. Ownership of BI program 2. Ownership and management of data 3. BI Skillsets and roles Business Intelligence is a hot topic in most organisations today. Currently, most organizations do not have the information they need to manage and drive the performance of its key objectives. More time is spent by analysts in finding, extracting, cleaning and integrating data, instead of analyzing and reporting it. Enterprises do not have a consistent platform across all services and locations upon which to measure and manage key metrics. There are various factors are driving enterprises to take an holistic view of Performance Management. These include:
Rapid Change - many enterprises are recognising that the fast pace of change in business markets requires a new view of how the business is managed, and a new approach to preparing for the future. Governance - Following a spate of corporate scandals and alleged frauds, governments are demanding higher standards of corporate governance. Stakeholder Demands - stakeholders demand that management deliver predictable execution, operational efficiency and corporrate accountability, which collectively make Performance Management more of an imperative. Expanding on ERP - ERP's manage transactions but do not provide sufficent feedback on operational and corporate performance. Data Availability - Significant increases in enterprise data have made Performance Management more practical.
The benefits of a Performance Management initiative will include more solid foundations for making decisions, the discovery of previously hidden inefficiencies and a more rational use of resources in planning and control activities. Business intelligence suffers from a strange dichotomy: It's among the most highly desired business technologies, a roughly $10 billion-a-year market growing at more than 10% a year, yet it has difficulty proving its worth. The main barriers to BI adoption are: cost and complexity. According to a 2007 InformationWeek survey of 388 business technology professionals, over 30 percent of respondents claimed that BI vendors were unable to demonstrate the benefits of BI to internal stakeholders. Whilst most companies purchase BI software to solve a specific problem in one business unit, few are able to use the resulting data silos to explore cross-company impacts, such as how a 5% drop in market share affects manufacturing, finance, and procurement. One of the main reasons for this lack of cross organizational capability is that no vendor excels in all areas of business intelligence. This means that customers generally have to pull together the various components. For instance, some BI users are cutting edge in customer scoring, operational analysis, and predictive analytics. However, to gain that capability requires: 1. Millions of dollars on data warehouses and software for moving and analyzing data 40% of the cost involved in developing sophisticated analytics and modeling for a 30-Tbyte Teradata data warehouse comes from moving data between systems. 2. Specialized IT talent 3. Significant businesss time to set up and manage.
One Solution
Although Teradata sells analytical apps for forecasting demand, a separate application must be implemented to extract data from the data warehouse and feed it into SAS analytics, which runs models for that function. SAS is a great analytical tool but not a data warehouse, and Teradata is the best data warehouse around but not a good analytical tool.
Operational BI
Businesses also want tools that blur the line between data transactions and decision support, delivering operational BI applications with built-in analytics, and performance management software. Could this be the reason SAP has acquired Business Objects?
SAP CEO Henning Kagermann commented that "blending Business Objects' data crunching capabilities with SAP's industry knowledge for end-to-end business processes with embedded analytics." This statement seems contrary to SAP's promise to keep Business Objects a separate unit. How successful that union will be depends upon the level of integration they will achieve. Claims that SAP's previous BI tools and data warehouse have not been successful, requiring an intertweeing data warehouse between SAP and Business Objects, really don't mean much when you consider the totally different processing and data transfer capabilities of a transactional ERP/CRM system compared to an OLAP analytical application.
Actionable BI
Many BI vendors, such as Business Objects are working to better integrate reports into operational processes. This may include:
Upgrades to Crystal Reports - to produce reports from data. Better support for XML and Adobe Flex - making it easier to create reports from operational data and act on that information. Reporting needs to become an integral part of an operational process.
Smaller niche BI vendors such as SeeWhy Software are focussing on enabling "actionable BI." SeeWhy constantly compares incoming data with historical information and trends, flags anomalies [such as a regular customer not making a booking at the usual time] and sends alerts to customer service reps. Oracle is investigating how to integrate its ERP applications and data warehouse with its BI portfolio, which includes Hyperion performance management tools and Siebel analytics. Oracle Real-Time Decisions, based on software from Sigma Dynamics, acts a transactional server that combines rules and predictive analytics to deliver real-time data into business processes and customer interactions. And the innovation is not all coming from the Analytics end. Data warehouse vendors are also getting in on the game. Teradata is making its way into operational analytics with its Active Enterprise Intelligence. The trend toward real-time analysis is gaining pace, but until the perfect solution appears, companies will still need to filter through very large data sets, such as customer segmentation analysis, to choose the right prospects for marketing campaigns. Real time is just a small part of the challenge. Either way, businesses not moving towards using modeling and analytics will fall behind.
1. A patient previously treated for high cholesterol comes to the pharmacy for insulin. 2. At the point of transaction, Medco could instantly run the patient's data through SAS analytical models and determine how to reclassify that patient and whether any type of intervention might be recommended.
The Technology Enablers
The increased use of Web services makes such links between operational systems and analytics models more feasible. Real-time BI could have huge implications in customer service.
A Teradata data warehouse GoldenGate's transactional data management software Oracle's data integrator Business Objects' reports and dashboards
How BI Is Used
1. The online retailer scores each customer after every purchase based on how profitable he or she is to the company 2. That information is used for its e-mail blasts and other customer interactions. 3. If a valued customer clicks on iPod accessories, for example, Overstock might send a coupon the next day. These simple case studies demonstrate how the strategic applicBI for greater business impact, in part by embedding analytical capabilities right into everyday operations for hereand-now decision making. Given the costs and complexity, however, few companies are that far along. The BI vendors--and you can now count SAP among them--are scrambling to get it right.
The BI Lifecycle
The Business Intelligence Lifecycle follows parallel to the Data Maturity Lifecycle to provide a comprehensive maturation model on information intelligence. This model provides a roadmap to use as a guidelines in the planning of a business intelligence program. A BI Program is a long term strategy, not a short term, one time project. As the business continues to change, and technology supporting analytics continues to improve, this lifecyle can reiterate into a complete new for of performance management.
Collection [Infancy]
Data creating and collection from:
CRM - Customer Relationship Management Custom Application Development Other Enterprise Applications - ERP, HR
Reporting [Adolescence]
Data Reporting And Export
Parameter driven reporting Report distribution via Email and Web Basic data derivatization and summation
Analytics [Adulthood]
Data Analysis
Ad Hoc Query and data analysis by end users Dimensional data analysis [OLAP] Data Transformation, Integration and Lineage
Dimensional Data Design Basic data warehouse, data mart of ODS Semantic Layer or Cubes
Visualization [Maturity]
Data Visualization
Metric Dashboards & Alerting Scorecards [Strategies and Goals] Advanced Analysis & Data Mininig
BI Strategy Document
The BI vision is documented in a BI Strategy document to ensure that implementation of specific technology or a data structure remains focused on the BI objectives for a particular organisation. The BI Strategy Plan starts with high-level diagrams, broad policy statements and general definitions. As the BI environment grows, details are added to the strategy document. The strategy document offers insight into the BI environment, with the focus on communicating: 1. What is to be built 2. How it will be built 3. When it will be ready to meet user requirements.
BI Strategy Document
BI strategy documents at a minimum include four core components: 1. 2. 3. 4. Conceptual View Data Architecture Technical Architecture Implementation View
Each of these components provides readers with a unique perspective of the BI environment being planned. Each section contains alternative approaches and technology solutions. Some aspects of the overall strategy are provided in detail, whilst other elements are at a very high level or conceptual perspective. Other sections [views] may be included as required, such as:
Each Section typically contains subsections: Architecture Goals and Constraints guidelines for architects and planners to set overall goals and objectives related to the BI initiative. Any constraints that impact the BI effort must also be defined. Conceptual View a BI organization conceptual diagram used as a road map for the enterprise initiative and guide for architects and project planners to define and describe individual components. Use-Case View - provides a more business functional view of the BI strategy and used to formalize the intention of the BI initiative. Architectural Design Components a high level architectural design including only significant components, such as spatial data and analysis or ODS for tactical reporting, a message broker or XML server as a data delivery mechanism for the warehouse. These design components are important in terms of the overall BI objectives. Standards - any standards that warehouse designers and developers must adhere to, for instance, who gives final approval to any particular iteration.
Conceptual BI Architecture
The Conceptual View provides a broad overview of the entire BI vision. A conceptual BI Technical diagram is useful for illustrating how all the BI technology fits together, and assists with both the strategy definition and subsequent implementation planning.
Logical data architecture - logical models that support data architecture goals. These will be limited until a specific warehouse iteration has been reached in the BI Roadmap. However, it is possible to provide a subject area model of the enterprise or a series of subject area models that describe core subjects within the enterprise. It may also include rules for mutating raw source data into atomic-level data and even guidelines defining how and when to use star schemas and OLAP cubes. Architecturally significant design components this may include items such as: establishment of an atomic layer or an ODS or an enterprise cube farm, or specific data elements such as geospatial data structures or specialized data staging. Test plans pre-rollout test plans are a critical part of the BI Roadmap. This section includes a testing and acceptance standard for all warehouse iterations, including: test templates, criteria for enterprise adherence and approval, criteria for test data selection and performance testing (including unit, suite and stress testing). Data architecture often starts at a high level and evolves details of the architecture with each successive iteration of the warehouse.
Technical architecture goals and constraints - specific to tangible components. Technical architecture - the hardware, software and network/communication components. Architecturally significant design components for instance, you may require a 7x24 implementation with mirroring across two distinct data centers.
Implementation View
The implementation view also typically starts as a high-level perspective, with detail added as it becomes known. This section is compiled by designers and project planners to: 1. Establish the guidelines necessary for building and maintaining the purposed warehouse structures and related technologies. 2. Detail the implementation of core processes 3. Outline the sequence of establishing data structures. The implementation view will contain three distinct perspectives: the overall strategy, architecture and process.
Strategy
Timelines and Resource availability schedules to help planning and prioritizing of BI iterations using process flows. Funding
Architecture
Size and performance requirements Data quality issues Meta data control and retention policies.
The purpose is to indicate potential areas of impact, such as, retention policies will impact both data architecture [partitioning] as well as technical architecture [disk storage].
Process
When SAS executives, industry experts and leading research analysts were asked about long term BI strategy, their responses included the following insights: Question: What should organisations be doing now to help lay the foundation for long-term BI strategies? 1. Develop a BI Roadmap to deploy Enterprise Wide BI 2. Invest in an enterprise-wide platform, not silo'ed solutions for each department. 3. Complete a BI Opportunity Map - review BI tools currently used and consider if these tools will continue to provide the analytics capability needed to sustain corporate growth. 4. Manage your competitive position - ensure current BI tools can achieve the level of performance your competitors tool can. 5. Invest in people with strong problem-solving skills - experience in exploratory data analysis and predictive modeling. This is essential to understanding how to automate processes well and knowing when the approach needs to change so the automation stays current. 6. If you are considering Operational BI [and you should be!], avoid the temptation to automate and forget. You may be able to do things faster but not better, and in some cases you may end up damaging your performance. When SAS executives, industry experts and leading research analysts were asked about the evolution of BI and the importance of investing in people with strong problemsolving skills. Their responses included the following insights: Question: How do you see business intelligence being used 2 to five 5 from now?
Operational BI will drive more analytics-driven automation, tied into information delivery tools, to warn business users about troubling trends that might cause problems in the future.
Operational BI
BI solutions will include more real-time capabilities to make it easier to push information to business users in context and in real time. Its a matter of doing smart things faster versus doing not-so-smart things faster. Real-time applications will continue to be more analytically driven, as opposed to employing simple business rules. The fully integrated solutions and industry specific configurations will also make it easier for business users to deploy BI models in real time and from any online device. This supports the concept of a service-oriented architecture.
For IT, BI innovation is about a single platform for delivering information in many different ways to many different users, depending on their needs. For business users, it means finding the answers they need automatically from whatever application they normally work in every day. Its also about thorough analyses based on the most up-to-date data.
Business users no longer have to wait for BA's or IT to provide them with answers they need daily. The answers will come to them automatically via the interfaces and applications they already use every day. Adopting best practices in business intelligence give you a better chance of success with your own BI deployment. There are certain imperatives that apply regardless of what type of business you have, what type of BI solution you select or how big your BI implementation is.
Real-time data feeds require more infrastructure and are more difficult to manage. Real-time processes must be constantly monitored. Issues arise within business hours, rather than during batch laods done overnight. This requires fast response support teams, putting further pressure on support staff. Additional hardware and capacity is required to store the data. Generally, each realtime feed requires two servers, one to run the load and a second to back it up. Real-time data feeds from some source systems can be expensive and even impossible to implement.
Justification for real time data is a critical part of your BI solution planning.
Strategic decision support - analysis of large amounts of data at less frequent periods, but requiring high processing capacity Tactical decision support - repeated access to only a limited amount of data, in real time or very frequent intervals. Queries are small, requiring less processing.
These differing requirements must be scoped to set priorities - for example, a data-mining query should have a lower priority than a tactical query. Each class of query must be sized for capacity planning and priority in the queue. Continuous monitoring alerts support IT if the queue gets out of control.
Extracting data from transactional systems and feeding it in real time to a data warehouse is a finely tuned process. A process that needs to be as automated as possible. Human intervention should only be required when an alert is triggered from the monitoring system. The process must also be be a 'services' model - flexible and reusable.
Using 3NL
Storing the enterprise warehouse data in 3rd normal form supports and encourages enterprise-wide use. Data is easily maintained, redundancies are eliminated, shared data definitions are agreed and an enterprise-wide view of the data is provided. This the the most likely model to provide data that meets the requirement of 'a single version of the truth'. Business users who are confident in enterprise wide data are less protective about wanting to retain their local data silos.
BI Excellence Survey
A recent Teradata-sponsored survey, conducted by IDC found that in spite of the increasing number of companies implementing BI, the key to its ROI is the excellence in the company at using business intelligence. Findings revealed that only about 200 of the 1,072 companies excelled in their use of business intelligence.
The following tables highlight key survey comments, with "Leaders" scored four or five on a five-point scale in their use of BI, datawarehousing and analytics, and had a very high satisfaction with their current business intelligence solution. This was well above the average. The differences between companies that use BI well, and those whose use is average at best is revealed below:
1. Business intelligence leaders are more adept than market-average companies at using real-time information.
Leaders Market Av.
67%
33%
54%
13%
53%
21%
3. Leaders are more likely to give BI-tool access to people outside the organization and to front-line staff.
Leaders Market Av.
47%
24%
78%
54%
4. Management in BI-excelling companies makes business intelligence a greater priority than does management in market-average companies.
Leaders Market Av.
66%
15%
31%
8%
88%
58%
My organization would experience an "immediate" negative impact if our business intelligence systems went down.
23%
10%
A Business Intelligence Scorecard works in exactly the same way. Once BI Opportunties have been defined and BI Roadmap developed, the scorecard provides tracking against both the roadmap and the BI Maturity Lifecycle. A BI Scorecard acts as a performance reality check on whether your BI projectsare on track, and if not, how to get them back on track. It acts as a visual connector between the BI Strategy and the BI Program Each BI Scorecard is unique for a particular company based on its own set of:
BI challenges are not insignificant, nor are they insurmountable. Some challenges are corporate wide, whilst others are isolated to the functional area relating to the current BI project iteration. Typical BI Challenges include:
Enterprise
Political - Sharing data among business functions and across BI applications Cultural - Building the BI organization Financial - Determining value received from the BI investment
Functional / Operational
Responding to new business requirements Resolving system performance issues Defining repeatable development processes Achieving user adoption and satisfaction
Project
Fulfilling the promise - ensuring all business requirements are realized with each new project iteration. Alignment - bridging the gap between the existing IT infrastructure and strategic objectives. Value perception - ensuring BI systems developed drive value. Scope creep - incorporate new business requirements into the BI development framework
BI Program
Funding - justify and obtain new or additional funding Stakeholder confidence - address business stakeholders dissatisfaction with existing BI capabilities. Resource - understand and resolve gaps in skill sets and organizational structures. Identifying and Tracking BI Opportunities
Developing A BI Scorecard
A BI Scorecard represents a 'current state' assessment upon which step-by-step improvement tactics are added. It is a living document that is continuoulsy updated. A typcial BI Scorecard includes seven categories and sub-categories each with its own metrics, interview questions, and scoring procedures.
Key Scorecard Categories
1. 2. 3. 4.
Organizational Roles and Responsibilities Requirements Gathering Data Management Database Platform and Architecture
5. ETL (Extract, Transform, and Load) 6. Production Management and Support 7. Project Management A BI Scorecard visually displays these performance categories, along with their scores and rankings. This is appended by a detailed description of the organizations current state for each ranking.
Corporate Scorecard
and understanding of the BI Program within the organization, and motivates it to take the next steps forward. Apart from being a program management tool, a BI Scorecard acts to facilitate alignment between IT and the business, around BI issues, opportunities, and action plans. This alignment helps build bridges between these two 'entities' which generally have long past histories of disparity. This can transcend into distinct benefits for the BI Program, such as:
Renewed management attention on data warehousing and BI as bona-fide business initiatives. Additional funding for recommended improvements. Willingness to invest in previously questioned projects, such as metadata management. Cost savings from application or platform integration. Revenue growth from new capabilities that support business organizations like marketing and sales. Job roles better aligned to actual work activities. Organizational re-alignment, often in support of the Center of Excellence model. Helps business stakeholders and IT executives with both the issues and opportunities around BI. Provides a platform for making investment decisions for significant improvement; accelerating the organization forward to the next level of BI excellence.
Scorecards creates a holistic picture of your BI and data warehouse environment then zeros in on common problem areas. This allows us to analyze the issues, make connections, draw relationships, and isolate root causes of the real problems.
Most leading BI vendors such as Cognos, SPSS and SAS and more horizontal solution providers such as Teradata, Unica and Oracle have developed integration solutions into ERP systems as well a other line of business applications such as call center, marketing and revenue assurance. ERP vendors may also bundle BI solutions into their enterprise packages at little additional cost, in an effort to shortcut a BI/ERP roadmap. Unfortunately the BI component generally falls short of best-of-breed BI solutions. Stand alone BI solutions better integrate with other applications to extract data from more sources.
SOA and BI
BI and predictive analytics leverage investments in service-oriented architecture [SOA]. SOA services are highly customizable. So if they are already being used to integrate your enterprise applications, they can also be customized to monitor the business activity flowing over those systems. Both Cognos 8 and Business Objects XI are built on SOA. SOA initiatives often get isolated as small, localized deployments with little strategic visibility or scale. By exposing BI and ERP services through SOA, a more strategic business case can be made for further deployments.
A key takeaway from this video is that for small businesses, competition is NOT just those other like businesses in thier local geographic regions. They are also competing with small businesses all over the world.
BI For SMB
The expense of BI tools does prevent many SMBs from making the investiment in business intelligence insight. This has lead many BI vendors to provide BI solutions in a 'Software as a Service', also called 'On-Demand' model. More on BI On Demand.