Collaborate'09, and the BI Survey on Project Critical Success Factors

I'm currently sitting in the coffee area at my hotel, just back from the Orange County Conference Center after giving one of my presentations at Collaborate'09. Collaborate is a joint user group conference run by IOUG, OAUG and Quest, and this year Rittman Mead have been giving a number of sessions in the OAUG and IOUG tracks. Borkur and I arrived late on Saturday and made our way to the conference centre around lunchtime on Sunday.

Orlando itself, as is usual for trips abroad, wasn't what I initially expected and like many cities in the States is spread out over a wide area. Our hotel is luckily right next door to the conference centre, we're actually a fair few miles out from downtown Orlando but there's quite a few restaurants and bars around our hotel. It's very hot over here, in the 30C/90F range which makes a nice change from home, in fact it's so hot that Borkur and I had to make an early trip to the mall to get some summer clothes. We've been out a couple of times during the week, once with Shyam Nath and Charlie Berger to Downtown Disney, another with Mike Durran, Mike Donahoe and Alan Lee to the Tommy Bahamas restaurant across the road, also we checked out Hooters around the corner which I thought was actually a bit tame. Anyway...

The first session I took part in was the IOUG Business Intelligence Panel, along with Charlie, Dan Vlamis, a guy from IBM and myself. The question put to the panel was "what do you feel are the critical success factors for a business intelligence project", something I've been actually thinking about recently as I'd been looking through the recently released BI Survey 8, which we get a review copy of through helping bring in Oracle respondents.

In this survey, over 2000 customers were asked, amongst other things, what were the problems that they encountered when implementing business intelligence tools. To summarize the findings, the top five issues customers faced were:

Query performance too slow : 18.1%
Poor quality data : 16.4%
Company politics : 13.8%
Administrative problems (people related) : 13.7%
Could not agree on requirements : 12.0%

Other problem included "tool could not handle data volumes", "missing key product features", "tool too hard to use", "unreliable software (bugs)" and so on. But interestingly, and this was confirmed by other questions and answers in the survey, the number one issue that caused problems for users was slow query response time, and there was a direct correlation between the speed of query response and the business benefits that users reported. Interestingly, vendors and consultants thought data quality was the number one customer issue, but customers themselves rated this below the speed of query response.

Another interesting finding from the survey, and one that I felt that I had to point out in the panel session ;-), was that customers reported far higher project success rates when the project was led by either the business users, or by specialist, independent consultants, compared to implementations by the vendors' own consultants. Now whether there is a direct cause-and-effect relationship between these, or whether in general more informed and more committed customers have more project success, and also they are more likely to seek out specialist consultancies is hard to say, but when you look at the particular project issues around Oracle business intelligence projects, I can take a guess.

For Oracle BI Enterprise Edition / BI Standard Edition One customers, slightly more than average (19.5% compared to 18.1%) reported problems with query performance; a lot more than average (14.6% compared to 10.1%) reported problems with unreliable software and a whopping 17.1% compared to an average of 10.1% reported problems with missing functionality. Now I would contend that only specialist and independent consultancies would have the breadth of experience, and ability to work around the problem, to still make a success of a customer implementation given these issues. You've got to know where it can go wrong, and have a pragmatic set of workarounds and alternative strategies, in order that you don't make customers go through your learning process as you both work out how to solve product issues.

Moving on to the Oracle Classic (BI Standard Edition) product line, just under twice as many (32.4% compared to 18%) customers reported problems with query performance, and 13.5% compared to an average of 3% reported problems with security limitations in the product. Again, you've got to know what you are doing to be able to anticipate these problems and come up with solutions that don't add months to the project timeline, something the general systems implementors are unlikely to know in enough detail. Looking at the Hyperion customer scores, a far greater proportion (20% compared to 10%) had issues with product reliability and bugs, and when this sort of thing happens but you've already made the committment to buy and implement the product, you need a consulting partner who's been there and knows how to work around it. It's important to understand that all the products had one area or another that they were weak in; for Business Objects, again it was unreliable software, for SAP Business Warehouse, query performance was a huge issue, more than for any other platform. But the point still stands that dedicated, specialist consultancies that aren't tied to the company "party line" and spend all their time understanding the products are more likely to lead to project success, a point I made at the panel session.

So other than the panel, I've been to a few new sessions, hooked up with some of the Oracle BI product managers and spent a bit of time with people like Shyam Nath, Joe Leva our US Managing Director, and met up with some customers who are doing some innovative work with OBIEE. My second session of the conference, on Optimizing the Performance of the Oracle BI Applications Data Warehouse, ran earlier today and I'll be writing up the session in a later blog post. In the meantime, you can follow me on twitter (@markrittman) where I'll be posting updates on the conference as the week progresses.