Towards a Future Oracle BI Architecture?

One of the presentations I'm giving over the next couple of months is one for the UKOUG BI & Performance Management event on future Oracle BI architectures. The driver for this for me is around all the different options that are now available for building Oracle BI systems, now that we've got products from Siebel, products from Hyperion and of course the traditional Oracle BI tools such as Discoverer, Portal, Daily Business Intelligence and Oracle Reports.

In particular, having all these new tools opens up a number of questions around how you put together a BI system going into the future.

• Do you still build a data warehouse upfront, dealing with all the data and integration issues upfront, or do you use the data integration features of BI tools such as Oracle BI Suite Enterprise Edition to integrate the data on-the-fly?
• Do you plan and design your warehouse upfront as an IT initiative, or do you let the users design the reports and BI metadata and have those drive the warehouse design?
• How do you incorporate the planning and budgeting process into your warehouse design?
• Given the productivity of tools like OBIEE, can we significantly shorten the time to deliver BI projects?
• If we go for a mixed BI/DW architecture featuring data warehousing, ETL, BI tools and their metadata, planning tools, OLAP servers and so on, how do create a consistent level of security over our environment, and how do we capture metadata and metrics across the entire system?
• Finally, give the adoption of technologies such as SOA, Web Services, data service layers and the like, how do we incorporate these sources and activities into our BI architecture so we can leverage their features, incorporate their data and still stay relevant when not all business logic and data is held in databases?
• Do you still base your BI systems around a single, monolithic data warehouse, that integrates data into a single database, deals with all the data issues once and once only, applies Oracle performance techniques such as materialized views, bitmap indexes and partitioning, with the BI element of reports and graphs delivered towards the end of the project? Or do you use the prototyping and data integration features of tools such as Oracle BI Enterprise Edition to create your data warehouse on?

So you see, it's quite an interesting, but tricky topic, but one that I think is very pertinent to Oracle BI users and customers given some of the feedback I've had at user group events - not quite knowing what to do next following all of Oracle's acquisitions is the number one issue I hear from feedback sessions at events, and it's this uncertainty I'm looking to address.

Anyway, to cut a long story short, an idea that I've been batting around for a while, and something I've discussed with people such as Doug Cackett and Andrew Bond in Oracle, is an attempt to try and pull together an architecture that recognizes the benefits of a properly structured and loaded data warehouse, but that incorporates some of the new options that are open to us now that tools like OBIEE are around.

Data Warehouses give us consistent, reliable data plus the benefits of database server technology designed to handle large sets of summary and detail-level data. However they take a long time to build and as far as BI systems are concerned, the value they provide dimishes rapidly the longer you take to deliver them. Your warehouse project might get signed off based on some immediate market opportunities your company has spotted, but if you take a year to deliver the system, by then the opportunity may well have gone.

Tools like OBIEE however allow you to put a BI system together now, mapping its metadata layer directly against the underlying source data, applying aggregates and caches to try and speed up query performance. In some cases, tools like OBIEE can actually allow you to integrate data across multiple systems, though this will never be as fast to query as data that actually resides in just one database. Over time, systems built like this get harder and harder to manage, but at least you get reports and analysis in people's hands fast and the business aren't usually interested in arcane technical discussions about the merits of a properly architected data warehouse.

The thing is though, what it you could combine the two approaches, having the business define the reports and the report metadata (in OBIEE terms, the semantic model) and with the metadata layer initially mapping through directly to the source data,. Then, as time goes by, you can migrate more and more of the data you're reporting on to a proper data warehouse, all the time keeping the user's reports going through the ability of the BI tool metadata to be "re-pointed" to the data warehouse as subject areas come online? Tools like OBIEE allow business users to define a logical dimensional model that query tools then work against, as a BI architect you can start with this metadata layer pointing to the source data and over time migrate it to proper data warehouse structures.

At the moment, building these warehouse structures requires you to define a dimensional model in a tool like OWB, then write your mappings to bring across data from source systems into the warehouse. In time though, features on the OBIEE product roadmap promise to considerably simplify this process, with OBIEE offering the option to persist the logical model it works with in a relational schema or a multi-dimensional database such as Essbase or Oracle OLAP, and in the future it wouldn't take too much work for OBIEE to also generate the ETL routines in a tool such as Oracle Data Integrator to copy data from your source systems into this persistence (or in other words, data warehouse) layer.

So you see the benefit here is that we put the drivers for our BI system into the hands of the business users, allowing them to create their business model and the reports that make use of it, whilst in the background we can gradually consolidate the data they report on into a proper data warehouse, in future using features to come in OBIEE to automatically generate the warehouse model or cube, and the mappings that move the data from the source systems into this data store.

The way in which we move data from sources into the warehouse, or persistence layer, is likely to evolve over time as well. At present, 99% of the data movements I see on projects are data mappings that move data from one database table to another. In future though, we're likely to see data coming in via web services, via enterprise service bus and messages, via business events and so on, and so the tools we use to move that data will need to evolve as well. Tools like Oracle Data Integrator are already able to handle data via these sources, whilst the next release of OWB is also slated to handle web service and message sources. Taking a step on, it's worth thinking about architecting the data integration element of your BI system as a formal SOA "Data Services Layer" where data is abstracted away from your BI tools, integrated using a set of loosely-coupled services, and handled in a framework such as Oracle Fusion Middleware. Is this likely to be a dominant pattern in the near future? I can't say for sure, but as organizations move to SOA architectures we likely to get our data more and more via non-traditional data warehouse sources.

Finally, what about the Hyperion tools, how are they going to fit into this architecture? Well in terms of specifics, who knows yet, but for now I'd say some likely integration points are in the BI tools metadata layer, in the dashboard frameworks that deliver planning, budgeting and operational BI reports, and I guess in time both planning & budgeting data, and the data on our operational performance, will more and more be available as services to be consumed by a wider BI architecture, so the distinction between reporting data, planning data, forecasting data, data derived from databases or from business events, and so on and so on, will eventually become blurred - they'll all just be services in a service-orientated architecture.

Anyway, that's some thoughts for me around how we can preserve the benefits of a data warehouse architecture but take advantage of the new capabilities of OBIEE, and also respond to the take-up of service-orientated architecure. I'd be keen to get other people's feedback (especially if you're going to the UKOUG event) to see whether this makes sense, whether it addresses the questions you've got around where Oracle's BI architecture is going, whether you think it's realistic to start off with an OBIEE model going against source data and eventually migrate it to running against a data warehouse, whether you think events and services will be big factors in BI architectures, and so on. Add some comments if you like, I'll respond to the comments and follow-up later in the week.