A Future Oracle OBIEE Architecture

Following on from my blog article earlier this week on future Oracle BI architectures, I've read through the comments and had a think about how things might work, and put together some thoughts on how organizations might use Oracle's Enterprise Edition BI tools now and going into the future.

To recap, the idea here is to take the traditional BI architecture of a data warehouse, metadata layer and a query tool and update it for the heterogeneous, service-orientated world that is Oracle now. In this architecture, we're going to take advantage of the ability for Oracle Business Intelligence Enterprise Edition Plus (OBIEE, previously known as Siebel Analytics) to map to multiple data sources (including applications, relational databases and OLAP servers) and create a single logical business model, and the fact that Oracle now have an ETL tool in Oracle Data Integrator that can extract from and load to a range of different databases, web services and other business services. We're also going to try and think about how the tools from Hyperion will work in this architecture, both now in their unintegrated state and going into the future, when Oracle will no doubt try and integrate them with the OBIEE platform.

To start at a high level, the proposed future Oracle BI architecture will have three distinct layers;

a Data Services Layer that contains the data and associated ETL processes
a Business Logic Layer that maps to the Oracle BI Server, and
a Presentation Layer that maps to the Oracle BI Presentation Server and the Hyperion planning and performance management tools.

The idea of a data services layer comes from the SOA world, and is a way of creating an abstraction layer for SOA business processes that need to access business data. In our architecture, it contains the application data sources, the data warehouse when we create it, OLAP data sources, web service and business event data, and the ETL processes that moves and transforms data. The business logic layer is where the BI Server holds physical, logical and presentation models of your data, together with hierarchies, calculations, metrics, KPIs and access rules, with the presentation layer mapping to Oracle BI Presentation Server and the legacy Hyperion CPM tools that take data from the business logic layer in the former case, and the data services layer (at least initially) in the latter case.

To illustrate how the architecture might be developed, we can consider a typical organization who wishes to implement an Oracle business intelligence solution. For this organization, their deployment can be broken down into three separate phases:

Initial deployment of pilot reports as "quick wins" and to determine detailed requirements
Gradual consolidation of reporting data into a data warehouse
Implementation of advanced functionality in the OBIEE 11g+ timeline

This assumes that the organization has no particular BI strategy in place at the moment, currently provides reports through disparate tools directly against source applications, and wishes to provide a reporting solution as part of a wider Oracle Fusion Middleware strategy.

1) Initial Pilot Deployment

Organizations looking to deploy BI solutions often have two conflicting drivers for how they approach their project; they want to get reports to users quickly, so that they can take advantage of the market opportunity that's led them to want a BI solution, however the IT department are often concerned with the long-term viability of the reporting solution and wish to build it based on a data warehouse.

In the past, it was often difficult to cover both of these requirements as once you starting building reports and a metadata layer against your operational applications, it was difficult to re-point these towards a data warehouse once you started building one. With OBIEE though, organizations can create a metadata model that separates the logical objects that users report against from the physical database tables that provide data for them.

Therefore, in this first phase, we will create an initial metadata layer in the business logic layer using Oracle BI Server and Oracle BI Administration, and initially point it towards the source applications that contain the data users want to report on. Where possible, a single logical model will be created, however at this early stage it's likely that the business logic later will contain individual logical, physical and presentation models for each source system being accessed.

Where appropriate, Oracle Data Integrator will be used to created pre-joined copies of certain source application tables, together with some limited aggregations, which will be placed in a database held within the data services layer. These tables will go on to form the kernel of the data warehouse that will be built in the next phase. Using this initial data services, business logic and presentation layers, the business can provide some initial reports, gain some quick wins and also provide a pilot platform that helps the process of gathering more detailed requirements.

2) Gradual Consolidation of Data into a Data Warehouse

The main phase in development follows the initial pilot and quick wins and looks to put in place a scalable, consolidated reporting environment. Where appropriate, data is extracted in real-time or batch from the source applications and consolidated into a classic data warehouse, with staging, atomic and dimensional (performance) layers, ideally using a database such as Oracle to provide indexing, summary management, in-database OLAP and storage of large amounts of detailed and summary-level data.

For more complex analytical needs, or if the business wishes to use planning, budgeting or financial consolidation tools, data can be loaded from the warehouse into an OLAP server such as Essbase, with all transformations and data loading taking place through Oracle Data Integrator and its repository. ODI also provides a means to integrate data via messages, enterprise service bus, web services and business events, with the overall data service layer refresh process being orchestrate through BPEL and ODI agents.

The primary route for data out of the data services layer is through this data warehouse. For data sources that are transitory in nature, or that have not been incorporated into the warehouse yet, direct access to these sources will still be supported with the business logic layer continuing to map to these sources as needed, as well as the new data warehouse.

The business logic layer, initially defined in the pilot phase, contains models describing the data sources within the data access layer, the business model over these data sources, and the customized views of the business model used by different parts of the organization.

Initially, most of the query tools within the presentation layer will access their data through this business model. Legacy Hyperion tools will for the time being bypass this layer and talk directly to the Essbase server in the data services layer.

In the initial phases of the BI implementation it is likely that multiple logical business models will exist in this layer whilst separate data sources are being combined into combined fact tables and conformed dimensions, the end goal though is to produce a single unified logical model that spans the entire organization and allows analysis across linked dimensions and facts.

3) Implementation of advanced functionality in the OBIEE 11g+ timeline

Everything in the architecture so far is supported by the current generation of OBIEE, ODI, Oracle SOA Suite and Hyperion tools. Going into the future though, there are obvious ways in which this architecture can be improved as integration continues between Oracle's BI tools.

At present, OBIEE can generate aggregates to speed up user queries through the use of the Aggregate Persistence Wizard. It is likely that in future, OBIEE will also be able to generate a physical database schema to match a logical business model in the OBIEE repository, either in a relational database or as a multi-dimensional cube in Essbase or Oracle OLAP.

It's also reasonably likely that OBIEE will also be capable of automatically generating ETL mappings for tools such as ODI, "function-shipping" the process of transforming data to an ETL tool rather than have the BI Server do it at runtime. To what extent these function-shipped ETL mappings will be able to handle complex mappings is obviously not known.

It is also likely that much of the metadata used by the former Hyperion tools will make it's way into the OBIEE physical, business and presentation models, with the business model undergoing development so that it can hold items such as metrics, KPIs and goals as first-class data items.

Eventually, you can imagine the existing OLAP query tools used by Essbase (Web Intelligence etc) being replaced by the forthcoming release of Oracle Answers, which can present data as a multi-dimensional hierarchical model, negating the need for standalone OLAP query tools.

So what's left? Well we haven't tackled some of the "edge case" OBIEE tools such as Real-Time Decisions (this is likely to sit in the business logic layer and provide services for the presentation layer), and tools such as Brio, Discoverer and the like which are not part of Oracle's strategic direction.

It also rather glosses over the slightly awkward manner in which ODI provides access to service and event-based data (more on that in a later posting), and of course bringing data together into a warehouse whilst at the same time providing consistent data for the business logic layer is of course a complex task, but this is I guess where the data mart automation tools coming in OBIEE 11g+ are likely to help out - the idea here is that persisting your report data in a dimensional model should be a fairly automated process, not one that has to be re-invented for every deployment.

For now though, hopefully it's a fair stab I hope at pulling together a coherent, next-gen Oracle BI architecture, as usual comments are welcome and I'd appreciate any feedback before I start to put the presentation together.