An Interview with Jean-Pierre Dijcks, OWB Product Manager

The other week I posted an interview on the blog with Phil Bates , the man behind the architecture of Oracle Business Intelligence Enterprise Edition. As the interview with Phil worked out so well, I thought I'd follow this up with an interview with Jean-Pierre Dijcks, one of the Oracle Warehouse Builder Product Managers and one of the contributors to the Oracle Warehouse Builder blog. Jean-Pierre is a great contributor to the OTN forums, the OWB blogosphere and user group events in Europe and the USA, and kindly agreed to answer some of my questions on OWB and it's future direction. As with Phil, if you've got any follow-up questions just add them as comments to the article, and hopefully Jean-Pierre will be able to get back to you with some answers. Here we go...

[Mark Rittman] "What are the key new features in the next release of OWB, and what impact will these have on how OWB developers work?"

[Jean-Pierre Dijcks] "Well, 11g Release 2 is going to be a major release again. But before going into that, I wanted to mention something we did in 11g Release 1 as an update. We have added interesting parts for the Oracle MDM suite into OWB. One part is a connector for the MDM systems (Universal Customer Master, Customer Data Hub and Product Information Management Hub) and pre-created content in the form of MDM specific data rules. When combined with the OWB Data Quality option you get a solution called Data Watch and Repair for MDM. This solution allows you to profile and correct data in your MDM solution based on those pre-created rules. For more information have a look at this blog entry: http://blogs.oracle.com/warehousebuilder/newsItems/viewFullItem$558

Now, on to the 11g Release 2 features. In ETL (or should I say ELT) the biggest ticket item is the integration of Oracle Data Integrator Knowledge Modules (KM). You will be able to import the KMs straight into OWB and use them. You can create new KMs, change existing ones etc. This gives OWB more flexible connectivity but will not replace the gateways for example. Both methods are available. The plan is to support things like LKMs, IKMs, JKMs and CKMs. The latter two are probably the most interesting. Imagine CKMs and data rules combined... We are also adding something called Transformation KMs, allowing you to use your regular operators as if they were a KM. As we continue the beta and get closer to our release dates you will see more information around all of this. If you are at ODTUG, you will be able to see most of this in action.

A second feature area that might be interesting for the BI realm is the tighter integration with Oracle BI Enterprise Edition. In short we will do for OBI EE what we did for Oracle Discoverer. You can derive your dimensional model from OWB and generate a complete OBI EE application, including the presentation layer. This integration will give you end-to-end lineage from within OWB, spanning source to BI at attribute level.

To continue the data warehouse / BI angle a bit, we are also extending the dimensional modeling piece to include something we called Orphan Management. Orphan management productizes the common best practice of creating dimension members that act as capture all buckets. If my customer does not yet exist, but there is an order placed, instead of discarding the order OWB will now load it to the default dimension. Commonly we use surrogate keys like -1. Since everyone does this in projects, having it in the tool made sense. So here it is.

The other improvement we made is around web services. As you all hopefully know, OWB supports reading from and publishing to web services today. All based on the Oracle database infrastructure. However we figured we should make it a little simpler in the next release and expand it to non-Oracle platforms. This ties back to the KM feature, where a mapping may not run on Oracle. These mappings should be available as web services as well, so that is what we added."

[MR] "A question I am often asked is how OWB and ODI will relate together going into the future. Will the two products merge, or will one become the favored ETL tool for Oracle developers. Will OWB still be around in two or three years time?"

[JP] "Lets start with the important thing first. OWB will be around for years to come and we are seriously investing in it as you can see from the answer to the first question. One of the goals in the next release is to extend the product with heterogeneous connectivity (so we leverage the ODI KMs) and make OWB a tool of choice for every Oracle database customer. This is the first big step in OWB to get tighter integration.

What is the future of the two products? I think they will live happily ever after... and as you see in OWB plans, technology is being shared between the products. Both products will move into the same IDE paradigm. Is there overlap, sure, but as a customer you can easily pick one and solve all your integration problems with just that one tool.

For OWB, if you are running mostly on Oracle database targets you should be able to solve all your data integration and data quality needs with it. Data Quality is tightly integrated, runs within Oracle for optimal performance. Data integration is high-performance set based code (EL-T) or row based PL/SQL for exceptional flexibility. There is a large set of source connections possible to many databases, XML and packaged applications, even without the KM integration. Most importantly the flat file support in OWB is extremely good compared to the competition."

[MR] "One result of all the acquisitions Oracle has made is that there are many repositories of BI metadata in the Oracle world. How can OWB address this and are there plans to create a single BI metadata store, and indeed single BI administration tool, going into the future?"

[JP] "I think we all are searching for the Holy Grail in metadata, which is that single repository. With the expansion of Oracle BI I think the problem just got harder... As you can see in the new features I discussed before, we are trying to make your life easier with better OBI EE integration. We already model Oracle OLAP metadata and have all of it in a single repository. So you can make OWB a central point in your BI environment, but it is not that single ultimate metadata place that is a global repository. At the end of the day metadata is a means to an end. Taking all priorities into account, creating a single Oracle BI repository will cost a lot of resources and time. It is complex and might go at the cost of features an end user needs day-to-day."

[MR] "One key feature announced for OWB11gR2 is support for SOA and web services. What form will this support take, and how does data from web services fit into an ETL infrastructure?"

[JP] "As with metadata, I think web services are a means to an end. It is a way to modularize pieces of logic and expose them to an ecosystem of users. For ETL I think that is an important thought. The more we invest in modular components that do certain (highly complex) tasks, the simpler our ETL construction becomes and hopefully one day we can have true "end-user ETL". To me that is one goal web services could bring us closer to. End users would have ETL Lego blocks, pick from a library with data sources, transformation services, verification services and data targets and build their ad-hoc reporting system as if they were using Lego blocks.

The other angle is of course the much simplified access to external data. I'm thinking data enrichment, validation etc. Those are some of the common SOA examples, but could play a very interesting role in data integration. I can augment my data before loading it into my customer dimension to ensure I have the correct addresses and phone numbers for all my customers. It is not feasible to have all that lookup data in house. Web services can help here.

For now that latter use case is what we are focusing on in OWB. And we focus on both angles, e.g. being a provider and a consumer. From the provider side we are exposing mappings and processes as a web service to allow others to leverage the OWB transformation power. On the consumer side we are adding native support for calling web services to deliver data into a process. You can do that with today's OWB, but we are trying to make this easier and allow a wider audience to use web services."

[MR] "If you could have three wishes for new features in a possible OWB12g, what would they be?"

[JP] "If they are wishes I can comment... just don't expect us to deliver them all!

1) A release management system allowing me to create and manage releases of the data integration application completely within OWB. And yes we have features to help today like multi-configuration and snapshots, but I would love to build on that and manage it all within the OWB context. Oh, and I'm not talking about being able to check out some objects, I'm talking the whole thing. Pick the version of the object for a release, verify and check dependencies, check out the objects required in the required versions and manage the deployment actions start to finish. Then report back on it. And maybe just do it all from a browser.

2) A parser for custom PL/SQL and SQL that would allow me to convert all the custom code out there into OWB mappings (hey you said wishes...). That way we can leverage the metadata facilities in OWB to help people manage their systems better and easier. Make changes quicker etc. The only way to do that is to really move custom code into code generation tools.

3) For the third one I'm on the fence and have to choose between end user ETL or end user data quality. But why choose, lets just do both. I would love us to expand the capabilities of OWB into the end user realm. For both these topics we should have an interface that business users understand and where they can describe a data integration process or a data quality rule in almost regular language or at least in their language (being E-Business screens where they see their data, being a rule expressed in Siebel CRM contact references and other business speak). All of this would empower end users more while still keeping the system centrally managed and secure. Flexibility for end users without creating spreadmarts. The best of both worlds in my opinion."

Thanks again Jean-Pierre. As I mentioned earlier, if you've got any follow-up questions, just add to them to this posting. If you've got any suggestions for more product manager interviews, again let me know and I'll see what I can do.