Oracle Data Integrator 12c release : Part 2

In the first part of this article, we discovered the new mappings and reusable mappings, the debugger and the support of OWB jobs in the freshly released ODI 12c. Let's continue to cover the new features in ODI 12c with the Weblogic Management Framework, the new component KMs and some features to improve performances or ease of use.

WebLogic Management Framework

Another big change with this new version is the new WebLogic Management Framework used to manage the standalone agent in replacement of OPMN (Oracle Process Manager and Notification). Based on Weblogic Server, it brings a unified management of Fusion Middleware components and it provides a wizard for the configuration - the Fusion Middleware Configuration Wizzard.

Practically the standalone agent is now installed outside of the ODI installation, in a domain similar to WLS domains. The benefits of it, apart from the centralised management, is that it supports Weblogic Scripting Tool (WLST) and can be monitored through Enterprise Manage Fusion Middleware Control or Enterprise Manager Cloud Control. The standalone agent can be launched manually or through the Node Manager.

Next to the standard KMs - now called Template KMs -, a new type of knowledge modules has been introduced in this release. Going one step further in the reusability, the Component KMs contain steps that are defined once and shared among several of them. Instead of being templates of code with some call to the Susbtitution API at runtime , these KMs are actually compiled libraries that will generate the code based on the components present in a mapping.

The Component KMs are provided out-of-the-box and are not editable. You can check the type of a KM in the physical tab of a mapping.

Knowledge Module Editor

Don't worry, you can still import, edit and create Template KMs. You can now find them in the folder <ODI_HOME>/odi/sdk/xml-reference.

Good news, the KM Editor has been reviewed to avoid us to constantly switch from one tab to another. Now when the focus is set on a KM task, it is directly displayed in the Property Editor. In addition the command field supports syntax highlighting and auto-completion (!!). Options are also now directly managed from the KM editor and not from the Project tree.

In-Session Parallelism

With ODI 12c, it is now possible to have the extract tasks (LKMs) running in parallel. It's actually done by default. If two sources are located in the same execution unit on the physical tab, they will run in parallel. If you want a sequential execution, you can drag and drop one of your units onto a blank area. A new execution unit will be created and ODI will choose in which order it will be loaded.

Thanks to this the execution time can be reduced, especially if your sources come from different dataservers.

In the topology, you can now define the number of threads per Session for your physical agents to avoid one session to get all the resources.

Parallel Target Table Load

With ODI 11g, if two interfaces loading data in the same datastore are executed at the same time or if the same interface is executed twice, you can face some problem. For instance a session might delete a worktable in which the other session wants to insert data.

This now belongs to the past. With 12c, a new "Use Unique Temporary Object Names" checkbox appears in the Physical tab of your mapping. If you select it, the worktables for every session will have a unique name. You are now sure that another session won't delete it or insert other data.

I can hear a little voice : "Ok but what if the execution of the mapping fails? With these unique name, these tables won't be deleted the next time we run the mapping." Don't worry, the ODI team thought about everything. A task in a KM can now be created under the section "Mapping Cleanup". It will be executed at the end of the mapping even if it fails. That's a nice feature that I will also use for my logging tasks!

"But eh, what if the agent crashes? These cleanup tasks won't be executed anyway." No worries... A new ODI Tool is now available : OdiRemoveTemporaryObjects. It runs every cleanup tasks that have not run normally and by default it is automatically called when a agent restarts. Little voice 0 - 2 ODI Team.

Session Blueprint

When doing real-time integration with ODI, you can have a lot of jobs running every few seconds or minutes. For each call of a scenario, the ODI 11g agent retrieves all the steps and tasks it should execute from the repository and write it synchronously in the logs before starting to execute anything. After the execution it deletes the logs that shouldn't be kept when the log level is low. At the end of the day, even if you set the logs to a low level, you end up with a lot of redo logs and archive logs in your database.

Instead of retrieving the scenario for every execution, the ODI 12c agent now only retrieves a Session Blueprint once and keep it in the cache. As it is cached in the agent, there is no need to get anything from the repository if the job runs again a few minutes later. The agent also write - asynchronous - only the needed logs defined by the log level, as it only relies on the blueprint for it's execution and not on the logs. The parameters related to blueprints are available in the definition of the physical agents in the topology.

In summary, thanks to the Session Blueprints the overhead of executing session is greatly reduced. The agent needs to retrieve less data from the repository, doesn't insert-and-delete logs anymore and write it asynchronously. Isn't it great?

Datastore Change Notification

This happened to me once again last week : we needed to rename a datastore column in ODI but we couldn't because it was used as a target datastore in an interface. The solution is to remove the mapping expression for this column in every interface where the datastore is used as a target.

It will not happen anymore with 12c. Now, you can change it in the model and a change notification will be displayed in every mapping using this datastore. In the following example, I removed the LOAD_DATE column.

Wallet

Ever faced this dilemma in 11g ? Giving the Master Repository password to everyone is not safe, but saving it in the Login Credentials is not better if a developer forgets to lock his screen when he is away from keyboard.

Once again, 12c is there to help you. You can now store your Login Credentials in a Wallet, itself protected by a password. One password to rule them all... and keep them safe!

Try it !

If you want to give a try to ODI 12c and follow the Getting Started Guide, you can download a Pre-Built Virtual Machine. It's running on Oracle Enterprise Linux with an Oracle XE Database 11.2.0. Of course, you will need Virtual Box to run it.

See you soon

Needless to say I'm very exited about this release. Combining the greatness of the ODI architecture with the new flow-based paradigm and all the new features, ODI is now a very mature tool and the best-of-breed for Data Integration regardless of the source or the target technology.

Keep watching this blog, there are more posts to come about data integration! Stewart Bryson already announced in his last post he would talk about the ODI 12c JEE agent and Michael Rainey started with GoldenGate 12c New Features. Who will shoot next?

[Update] Peter Scott will also present ODI 12c in his session at the Bulgarian Oracle Users Group Autumn Conference on 22nd November 2013.  Stewart and myself will also talk about it at RMOUG Training Days in February.