OBIEE 11.1.1.6 : MDS XML Repositories, and Integration with Source Control

So if you're a repository developer working on Oracle Business Intelligence 11g, probably the most interesting new feature announced with the release of OBIEE 11.1.1.6 yesterday was support for version control within the BI Administrator tool, and a new repository format called MDS XML. Let's take a look now into what MDS XML is, and how version control integration with the BI Administrator tool works.

Up until now, Oracle BI Repositories were stored in binary files called "RPD" files, after the .rpd filename extension. Using a single monolithic file was a design decision nQuire took back when the product was first developed, and allowed their consulting team to rapidly deploy metadata sets onto servers without having to worry about database connections, deployment procedures and so on. However, having all of the repository metadata in a single file causes us lots of issues now, in particular:

Its pretty-much impossible to extract individual objects out of the repository, and put those objects under version/source control, and
Having all metadata in a single file means we end up jumping through hoops when trying to run multi-developer projects, giving us the pleasures of the Multi-User Development Environment

So, primarily I think driven by the requirements of the Fusion Apps development team, the 11.1.1.6 release of Oracle Business Intelligence introduces a new format for saving repositories; the MDS XML format. Similar, but not quite the same as the XUDML format that's used when patching repositories, MDS XML breaks the repository into individual XML files, with each one corresponding to a first-class object such as a logical table, logical dimension, physical table, physical join and so on. MDS XML is only available when working in offline mode, so to save a repository in MDS XML format, first open it offline, and then select File > Save As > MDS XML Documents...

Once you've saved the repository in this format, the title bar for the Oracle BI Administrator tool will display the folder name, and if you look inside the filesystem folder, you'll see your repository stored as a set of subfolders containing individual XML documents.

If you then subsequently want to open the repository saved in this format, you open the Oracle BI Administration tool as normal, but this time select File > Open > MDS XML ..., which then prompts you to select a folder rather than an RPD file, and then supply the repository password. So what's the point of this? Well, the answer is in the next question that the Oracle BI Administration tool asks you - do you want to open the MDS XML repository standalone, or use source control?

Choosing the Standalone option just opens the repository as normal. If you subsequently want to upload this repository to the Oracle BI Domain using Enterprise Manager, you'll need to re-save it in the classic RPD format, and then do the upload using that file. But how does the source control option work?

Let's look at a situation where you're using Subversion as your source control system, and TortoiseSVN as your SVN client. You've set up your SVN repository using the standard folder structure, so that it looks like this (I'm using the free-as-in-beer Visual SVN Server as my SVN server and Apache server combined):

Now to configure the Oracle BI Administration tool to work with Subversion, I need to go into the Tools > Options... menu and select the Source Control tab. This presents me with a dialog with two buttons; one to Edit a configuration file, the other to create a New one. The Oracle BI Administration tool is actually designed in this release to work with any (in theory) source control tool, by mapping a set of internal commands to add files, remove files and mark files for update ito the actual commands used by the source control system you've chosen; let's press the New button to see how this works.

When I press New, I get a dialog where I can enter the actual command-line commands used by my source control tool to match the generic source control operations that the Oracle BI Administration tool will perform.

Now obviously this might sound a bit daunting, so if you're using Subversion, there's a pre-defined configuration file you can use instead. Pressing Cancel, I then press the Edit button instead, select the subversion template file, and view the options that have been added for me.

This looks OK, so I press Cancel and go back to the main menu. Let's now go through the check-out, development, save and check-in process.

I'm starting now with an empty /trunk/Repository folder in the Subversion repository. I start therefore by using TortoiseSVN to check-out this directory to somewhere on my filesystem, like this:

Now, I want to save my repository in MDS XML format into this checked out folder (in SVN terms, the working copy folder). I do so by opening the regular RPD file in offline mode, and then selecting the File > Save As > MDS XML Documents... option. I select the folder that's just been created by TortoiseSVN as my working copy folder, and then save the repository in the MDS XML format to this folder.
Then, if I want to upload this first source-controlled version of the repository to SVN, I can come out of the Oracle BI Administration tool and right-click on the folder, and select SVN Commit... from the TortoiseSVN shell menu.

As at this point, all of the folders and XML files will be unversioned, I need to add them to the working set, and then press OK.

TortoiseSVN will now upload the folders and files to the SVN repository, like this:

If I then check in the SVN repository, I'll see my uploaded BI repository in this MDS XML format.

So, what the process that you go through if you subsequently want to check this repository out again, make some changes, and then upload the changes back to the source control system? This is where the source control integration in the Oracle BI Administration tool comes in. Let's start by using TortoiseSVN to update our working copy of the SVN repository files, in case anyone else has committed and changes in the meantime.

Now, to open this repository under version control, I select File > Open > MDS XML... from the Oracle BI Adminstrator tool, enter the repository password, but then select Use Source Control rather than Standalone, and use the Browse... button to pick up the SVN configuration file I opened a while ago.

When I now go on to work with the repository, the title bar for the Oracle BI Administration tool has the message (under source control) next to the folder name where the MDS XML-format repository is saved, and it now keeps track of all the object add, delete and edit operations I perform whilst working with the repository.

Let's now make some changes to the repository, deleting that Staff (Parent Child) logical dimension, and adding a new subject area. Now, when I save the repository, I'm asked first to check global consistency, and then a new dialog comes up confirming the source control actions that this will lead to. In this case, my changes have lead to five new MDS XML files being created, six being modifed, and two being deleted; information that the Oracle BI Adminstration tool will then pass to the Subversion client, via those command-line mappings we saw in the configuration file.

Once I press the OK button, the Source Control log dialog comes up, and shows me the progress of my files being synchronized to the working set in the folder I checked-out of the SVN repository.

Then, I press Close to close the dialog, and my repository is saved.

All that's left to do now is to return to TortoiseSVN, and make the final commit back to the SVN repository (remember, all that the Oracle BI Administration tool has done use SVN commands to synchronize the list of changes you've made with SVN's local record, it's not committed the files back to the SVN repository). So, I right-click on the folder and select SVN Commit, and I can see from the Commit dialog that the SVN client knows about the files I've added, deleted and modified in the Oracle BI Administrator tool.

I then press OK and TortoiseSVN, the SVN client, updates the SVN repository with the changes.

If I then want to view the history of changes in SVN, I can use TortoiseSVN again to view the change log, where I can see the repository revisions and who's made the changes.

So, there's the basic process, and that's what MDS XML is for. I can certainly think of lots of ways this could be improved, but it's better than where were were before, and it'll be interesting to see how this work with branching and merging repositories, or trying to take versions of objects from previous check-ins, or even other repositories, and trying to merge them together using these XML files. But the thing to bear in mind is that you'll need access to a version control system such as Subversion, and a client such as TortoiseSVN, to make this all work, with the Oracle BI Administrator's role being mainly to keep track of what you've changed, and relay that information to the version control tool before you try and check-in your updated repository.