DevOps in OAC: Scripting Oracle Cloud Instance Management with PSM Cli

Francesco Tisiot -

This summer we unselfish Italians decided to not participate to the World Cup to give another country the opportunity to win (good luck with that England!). This decision, which I strongly support, gives me lot of time for blogging!

As already written, two weeks ago while in Orlando for Kscope18, I presented a session about DevOps and OBIEE focusing on how to properly source control, promote and test for regression any component of the infrastructure.

Development Isolation

One key aspect of DevOps is providing the Development Isolation: a way of allowing multiple development streams to work independently and merging the outcome of the process into the main environment only after this has been tested and validated. This is needed to avoid the standard situation where code promotions are blocked due to different working streams not being in sync: forcing a team to postpone a code release just because another team doesn't have the UAT OK is just an example of non-isolated development platforms.

We have been discussing development isolation topic in the past focusing mainly on concurrent repository development and how to integrate it with versioning tools like Git and SVN. The concurrent online editing option is not viable since multiple developers are modifying the same artifact (RPD) without a way of testing for regression the changes or to verifying that what has been done is correct before merging the changes in the RPD.
Alternative solutions of using MUDE (default multi-user development method provided by the Admintool) or pure offline RPD work encounter the same problems defined above: no feature or regression testing available before merging the RPD in the main development environment.

Different RPD development techniques solve only partially the problem: almost any OAC/OBIEE development consist at least in both RPD and catalog work (creation of analysis/dashboards/VA projects) we need an approach which provides Development Isolation at both levels. The solution, in order to properly build a DevOps framework around OAC/OBIEE, it's to provide isolated feature-related full OBIEE instances where the RPD can be edited in online mode, the catalog work can be done independently, and the overall result can be tested and validated before being merged into the common development environment.

Feature-Related Instances

The feature instances, as described above, need to be full OAC/OBIEE development instances where only a feature (or a small set) is worked at the time in order to give the agility to developers to release the code as soon as it's ready and tested. In the on-premises world this can "easily" be achieved by providing a number of dedicated Virtual Machines or, more in line with the recent trends, an automated instance provisioning with Docker using a template image like the one built by our previous colleague Gianni Ceresa.

However, when we think about Oracle Analytics Cloud (OAC), we seem to have two problems:

  • There is a cost associated with every instance, thus minimizing the number of instances and the uptime is necessary
  • The OAC provisioning interface is point and click, thus automating the instance management seems impossible

The overall OAC instance cost can be mitigated by the Bring Your Own License (BYOL) licensing method which allows customers to migrate on-premises licenses to the cloud and have discounted prices on the hourly/monthly instance cost (more details here). However, since the target is to minimize the cost thus the # of instances and the uptime, we need to find a way to do so that doesn't rely on a human and a point and click interface. Luckily the PaaS Service Manager Command Line Interface (PSM Cli) allows us to solve this problem by providing a scriptable way of creating, starting and stopping instances.

PaaS Service Manager Command Line Interface

PSMCLI is a command line interface acting as a wrapper over the PaaS REST APIs. Its usage is not limited to OAC, the same interface can be used to create and manage instances of the Oracle's Database Cloud Service or Java Cloud Services amongst the others.
When talking about OAC please keep in mind that, as of now, PSM Cli works only with the non-autonomous version but I believe the Autonomous support will be added soon.

Installing and Configuring PSM Cli

PSMCLI has two prerequisites before it can be installed:

  • cURL - a command line utility to transfer data with URLs
  • Python 3.3 or later

Once both prerequisites are installed PSM can easily be downloaded with the following cURL call

curl -X GET -u <USER>:<PWD> -H X-ID-TENANT-NAME:<IDENTITY_DOMAIN> https://<REST_SERVER>/paas/core/api/v1.1/cli/<IDENTITY_DOMAIN>/client -o psmcli.zip

Where

  • <USER> and <PWD> are the credentials
  • <IDENTITY_DOMAIN> is the Identity Domain ID specified during the account creation
  • <REST_SERVER> is the REST API server name which is:
  • psm.us.oraclecloud.com if you are using a US datacenter
  • psm.aucom.oraclecloud.com if you are in the AuCom region
  • psm.europe.oraclecloud.com otherwise

Next step is to install PSM as a Python package with

pip3 install -U psmcli.zip

After the installation is time for configuration

psm setup

The configuration command will request the following information:

  • Oracle Cloud Username and Password
  • Identity Domain
  • Region, this need to be set to
  • emea if the REST_SERVER mentioned above contains emea
  • aucom if the REST_SERVER mentioned above contains aucom
  • us otherwise
  • Output format: the choice is between short, json and html
  • OAuth: the communication between the CLI and the REST API can use basic authentication (flag n) or OAuth (flag y). If OAuth is chosen then ClientID, Secret and Access Token need to be specified

A JSON profile file can also be used to provide the same information mentioned above. The structure of the file is the following

{ 
    "username":"<USER>",
    "password":"<PASSWORD>",
    "identityDomain":"<IDENTITY_DOMAIN>",
    "region":"<REGION>",
    "outputFormat":"<OUTPUT_FORMAT>",
    "oAuth":{ 
        "clientId":"",
        "clientSecret":"",
        "accessTokenServer":""
    }
}

If the profile is stored in a file profile.json the PSM configuration can be achieved by just executing

psm setup -c profile.json

One quick note: the identity domain Id, shown in the Oracle Cloud header, isn't working if it's not the original name (name at the time of the creation).

In order to get the correct identity domain Id to use, check in an Oracle Cloud instance already created (e.g. a database one) and check the Details, you'll see the original identity domain listed there (credits to Pieter Van Puymbroeck).

Working With PSM Cli

Once the PSM has been correctly configured it's time to start checking what options are available, for a detailed list of the options check PSM documentation.

The PSM commands are product related, so each command is in the form:

psm <product> <command> <parameters>

Where

  • product is the Oracle cloud product e.g. dbcs, analytics, BigDataAppliance, for a complete list use psm help
  • command is the action to be executed against the product e.g. services, stop, start, create-service
  • parameters is the list of parameters to pass depending on the command executed

The first step is to check what instances I already created, I can do so for the database by executing

psm dbcs services

which, as expected, will list all my active instances

I can then start and stop it using:

psm dbcs start/stop/restart -s <INSTANCE_NAME>

Which in my example provides the Id of the Job assigned to the stop operation.

When I check the status via the service command I get Maintenance like in the web UI.

The same applies to the start and restart operation. Please keep in mind that all the calls are asynchronous -> the command will call the related REST API and then return the associated Job ID without waiting for the command to be finished. The status of a job can be checked with:

psm dbcs operation-status -j <JOB_ID>

The same operations described above are available on OAC with the same commands by simply changing the product from dbcs to analytics like:

psm analytics start/stop/restart -s <INSTANCE_NAME>

On top of the basic operation, PSM Cli allows also the following:

  • Service Instance: start/stop/restart, instance creation-deletion
  • Access Control: lists, creates, deletes, enables and disables access rules for a service.
  • Scaling: changes the computer shape of an instance and allows scaling up/down.
  • Storage: extends the storage associated to OAC
  • Backup Configuration: updates/shows the backup configurations
  • Backups: lists, creates, deletes backups of the instance
  • Restore: restores a backup giving detailed information about it and the history of Restores
  • Patches: allows patching, rollbacking, doing pre-checks, and retrieving patching history

Creating an OAC Instance

So far we discussed the maintenance on already created instances with start/stop/restart commands, but PSM Cli allows also the creation of an instance via command line. The call is pretty simple:

psm analytics create-service -c <CONFIG_FILE> -of <OUTPUT_FORMAT>

Where

  • CONFIG_FILE: is the file defining all OAC instance configurations
  • OUTPUT_FORMAT: is the desired output format between short, json and html

The question now is:

How do I create a Config File for OAC?

The documentation doesn't provide any help on this, but we can use the same approach as for on-premises OBIEE and response file: create the first instance with the Web-UI, save the payload for future use and change parameters when necessary.

On the Confirm screen, there is the option to Download the REST payload in JSON format

With the resulting json Config File being

{
  "edition": "<EDITION>",
  "vmPublicKeyText": "<SSH_TOKEN>",
  "enableNotification": "true",
  "notificationEmail": "<EMAIL>",
  "serviceVersion": "<VERSION>",
  "isBYOL": "false",
  "components": {
    "BI": {
      "adminUserPassword": "<ADMINPWD>",
      "adminUserName": "<ADMINUSER>",
      "analyticsStoragePassword": "<PWD>",
      "shape": "oc3",
      "createAnalyticsStorageContainer": "true",
      "profile_essbase": "false",
      "dbcsPassword": "<DBCSPWD>",
      "totalAnalyticsStorage": "280.0",
      "profile_bi": "true",
      "profile_dv_forced": "true",
      "analyticsStorageUser": "<EMAIL>",
      "dbcsUserName": "<DBUSER>",
      "dbcsPDBName": "<PDBNAME>",
      "dbcsName": "<DBCSNAME>",
      "idcs_enabled": "false",
      "analyticsStorageContainerURL": "<STORAGEURL>",
      "publicStorageEnabled": "false",
      "usableAnalyticsStorage": "180"
    }
  },
  "serviceLevel": "PAAS",
  "meteringFrequency": "HOURLY",
  "subscriptionId": "<SUBSCRIPTIONID>",
  "serviceName": "<SERVICENAME>"
}

This file can be stored and the parameters changed as necessary to create new OAC instances with the command:

psm analytics create-service -c <JSON_PAYLOAD_FILE> -of short/json/html

As shown previously, the result of the call is a Job Id that can be monitored with

psm analytics operation-status -j <JOB_ID>

Once the Job is finished successfully, the OAC instance is ready to be used. If at a certain point, the OAC instance is not needed anymore, it can be deleted via:

psm analytics delete-service -s <SERVICE_NAME> -n <DBA_NAME> -p <DBA_PWD>

Where

  • SERVICE_NAME is the OAC instance name
  • DBA_NAME and DBA_PWD are the DBA credentials where OAC schemas are residing

Summary

Worried about providing development isolation in OAC while keeping the costs down? Not anymore! With PSM Cli you now have a way of creating instances on demand, start/stop, up/down scaling with a command line tool easily integrable with automation tools like Jenkins.

Create an OAC instances automatically only when features need to be developed or tested, stop&start the instances based on your workforce timetables, take the benefit of the cloud minimizing the cost associated to it by using PSM Cli!

One last note; for a full DevOps OAC implementation, PSM Cli is not sufficient: tasks like automated regression testing, code versioning, and promotion can't be managed directly with PSM Cli but require usage of external toolsets like Rittman Mead BI Developer Toolkit. If you are interested in a full DevOps implementation on OAC and understanding the details on how PSM Cli can be used in conjunction with Rittman Mead BI Developer Toolkit don't hesitate to contact us!