Janis Rumnieks - Rittman Mead

Custom Preprocessing Transformers in Snowflake ML

Introduction: Input Data Preprocessing with Snowflake ML This blog post provides a code sample demonstrating how to implement a custom data preprocessing transformer for Snowflake ML, similar to creating a custom step in a Scikit-learn pipeline. But wait, can't we just ask the AI to write the code

Data Lineage

Data Lineage Analysis with Python and NetworkX

Use Network Graphs for Data Analysis Python is one of the most capable and flexible data analysis tools out there. The data to be analysed comes in different formats and with different requirements for analysis. Some data is best represented and analysed as a network graph. A good example of

Tip Tuesday

Tip Tuesday | Build and Analyse Network Graphs in Minutes with Python's NetworkX

Python's NetworkX library offers a simple yet powerful entry point into the world of network graph construction and analysis. Let us build a directional graph with three types of nodes, similar to entities in a relational database: * Author, * Book, * Reader. import networkx as nx G = nx.DiGraph() authors

Tip Tuesday

Tip Tuesday | Quick CSV Data Analysis with Python and Pandas

If you have a CSV file and quickly need to do a bit of analysis on it, you do not always have to reach for the big guns like loading it into a relational database table or using Apache Spark. The Pandas Python library allows for quick analysis without the

oci

OCI Data Flow: ETL with Apache Spark in the Cloud

About OCI Data Flow Oracle Cloud Infrastructure (OCI) Data Flow is a managed service for running Apache Spark applications, promising simplified infrastructure management, development, and deployment. Initial impressions suggest it delivers on this promise, showcasing an elegant approach to certain tasks. To assess OCI Data Flow's capabilities, a

OpenAI

OpenAI Gives Vision to Oracle DB, Part 2: Analysing Image Data with GPT-4o

The Context In my last blog post, we successfully loaded image data into an Oracle database table. Now, we will focus on building an Oracle function to connect this data with the OpenAI GPT-4o model for analysis. We will adapt the framework from a previous blog post where we used

OpenAI

OpenAI Gives Vision to Oracle DB, Part 1: Preparing the Image Data

Image Data in Oracle DB OpenAI's newly released GPT-4o model, with its ground-breaking multimodal processing capabilities, has generated significant interest in the AI community. Building on my previous exploration of integrating OpenAI's text-based models with Oracle database queries, this series of articles will delve into utilizing

Tip Tuesday

Tip Tuesday | Use Python For Querying JSON Files

JSON files are prevalent in the IT industry, commonly used in web service API data exchange. Python offers a streamlined alternative to manual analysis or database importation, often proving as efficient and concise as SQL queries. Let us have a look at an example of a JSON file that consists

OpenAI

Call OpenAI from Oracle Query

Introduction The Old meets the New ChatGPT's impact on our daily lives has been surprisingly significant in the short time since its release. Reshaping work practices, OpenAI's ChatGPT is expected to drive further change, even disruption in the coming years. Meanwhile, relational databases, the workhorses of

odi

Migration from Informatica PowerCenter to ODI: accelerated with Sql2Odi

Streamlining ETL Migrations with Sql2Odi I have previously blogged about Rittman Mead's Sql2Odi tool, which translates SQL statements (SELECT or WITH) into Oracle Data Integrator (ODI) Mappings. After years of use, we recently put Sql2Odi to the test during a large ETL migration project. Our goal was to

Using ODI Java SDK with Groovy and Scala

Scripting for ODI Repository If you need to create a quick script that manipulates ODI Repository objects, Groovy is the programming language of choice. You can run Groovy scripts from within ODI Studio, which offers two major advantages: * Direct access to the odiInstance object: This object represents the current ODI

odi

Assessing the Performance of Looping ODI Package Tasks

Our SQL query extracts data from ODI Repository log tables and can be used to find performance bottlenecks in ODI Load Plans that execute Packages with loops by identifying the most time-consuming Package tasks. ODI Packages often use loops, especially to load CSV files or query APIs that have limits

Oracle Database

Oracle Database Performance Testing the Easy Way

We have developed a tool for testing Oracle database performance. The tool connects to a database, executes a set of SELECT statements, and records their execution times. It can also execute queries in parallel by using a connection pool of multiple connections, thus simulating a real-life workload on a database.

Odi2Package tool - reverse-engineers and generates ODI Packages

Introducing new reverse-engineering capability In my previous blog post, I introduced our new Odi2Package tool that allows you to auto-generate ODI Packages. Our ODI development teams were quick to adopt the tool for their project needs. Soon, however, we noticed a significant weakness: the manual creation of templates for new

Odi2Package tool - auto-generates ODI Packages

Introducing Odi2Package We have added a new tool to our arsenal - Odi2Package. It auto-generates ODI Packages based on a template. In combination with our Sql2Odi tool, we are now one step closer to the utopian dream of generating ETL straight from Low-level Design. Generating DDL statements (CREATE TABLE, CREATE

Sql2Odi - best practices for auto-generating your ETL content

I blogged a while ago about our Sql2Odi tool that converts SELECT (and WITH as well) statements into Oracle ODI Mappings. (Blog posts 1, 2 and 3.) Now that the tool has been around for close to 2 years, let us reflect on lessons learnt and best practices adopted. Let

OAC Semantic Modeler and Version Control with Git

OAC Semantic Modeler features native version control with Git

OAC Semantic Modeler and the SMML Language

There are a few reasons to like the Oracle Analytics Cloud (OAC) paired with the new Semantic Modeler: * It works in the cloud. I mean, it actually works. (Check my blog post [https://www.rittmanmead.com/blog/2022/07/giving-oac-semantic-modeller-a-try/] for an overview of Semantic Modeler.) * It has a native,

OAC

Giving OAC Semantic Modeler a Try

Kids just love it when old folk go on about good old days. The particular good old days I have in mind is when Oracle CRM On Demand was released. Competition in the Cloud CRM space was tough but the Oracle product had one trick up its sleeve: its data

Data Lineage

Tracing Data Lineage from Source Column to End User

A while ago I blogged about our Lineage Tool [https://www.rittmanmead.com/blog/2022/04/introducing-rittman-mead-lineage-tool/] , which back then was limited to diving into the ODI Work Repository and emerging from it with source-to-target data flow diagrams. The task at hand was simple and the design straightforward: show Source

A quick way of generating Informatica PowerCenter Mappings from a template

Generating Informatica PowerCenter Content - the Options In our blogs we have discussed the options for Oracle Data Integrator (ODI) content generation here [https://www.rittmanmead.com/blog/2021/03/sql2odi-part-one-convert-a-simple-select-statement-to-an-odi-mapping/] and here [https://www.rittmanmead.com/blog/2017/03/accelerating-your-odi-implementation-rittmanmead-style/] . Our go-to method is to use the ODI Java

ODI Lineage

Introducing Rittman Mead Lineage Toolkit

Introduction Rittman Mead Lineage Toolkit analyses ODI Mappings and produces neat visualisations of source-to-target mappings. The Lineage Toolkit is quick to set up and easy to run. Rittman Mead Lineage Tool: from ODI Mappings to Mapping Lineage VisualisationThe Lineage Toolkit can be used to reverse-engineer legacy ELT logic as the

Sql2Odi now supports the WITH statement

The Rittman Mead's Sql2Odi tool that converts SQL SELECT statements to ODI Mappings, now supports the SQL WITH statement as well. (For an overview of our tool's capabilities, please refer to our blog posts here [https://www.rittmanmead.com/blog/2021/03/sql2odi-part-one-convert-a-simple-select-statement-to-an-odi-mapping/] and here [https:

SQL Query to ODI Mapping Translation

Sql2Odi, Part Two: Translate a complex SELECT statement into an ODI Mapping

Discusses translation of complex SQL SELECT statements (queries) into ODI Mappings. Shows Sql2Odi in action.

ODI Content Generation with Groovy

Sql2Odi, Part One: Translate a simple SELECT statement into an ODI Mapping

Discuss Groovy scripting for Oracle Data Integrator (ODI) to generate ODI content. Discusses translation of simple SQL SELECT statements (queries) into ODI Mappings.