Rittman Mead Blog - Spark Streaming

Data Processing and Enrichment in Spark Streaming with Python and Kafka

In my previous blog post [https://www.rittmanmead.com/blog/2017/01/getting-started-with-spark-streaming-with-python-and-kafka/] I introduced Spark Streaming and how it can be used to process 'unbounded' datasets. The example I did was a very basic one - simple counts of inbound tweets and grouping by user. All very

spark

Getting Started with Spark Streaming, Python, and Kafka

Last month I wrote a series of articles [https://www.rittmanmead.com/blog/2016/12/etl-offload-with-spark-and-amazon-emr-part-5/] in which I looked at the use of Spark for performing data transformation and manipulation. This was in the context of replatforming an existing Oracle-based ETL and datawarehouse solution onto cheaper and more elastic

odi

ETL Offload with Spark and Amazon EMR - Part 5 - Summary

This is the final article in a series documenting an exercise that we undertook for a client recently. Currently running an Oracle-based datawarehouse platform, the client asked for our help in understanding what a future ETL and reporting platform could look like, given the current landscape of tools available. You

Technical

Stream Analytics and Processing with Kafka and Oracle Stream Analytics

In my previous post [http://www.rittmanmead.com/2016/07/introduction-oracle-stream-analytics/] I looked the latest release of Oracle Stream Analytics (OSA), and saw how it provided a graphical interface to "Fast Data". Users can analyse streaming data as it arrives based on conditions and rules. They can also

Big Data

Oracle OpenWorld 2015 Roundup Part 2 : Data Integration, and Big Data (in the Cloud...)

In yesterdays part one of our three-part Oracle Openworld 2015 round-up [https://www.rittmanmead.com/blog/2015/11/oracle-openworld-2015-roundup-part-1-obiee12c-and-data-visualisation-cloud-service/] , we looked at the launch of OBIEE12c just before Openworld itself, and the new Data Visualisation Cloud Service that Thomas Kurian demo’d in his mid-week keynote. In part two we’

Big Data

Combining Spark Streaming and Data Frames for Near-Real Time Log Analysis & Enrichment

A few months ago I posted an article on the blog around using Apache Spark to analyse activity on our website [https://www.rittmanmead.com/blog/2014/12/going-beyond-mapreduce-for-hadoop-etl-pt-3-introducing-apache-spark/] , using Spark to join the site activity to some reference tables for some one-off analysis. In this article I’ll be