Scala Spark Streaming and Kafka, Part 3 - Analysing Data in Scala and Spark In my first two blog posts of the Spark Streaming and Kafka series - Part 1 - Creating a New Kafka Connector [https://www.rittmanmead.com/blog/2019/02/kafka/] and Part 2 - Configuring a Kafka Connector [https://www.rittmanmead.com/blog/2019/02/spark-streaming-and-kafka/] - I showed how
Spark Streaming Data Processing and Enrichment in Spark Streaming with Python and Kafka In my previous blog post [https://www.rittmanmead.com/blog/2017/01/getting-started-with-spark-streaming-with-python-and-kafka/] I introduced Spark Streaming and how it can be used to process 'unbounded' datasets. The example I did was a very basic one - simple counts of inbound tweets and grouping by user. All very
spark Getting Started with Spark Streaming, Python, and Kafka Last month I wrote a series of articles [https://www.rittmanmead.com/blog/2016/12/etl-offload-with-spark-and-amazon-emr-part-5/] in which I looked at the use of Spark for performing data transformation and manipulation. This was in the context of replatforming an existing Oracle-based ETL and datawarehouse solution onto cheaper and more elastic
odi ETL Offload with Spark and Amazon EMR - Part 5 - Summary This is the final article in a series documenting an exercise that we undertook for a client recently. Currently running an Oracle-based datawarehouse platform, the client asked for our help in understanding what a future ETL and reporting platform could look like, given the current landscape of tools available. You
emr ETL Offload with Spark and Amazon EMR - Part 3 - Running pySpark on EMR In the previous articles (here [https://www.rittmanmead.com/blog/2016/12/etl-offload-with-spark-and-amazon-emr-part-1], and here [https://www.rittmanmead.com/blog/2016/12/etl-offload-with-spark-and-amazon-emr-part-2-code-development-with-notebooks-and-docker/] ) I gave the background to a project we did for a client, exploring the benefits of Spark-based ETL processing running on Amazon's Elastic Map Reduce
spark ETL Offload with Spark and Amazon EMR - Part 2 - Code development with Notebooks and Docker In the previous article [https://www.rittmanmead.com/blog/2016/12/etl-offload-with-spark-and-amazon-emr-part-1/] I gave the background to a project we did for a client, exploring the benefits of Spark-based ETL processing running on Amazon's Elastic Map Reduce (EMR) Hadoop platform. The proof of concept we ran was on
obiee ETL Offload with Spark and Amazon EMR - Part 1 - Introduction We recently undertook a two-week Proof of Concept exercise for a client, evaluating whether their existing ETL processing could be done faster and more cheaply using Spark. They were also interested in whether something like Redshift [http://docs.aws.amazon.com/redshift/latest/mgmt/welcome.html] would provide a suitable
Technical Stream Analytics and Processing with Kafka and Oracle Stream Analytics In my previous post [http://www.rittmanmead.com/2016/07/introduction-oracle-stream-analytics/] I looked the latest release of Oracle Stream Analytics (OSA), and saw how it provided a graphical interface to "Fast Data". Users can analyse streaming data as it arrives based on conditions and rules. They can also
Big Data Using R with Jupyter Notebooks and Oracle Big Data Discovery Oracle's Big Data Discovery [https://www.oracle.com/big-data/big-data-discovery/index.html] encompasses a good amount of exploration, transformation, and visualisation capabilities for datasets residing in your organisation’s data reservoir. Even with this though, there may come a time when your data scientists want to unleash their
Big Data Oracle OpenWorld 2015 Roundup Part 2 : Data Integration, and Big Data (in the Cloud...) In yesterdays part one of our three-part Oracle Openworld 2015 round-up [https://www.rittmanmead.com/blog/2015/11/oracle-openworld-2015-roundup-part-1-obiee12c-and-data-visualisation-cloud-service/] , we looked at the launch of OBIEE12c just before Openworld itself, and the new Data Visualisation Cloud Service that Thomas Kurian demo’d in his mid-week keynote. In part two we’
Big Data Combining Spark Streaming and Data Frames for Near-Real Time Log Analysis & Enrichment A few months ago I posted an article on the blog around using Apache Spark to analyse activity on our website [https://www.rittmanmead.com/blog/2014/12/going-beyond-mapreduce-for-hadoop-etl-pt-3-introducing-apache-spark/] , using Spark to join the site activity to some reference tables for some one-off analysis. In this article I’ll be
Big Data Oracle Data Integrator Enterprise Edition Advanced Big Data Option Part 1- Overview and 12.1.3.0.1 install Oracle recently announced Oracle Data Integrator Enterprise Edition Advanced Big Data Options [https://www.oracle.com/middleware/data-integration/enterprise-edition-big-data/index.html] as part of the new 12.1.3.0.1 release of ODI. It includes various great new functionalities to work on an Hadoop ecosystem. Let's have
Technical Introducing Oracle Big Data Discovery Part 1: "The Visual Face of Hadoop" Oracle Big Data Discovery [https://www.oracle.com/bigdata/big-data-discovery/index.html] was released last week, the latest addition to Oracle's big data tools suite that includes Oracle Big Data SQL [http://www.rittmanmead.com/tag/big-data-sql/], ODI and it’s Hadoop capabilities [https://www.rittmanmead.com/blog/
Technical OBIEE and ODI on Hadoop : Next-Generation Initiatives To Improve Hive Performance The other week I posted a three-part series (part 1 [https://www.rittmanmead.com/blog/2014/12/going-beyond-mapreduce-for-hadoop-etl-pt-1-why-mapreduce-is-only-for-batch-processing/] , part 2 [https://www.rittmanmead.com/blog/2014/12/going-beyond-mapreduce-for-hadoop-etl-pt-2-introducing-apache-yarn-and-apache-tez/] and part 3 [https://www.rittmanmead.com/blog/2014/12/going-beyond-mapreduce-for-hadoop-etl-pt-3-introducing-apache-spark/] ) on going beyond MapReduce for Hadoop-based ETL, where I l
Technical Going Beyond MapReduce for Hadoop ETL Pt.1 : Why MapReduce Is Only for Batch Processing Over the previous few months I’ve been looking at the various ways you can load data into Hadoop, process it and then report on it using Oracle tools [https://www.rittmanmead.com/blog/2013/11/why-odi-dw-and-obiee-developers-should-be-interested-in-hadoop/] . We’ve looked at Apache Hive and how it provides a SQL layer
Technical Exploring Apache Spark on the New BigDataLite 3.0 VM The latest version of Oracle’s BigDataLite VirtualBox VM [http://www.oracle.com/technetwork/database/bigdata-appliance/oracle-bigdatalite-2104726.html] went up on OTN last week, and amongst other things it includes the latest CDH5.0 version of Cloudera Distribution including Hadoop [http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Release-Notes/
