Want to learn how Fyber built a spark pipeline on Kubernetes using Hashicorp Terraform and Consul?
In the video below, you can watch a detailed walkthrough of how Fyber built an epic-scale, stream-ingested data platform using Spark, Airflow, K8s and Spotinst Ocean to maintain costs and keep operational overheads extremely low.
Our data pipeline solution: “Spark On-Demand”
An overview of Fyber’s On-Demand Spark architecture and how it leverages Kubernetes and Spotinst Ocean, playing together over a single AirFlow workflow.
Outline of the webinar
- Introduction to Spotinst Ocean: A serverless container engine
- Introduction to HashiCorp Terraform and Consul
- Introduction to Fyber: An on-demand big data architecture using Apache Spark, Kafka, Kubernetes, AirFlow, Spotinst Ocean, and more.
- Live Demo: A working example of running a Spark workload on the Fyber platform (Kubernetes and Spotinst Ocean-based)
Click here, to view Hashicorp’s web page of the joint webinar.
Fyber’s big data architecture
Below is a meet-up recording conducted on the same topic in Israel (Hebrew):
