In simple terms, a data brick on Google Cloud is a Databricks environment that runs on Kubernetes and is hosted on the cloud. It also provides Google Cloud built-in capabilities by this hosting, like Cloud Storage, BigQuery, Google cloud identity etc.
Why Databricks on google cloud?
Apart from associated with and hassle-free integration with all technologies that Google offers under its umbrella, it is increasingly enabling analytics, and ML-AI use cases in Data engineering and allied fields.
- Analytics for graphs and infographic processing for dashboards.
- Deep learning on unstructured data for machine learning, natural language processing, digital image processing etc.,
- Monitoring and safekeeping of data lakes
- Analysis of low latency, high-frequency IoT data in real-time, live streams and dynamic data pools.
In this post, we will look at Databricks concepts that can be used in the Self-paced Course for Databricks Certified Associate Developer for Apache Spark.
While a few concepts are general to Databricks on the whole, specific concepts depending on the application of Databricks in Data Science, Machine learning or Data engineering may differ.
General concepts
Let us look at universal concepts of accounts and workspaces in Databricks.
Accounts and workspaces
Accounts:
In Databricks, an account is treated as a single entity or subscription for accounting and billing. It might however contain multiple workspaces.
Workspaces:
In Databricks workspace could mean one of the following two
- A Databricks workspace on the cloud is an environment that functions like a dashboard that you can use to access all assets on data bricks. You can classify data bricks as per your convenience into multiple workspaces.
- If you come across a workspace browser or any persona-based environments, chances are you are dealing with data brick workspaces in data science and engineering.
Databricks concepts in Data Science & Engineering
Data science and data engineering is the most used data bricks environment nowadays. With data science and data analytics on the rise across all industries, data bricks in data science and data engineering is the go-to for collaboration and data sharing among data scientists, analysts and engineers.
Data Science & Engineering interface
Databricks uses UI and APIs to access assets. Let us take a detailed look at these access mechanisms.
API
Computation management in Data Science & Engineering
To run computations on data in data bricks, a crucial part of Databricks Certified Associate Developer for Apache Spark – Preparation Toolkit is understanding computational concepts.
Cluster
Clusters are resources on which jobs and notebooks are run. In data bricks, we have two types of clusters – job clusters and all-purpose clusters.
Runtime in databricks:
Runtime for machine learning has a built-in environment for data science with several libraries specific to fields like biomedical or pharma.
Apache Spark has components and updates that aid Big Data analytics.
Workflows in data bricks:
These are in-built frameworks that help develop and run data processing pipelines. There are two types of workflows:
Live tables: Delta lives tables help build reliable processing pipelines that are easy to test and maintain.
To conclude…
Databricks on google cloud is popular as it offers a unified platform for business intelligence and data analytics using Ai/ML, adobe spark and other data engineering tools. A Roadmap to becoming Databricks Certified Associate Developer for Apache Spark must include this fast-growing, user-friendly aspect.
