Responsibilities:
Design, deploy, and manage Databricks clusters on Google Cloud Platform (GCP), ensuring optimal performance, scalability, and reliability.
Provide expertise in Databricks architecture, configuration, and administration, following best practices and industry standards.
Collaborate with data engineering, data science, and analytics teams to understand their requirements and provide technical guidance and solutions.
Optimize Databricks workloads and processes for performance, cost-efficiency, and scalability.
Monitor and troubleshoot Databricks clusters, identifying and resolving issues proactively.
Implement security measures and ensure compliance with data governance and privacy regulations.
Design and implement data pipelines and workflows, integrating Databricks with other GCP services and tools.
Stay up to date with the latest advancements in Databricks, GCP, and data engineering technologies, and evaluate their applicability to our environment.
Mentor junior team members, providing guidance and support in Databricks administration and architecture.
Requirements:
Minimum of 5 years of experience as a Databricks Admin/Architect, with a strong focus on Google Cloud Platform (GCP).
Extensive experience in designing, deploying, and managing Databricks clusters, notebooks, jobs, and libraries.
Deep understanding of Databricks architecture, configuration, and performance optimization techniques.
Strong knowledge of GCP services, including Compute Engine, BigQuery, Cloud Storage, Dataflow, and Pub/Sub.
Proficiency in scripting languages like Python or Scala for automation, terraform, and data engineering tasks.
Experience with data engineering principles and tools, such as ETL, data pipelines, and data modeling.
Strong troubleshooting and problem-solving skills, with the ability to identify and resolve complex issues in a timely manner.
Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams.
GCP certifications, such as Google Cloud Certified - Data Engineer or Google Cloud Certified - Professional Cloud Architect, are highly desirable.