Azure Data Engineer Location: Reading and 2 days on site per week
Contract Type: 6-18 months
We are seeking an experienced Azure Data Engineer to join our dynamic team. As part of our data engineering division, you will work with cutting-edge tools and technologies to support data ingestion, integration, and transformation within the Azure ecosystem. The ideal candidate will have a strong background in data engineering, specifically with Azure Databricks, PySpark, and SparkSQL, as well as experience in testing and Azure DevOps.
Key Responsibilities: Data Ingestion & Transformation: Utilize PySpark and SparkSQL in Azure Databricks to ingest, process, and transform large datasets efficiently.Testing & Validation: Implement automated testing using Pytest to ensure the quality and integrity of the data pipelines.Data Integration: Work with Azure SQL Database to develop and optimize T-SQL queries and stored procedures for data integration and reporting.CI/CD Pipeline Management: Leverage Azure DevOps pipelines for automation of deployment, testing, and monitoring of data engineering workflows.Collaboration: Work closely with data analysts, data scientists, and other engineering teams to understand data needs and deliver data solutions.Optimization & Maintenance: Continuously improve data processes, ensuring high performance and reliability in the data ingestion and transformation workflows.Primary Skills: Data Ingestion & Processing: Expertise in PySpark and SparkSQL for efficient data processing and transformation within Azure Databricks.Automated Testing: Hands-on experience with Pytest for testing data pipelines and ensuring their reliability and performance.Secondary Skills: Azure SQL Database: Experience in developing T-SQL queries and stored procedures for data processing, integration, and reporting.DevOps Automation: Proficient in Azure DevOps pipelines to automate workflows, testing, and deployment of data solutions.Qualifications: Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent work experience).Proven experience in Azure Databricks, PySpark, SparkSQL, and automated testing.Familiarity with Azure SQL Database, T-SQL, and stored procedures.Solid understanding of Azure DevOps and CI/CD pipelines.Strong problem-solving skills and ability to work collaboratively in an agile environment.
#J-18808-Ljbffr