Azure Data Engineer-Databricks /ADF

Are you a seasoned Data Analyst with expert-level skills in Azure and Databricks? Apply Now!

Working with one of our top financial clients, this role calls for an Azure Data Engineer-Databricks /ADF. This position involves working within a transformative data environment focused on advancing Databricks functionality and supporting large-scale investment transaction processes for institutional clients.

Pay rate range (CAD): $86.96/hr – $104.35/hr

Responsibilities:

Develop and maintain pipelines using Azure Data Factory (ADF), Databricks, and Azure Functions (C#)
Build and optimize Spark/Pandas notebooks for data processing and analysis
Configure and manage Unity Catalog for data governance and security in Databricks
Migrate Databricks workspace from hive metastore to Unity Catalog, handling permissions and security configurations
Design delta tables with proper schema, properties, and security controls
Implement Delta Live Tables and related features to enhance data workflows (nice to have)
Manage DevOps and deployment pipelines using Azure DevOps and GitHub Enterprise for both ADF and Databricks
Work with Azure Entra ID for SSO and RBAC setup
Deploy and manage Azure PaaS resources such as Azure SQL, ADF/SHIR, ADLS, and Synapse SQL databases
Collaborate with a team of Data Engineers, Business Analysts, and Project Managers to meet project goals
Work independently and lead small technical team sessions to troubleshoot and resolve issues
Ensure compliance with data governance and security standards for highly regulated environments
Desired Skill-Set:

7+ years of experience with ADF, Databricks, Azure Functions (C#)
Extensive development experience with Python/notebooks on Databricks/Spark/Pandas
Expert knowledge of Unity Catalog and data governance setup
Experience loading datasets from various sources using ADF with Medallion architecture
Proven track record migrating Databricks from hive metastore to Unity Catalog and managing permissions
Ability to design delta tables with security considerations
Strong DevOps skills using Azure DevOps and GitHub Enterprise
In-depth experience with Azure Entra ID for SSO/RBAC
Proficiency working with Azure SQL PaaS, ADLS, Synapse, and other Azure services
Familiarity with working in a highly regulated environment
Ability to work independently, lead a team, and solve complex technical problems
Financial services industry experience (asset)
Excellent communication skills
Nice to Have:

Previous experience working with financial institutions
Knowledge of Delta Live Tables
Experience with Azure Synapse Analytics
Familiarity with Windows environments in regulated industries