Data Engineer

Hey everyone, i have open one urgent position hashtag#Azure hashtag#DataEngineer who strong in hashtag#Spark hashtag#Databricks and at least 6+ year of experiance

Position – Data Engineer
Location – hashtag#Mississauga/ hashtag#Montreal
Contract
Canada
Job Summary:
We are seeking highly skilled Azure Data Engineer with strong expertise in Databricks to join our data team. The ideal candidate will design, implement and optimize large-scale data pipeline, ensuring scalability, reliability and performance. This role involves working closely with multiple teams and business stakeholders to deliver cutting-edge data solutions.
Key Responsibilities:
Data Pipeline Development:
·
o   Build and maintain scalable ETL/ELT pipelines using Databricks.
o   Leverage PySpark/Spark and SQL to transform and process large datasets.
o   Integrate data from multiple sources including Azure Blob Storage, ADLS and other relational/non-relational systems.
Collaboration & Analysis:
·
o   Work Closely with multiple teams to prepare data for dashboard and BI Tools.
o   Collaborate with cross-functional teams to understand business requirements and deliver tailored data solutions.
Performance & Optimization:
·
o   Optimize Databricks workloads for cost efficiency and performance.
o   Monitor and troubleshoot data pipelines to ensure reliability and accuracy.
Governance & Security:
·
o   Implement and manage data security, access controls and governance standards using Unity Catalog.
o   Ensure compliance with organizational and regulatory data policies.
Deployment:
·
o   Leverage Databricks Asset Bundles for seamless deployment of Databricks jobs, notebooks and configurations across environments.
o   Manage version control for Databricks artifacts and collaborate with team to maintain development best practices.
Technical Skills:
·        Strong expertise in Databricks (Delta Lake, Unity Catalog, Lakehouse Architecture, Table Triggers, Delta Live Pipelines, Databricks Runtime etc.)
·        Proficiency in Azure Cloud Services.
·        Solid Understanding of Spark and PySpark for big data processing.
·        Experience in relational databases.
·        Knowledge on Databricks Asset Bundles and GitLab.
Preferred Experience:
·        Familiarity with Databricks Runtimes and advanced configurations.
·        Knowledge of streaming frameworks like Spark Streaming.
·        Experience in developing real-time data solutions.
Certifications:
Azure Data Engineer Associate or Databricks certified Data Engineer Associate certification. (Optional