As a Data Engineer II , you will build, test, monitor, validate, and support data pipelines using Azure Data Factory, Databricks, PySpark, Spark SQL, SQL, and Python. The role requires strong hands-on engineering experience and significant Python expertise to develop, automate, validate, and operationalize data solutions supporting reporting, analytics, and operational decision-making.
This role will initially focus on validation activities, including source-to-target validation, data profiling, reconciliation, anomaly detection, test automation, defect analysis, and data quality controls. Over time, the role is expected to contribute more broadly to pipeline development, optimization, deployment, and operational support within Azure and Databricks environments.
Working with minimal supervision, you will perform intermediate to complex data engineering, data preparation, validation, evaluation, deployment, and operational support activities. You will partner with engineering, analytics, business, and operations teams to deliver reliable, analytics-ready data assets while ensuring data quality, performance, scalability, security, and compliance requirements are met. You may lead small project teams and contribute to engineering standards, reusable validation frameworks, and platform improvements.
Key Responsibilities

• Perform data validation activities for enterprise data assets, including source-to-target validation, reconciliation, profiling, anomaly detection, and defect analysis.

• Develop automated validation, testing, and data quality controls using Python, PySpark, Spark SQL, SQL, and related frameworks to ensure the accuracy, completeness, consistency, and timeliness of enterprise data.
• Build, enhance, and support ETL/ELT pipelines using Azure Data Factory, Databricks, Python, PySpark, and Spark SQL, with an initial focus on validation and quality engineering use cases.
• Troubleshoot data issues, analyze root causes, document findings, and partner with Business, Technology, Operations, and Data & Analytics teams to resolve defects and improve data reliability.
• Design, develop, and optimize scalable data processing and validation solutions that support reporting, analytics, and operational decision-making.
• Implement and support Delta Lake and Lakehouse architecture patterns to enable reliable, scalable, and efficient data processing.
• Ensure data processing and validation solutions comply with established security, quality, and operational standards.
• Contribute to reusable validation frameworks, engineering standards, operational playbooks, and continuous improvement initiatives across the data platform.

Required Qualifications

• Bachelor’s or master’s degree in computer science, Engineering, Information Systems, Mathematics, Statistics, Operations Research, or a related quantitative field, or equivalent experience.
• 3-5 years of experience in data engineering, analytics engineering, data validation engineering, data platform development, or related disciplines.
• Strong hands-on experience developing data engineering and validation solutions using Python, PySpark, Spark SQL, and/or SQL.
• Experience building, testing, validating, and supporting ETL/ELT pipelines using Azure Data Factory, Databricks, and Delta Lake architectures.
• Experience developing, troubleshooting, and optimizing scalable cloud-based data solutions in Azure, including data quality, reconciliation, and validation activities.

• Are you interested in this position?

   

  Apply by clicking on the “Apply Now” button below!

   

  #AlbionarcJobs#FintechJobs

  #AsiaJobs#MiddleEastCareers

  #TechTalent#FintechRecruitment

  #FinanceOpportunities#