Our client is a British multinational mining company with headquarters in London. They are the world’s largest producer of platinum, as well as being a major producer of diamonds and other precious metals.
An opportunity has arisen for an experienced Data Engineer to join their team on contract basis. Please note that this role will be initially remote working with some travel to London once office working resumes.
Data Engineer Lead Responsibilities:
Develop, test, operate and support the end-to-end batch, near real time data engineering solution
Be able to develop quality Python/PySpark and adopt/implement software development best practices to ensure high quality standards are met
Performs thorough code reviews of fellow developers and ensure quality code
Use DevOps knowledge, cloud and data engineering expertise to operate, monitor and troubleshoot the pipelines / code bases
Effectively test and adopt testing frameworks to embed and codify tests
Take technical ownership for the implementation of the DE solution at an allocated mining site
Clearly communicate and explain concepts to senior stakeholders and key teams during sprint reviews and other programme meetings
Have a clear understanding of agile best practices and proactively collaborate and support the team if needed during the whole sprint cycle
Document technical implementation details and artefacts created as part of the process
Python, PySpark, version control (Git), performance engineering, Azure cloud analytics services (ADF, Databricks, SQLDB, Event Hub
Cloud – Databricks , Microsoft AzureDevOps, DevOps, Data Modelling, Test Automation