The role will be expected to build an excellent knowledge of my client’s products with a deep understanding of how data is managed, transformed and used within the company. The data engineer will work within a product team to guide and advise on best practices around data storage, transformation and cleanliness.
The role will involve understanding my client’s current data and data pipeline architecture and looking at how they can optimise this for cross-functional teams.
The data engineer will work closely with, and support, data scientists, software developers and data analysts to ensure that data is prepared and shipped in the best possible manner for their needs.
Work with the PO to understand how data should be managed within a product
Build out complex data transformation pipelines and processes
Advise on best practices around data lakes and related technologies in the cloud
Maintain and support creation of data warehouses in the cloud
Assemble large, complex data sets to meet functional/non-functional requirements
Support in the quality assurance of data to ensure it is fit for purpose
Define data access policies and help police them
Create data tools for analytics and data science teams
Essential Technical Experience
Excellent SQL knowledge and experience working with relational databases
Experience in manipulating, processing and extracting data from dispirit data sources
Experience building data pipelines, ideally in the cloud
Strong analytical experience
Knowledge of queues and big data stores such as Hadoop
Experience with NoSQL databases
Experience with AWS services such as RDS, EC2 and Redshift
Experience with stream processing services such as Apache Spark etc.
Ability to elicit requirements from stakeholders across the business
Desirable Technical Experience
Experience using JIRA
Insurance industry experience
Experience with programming languages such as Python or Java