Data Engineer - Pyspark
- Recruiter
- Sanderson
- Location
- Midlothian
- Salary
- Competitive
- Posted
- 13 Jun 2021
- Closes
- 18 Jun 2021
- Sectors
- Engineering
- Contract Type
- Contract
- Hours
- Full Time
-Data Engineer - PySpark
-Fully Remote
-Immediate Start
-6 Month Initial Contract
-Day Rates to GBP475 via Umbrella
A number of Data Engineers required for a large Edinburgh-based client, to join the existing team as they continue their transformation journey. This position will be working on a new data platform, supporting a large volume of project work across data services. The aim is to improve the overall customer experience via a new product suite and new digital analytics capability.
There are multiple roles, with the main technical requirements being commercial experience of PySpark, optimising AWS batches and working with core data feeds. In addition, you will have experience of migration programmes, ETL and Airflow. A key function will be to work closely with key stakeholders and collaborate with other members of the Scrum team.
Candidates from a variety of background will be considered for this assignment
Key Responsibilities
- Proficient in PySpark.
- Experienced in optimising AWS batches, working with core date feeds.
- Experience of migration programmes / ETL.
- Knowledge / Experience of Airflow.
- Strong stakeholder management skills and experience working within financial services
- Develop efficient and maintainable data workflows
- Perform code reviews
- Use your initiative to suggest appropriate methods and technologies to use within the data pipelines and data structures
- Collaborating with the other members of the Scrum Team
- Effectively communicating with the wider IT team and key stakeholders
- Deliver a complete data processing solution.
- Set up and manage cloud environments and create test data
Essential Skills
- An excellent understanding of the data processing capability of Azure Cloud.
- Demonstrable experience of data pipeline development
- Demonstrable experience of graph networks and graph theory.
- Solid experience of design and development of micro services
- Familiarity with Azure SQL and Azure Functions
- Experience of using Git source control
- Experience of unit-testing frameworks, service level unit and integration testing
- Experience of using Microsoft Azure DevOps, build pipelines, release management etc.
- Understanding of how to build data processing pipelines to be secure and responsive by design
- Ideally a blend of technical skills across SQL, SAS, Azure Cloud, Scala, Spark, Spark ML, Digital Twin data structures, Data Streaming, Streamsets, Kafka
If you are keen to find out further detail around this opportunity, please forward an updated CV to