Key Responsibilities:

Design and develop scalable and efficient data processing pipelines using Pyspark.
Collaborate with data scientists and engineers to understand data requirements and translate them into technical solutions.
Write and optimize complex Pyspark code to process and transform large datasets.
Monitor performance and troubleshoot any issues in the data processing pipelines.
Develop and maintain data quality and data governance processes.
Continuously improve and optimize existing data workflows.
Stay updated with the latest technologies and tools in the data engineering space and proactively suggest improvements to our data infrastructure.
Mentor and provide technical guidance to junior developers.

Qualifications:

Bachelor's or Master's degree in Computer Science, Information Systems, or a related field.
Minimum of 3 years of experience working as a Pyspark Developer or in a similar role.
Strong hands-on experience with Pyspark, Python, and SQL.
Experience with big data technologies such as Hadoop, Hive, and Spark.
Familiarity with cloud platforms such as AWS, Azure, or GCP.
Excellent problem-solving and analytical skills.
Good understanding of data structures, algorithms, and data modeling.
Ability to work independently as well as in a team environment.
Strong communication skills and ability to effectively collaborate with cross-functional teams.

As a Pyspark Developer, you will have the opportunity to work on challenging projects, learn new technologies, and make a significant impact on our data infrastructure.

Hashtags