Key Responsibilities:

Design, develop and maintain data pipelines using Apache Spark to process large volumes of data in real-time
Collaborate with data scientists and data engineers to understand data requirements and transform them into technical specifications
Optimize Spark jobs for performance and scalability
Troubleshoot and resolve any issues with Spark jobs in a timely manner
Develop and maintain ETL processes to ensure the accuracy and integrity of data
Implement best practices for data security and data governance
Keep up-to-date with the latest advancements in Spark and other big data technologies
Mentor and train junior Spark developers and promote a culture of continuous learning within the team

Qualifications:

Bachelor's or Master's degree in Computer Science, Data Science, or a related field
Minimum of 5 years of experience working with Apache Spark in a production environment
Strong programming skills in Scala or Java
In-depth knowledge of Spark's architecture, APIs, and internals
Experience with SQL and data warehousing concepts
Familiarity with cloud-based data platforms such as AWS, Azure, or GCP
Experience with data streaming technologies such as Kafka or Flume is a plus
Excellent problem-solving and analytical skills
Strong communication and collaboration skills
Ability to work in a fast-paced and agile environment

Opportunity to work with cutting-edge technologies and solve complex data challenges for our clients
Competitive salary and benefits package
Career growth and development opportunities
Collaborative and inclusive work culture
Flexible work arrangements
Fun team events and activities

Hashtags