Spark Engineer

If you are passionate about big data and have a strong background in Spark development, we would love to hear from you

Position: Spark Engineer

Location: New York, New York, United States

Salary: $107 Per hour

Overview: We are seeking a highly skilled and motivated Spark Engineer to join our dynamic team. As a Spark Engineer, you will be responsible for designing, developing, and maintaining our Spark-based data processing platform. Your primary focus will be on building scalable and efficient data pipelines, as well as optimizing our Spark clusters for performance and reliability. You will work closely with our data science and analytics teams to ensure that our data platform meets their evolving needs.

Key Responsibilities:

• Design and develop robust and scalable data processing workflows using Spark
• Optimize Spark jobs for performance, scalability, and fault tolerance
• Troubleshoot and resolve issues related to Spark cluster performance and stability
• Collaborate with cross-functional teams including data scientists, data analysts, and software engineers to understand their data and processing requirements
• Develop and maintain data quality and data governance processes for the Spark platform
• Stay current with the latest advancements in the Spark ecosystem and recommend updates and improvements to our data platform
• Provide technical guidance and mentoring to junior members of the team
• Participate in code reviews and contribute to the development of best practices for Spark development
• Collaborate with DevOps teams to ensure smooth deployment and operation of Spark applications

Qualifications:

• Bachelor's degree in Computer Science, Engineering, or a related field
• Minimum of 3 years of experience working with Spark in a production environment
• Strong understanding of distributed computing principles and experience with big data technologies such as Hadoop, Hive, and HDFS
• Proficiency in programming languages such as Scala, Java, or Python
• Experience with Spark streaming, Spark SQL, and Spark MLlib
• Experience with cloud-based environments such as AWS, Azure, or Google Cloud
• Strong troubleshooting and problem-solving skills
• Excellent communication and collaboration skills
• Ability to work in a fast-paced and agile environment
• Proven track record of delivering high-quality, scalable, and maintainable code

Join our team and be a part of our exciting journey towards building a cutting-edge data platform that will drive our business forward.