Spark Engineer

If you are a self-motivated and results-driven individual with a passion for data and technology, then we want to hear from you! Join our team of talented engineers and make a significant impact on our data-driven business

Job Title: Spark Engineer

Location: San Francisco, California, United States

Salary: $190000 - $224000

Position Overview: We are seeking a highly skilled and experienced Spark Engineer to join our dynamic and innovative team. As a Spark Engineer, you will be responsible for designing, building, and maintaining our Spark-based data processing platform. You will work closely with our data scientists, data analysts, and other engineers to ensure the efficient and effective use of Spark for our data-driven projects. The ideal candidate will have a strong understanding of distributed systems, data processing, and programming languages such as Scala, Java, or Python.

Key Responsibilities:

• Design, develop, and implement Spark-based data processing solutions to support our growing business needs.
• Collaborate with cross-functional teams to understand data requirements and develop scalable and optimized solutions.
• Monitor and troubleshoot Spark jobs to ensure high performance, reliability, and scalability.
• Develop and maintain ETL pipelines for data ingestion, transformation, and loading into Spark.
• Optimize and tune Spark configurations for efficient data processing and performance.
• Stay up-to-date with the latest advancements and best practices in Spark to continuously improve our data processing capabilities.
• Participate in code reviews, testing, and documentation to maintain high-quality and well-documented code.
• Provide technical guidance and mentorship to junior team members.

Qualifications:

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• Minimum of 3 years of experience as a Spark Engineer or a similar role.
• Strong understanding of distributed systems, data processing, and parallel computing.
• Proficient in Scala, Java, or Python with experience in developing Spark applications.
• Experience with big data technologies such as Hadoop, Hive, and Impala.
• In-depth knowledge of Spark architecture, internals, and performance tuning.
• Hands-on experience with cloud-based data platforms such as AWS, GCP, or Azure.
• Strong problem-solving and analytical skills with the ability to troubleshoot complex issues.
• Excellent communication and collaboration skills with the ability to work effectively in a team environment.

Apply now and take the next step in your career as a Spark Engineer.