Principal Spark Developer

We are looking for a highly motivated and talented individual who is passionate about big data and thrives in a fast-paced and dynamic environment

Job Title: Principal Spark Developer

Location: San Francisco, California, United States

Salary: $270000 - $310000

Position Overview: As a Principal Spark Developer, you will be responsible for leading the design, development, and implementation of complex data processing solutions using Spark and other big data technologies. You will work closely with cross-functional teams to understand business requirements and translate them into technical solutions. The ideal candidate will have extensive experience in developing and deploying large-scale data processing applications using Spark, as well as a strong understanding of distributed systems and data engineering principles.

Key Responsibilities:

• Lead the design and development of highly scalable and fault-tolerant data processing solutions using Spark and other big data technologies
• Collaborate with data scientists, data engineers, and business stakeholders to understand requirements and identify opportunities for data-driven insights
• Design and implement data pipelines using Spark to process large volumes of data in batch and real-time
• Optimize and tune Spark applications for performance and reliability
• Develop and maintain data ingestion processes from various sources, including structured and unstructured data
• Design and implement data quality and data validation processes to ensure accuracy and consistency of data
• Create and maintain technical documentation for data pipelines and processes
• Keep up-to-date with industry trends and best practices in big data processing and distributed systems
• Provide technical guidance and mentorship to junior developers on the team
• Collaborate with DevOps and infrastructure teams to ensure smooth deployment and operation of data processing applications

Qualifications:

• Bachelor's or Master's degree in Computer Science, Computer Engineering, or a related field
• Minimum of 8 years of experience in software development, with at least 3 years of experience in developing data processing applications using Spark
• Extensive experience in building and deploying large-scale data pipelines using Spark, Hadoop, or other big data technologies
• Proficiency in programming languages such as Java, Scala, or Python
• Strong understanding of distributed systems and data engineering principles
• Experience with cloud computing platforms such as AWS, Azure, or Google Cloud
• Familiarity with data warehousing concepts and data modeling techniques
• Excellent problem-solving and analytical skills
• Strong communication and collaboration skills
• Proven track record of delivering projects on time and within budget
• Prior experience in a leadership or mentorship role is a plus.

If you are a self-starter with a strong technical background and a desire to lead and innovate, then we want to hear from you! Come join our team and be a part of shaping the future of data-driven solutions.