Big Data Pipeline Architect

If you are a highly motivated and results-driven individual with a passion for building cutting-edge data pipelines, we want to hear from you! Join our team and be part of a company that values innovation, diversity, and growth.

Position: Big Data Pipeline Architect

Location: New York, New York, United States

Salary: $247000 - $320000

We are seeking a highly skilled and experienced Big Data Pipeline Architect to join our dynamic team. As a Big Data Pipeline Architect, you will play a critical role in designing, building, and maintaining our data pipeline infrastructure. You will be responsible for creating robust, scalable, and efficient data pipelines to support our growing data needs.

Key Responsibilities:

• Design, develop, and implement scalable data pipelines to extract, transform, and load large volumes of data from various sources into our data lake.
• Work closely with cross-functional teams to understand business requirements and translate them into technical solutions.
• Conduct data analysis to identify patterns and trends, and provide insights to inform data pipeline design decisions.
• Collaborate with data engineers, data scientists, and other stakeholders to ensure data quality and integrity throughout the pipeline.
• Research and evaluate new technologies and tools to optimize data pipeline performance and efficiency.
• Develop and maintain documentation for data pipeline architecture, processes, and data flows.
• Troubleshoot and resolve complex issues related to data pipeline performance, data quality, and data integration.
• Mentor and train junior team members on data pipeline best practices and technologies.

Qualifications:

• Bachelor's or Master's degree in Computer Science, Information Systems, or a related field.
• Minimum of 5 years of experience in data engineering or data architecture roles.
• Proven track record of designing and implementing data pipelines in a Big Data environment (Hadoop, Spark, etc.).
• Strong experience with data warehousing concepts, data modeling, and ETL processes.
• In-depth knowledge of programming languages such as SQL, Python, and Java.
• Familiarity with cloud platforms and services (AWS, Azure, GCP) and big data technologies (Hive, Kafka, etc.).
• Experience with data visualization tools (Tableau, Power BI) is a plus.
• Strong analytical and problem-solving skills with the ability to think critically and creatively.
• Excellent communication and interpersonal skills with the ability to work effectively in a collaborative team environment.