Join Seaspan Corporation as a Senior Databricks Developer to create scalable data pipelines using Databricks, SQL, and PySpark. This role combines ETL development and data transformation in Azure environments.
This position requires extensive experience in data engineering, particularly with Databricks and Apache Spark, over at least five years. Candidates should demonstrate proficiency in designing data ingestion pipelines, optimizing Spark jobs, and ensuring data quality. You’ll collaborate with Analysts and Data Scientists to deliver high-quality analytics-ready datasets in a fast-paced environment.
Key Responsibilities:
• Design and maintain data pipelines using Databricks and Azure Data Factory
• Optimize Spark jobs and manage Delta Lake performance
• Integrate data from APIs and transform JSON into Spark tables
• Develop real-time and batch processing pipelines for streaming data
• Collaborate on data governance and metadata documentation
Requirements:
...