Data Processing Engineer - I/O
Mountain View, CAAll locationsMountain View, CAHyderabad, IN Remote
Salary not listedJobFig found this opening at its original source and checks that it remains available.
About the role
We are looking for specialists to join our engineering team and shape the future of accelerated data processing. As a Data Processing Engineer - I/O, you will be a key individual contributor in advancing data read and write capabilities of DataPelago’s data processing engine. You will enhance functional breadth, performance, scale, and reliability of the DataPelago engine in reading and writing large scale data of various data types from diverse data sources and data sinks. This is a unique opportunity to make a significant impact on a category-defining product and work with a talented team of engineers.
What you'll bring
- Bachelor's degree in Computer Science or a related field with 7+ years of relevant experience OR a Master's degree in Computer Science or a related field with 5+ years of relevant
- 3+ years of deep technical experience in developing and optimizing data read and write interfaces for large-scale data processing, particularly related to Apache Parquet, Apache
- ORC, Apache Iceberg, Apache Spark, and similar technologies.
- Demonstrated experience in instrumenting, analyzing, and optimizing the performance of
- data processing engine components on benchmark and customer workloads.
- Demonstrated experience in the design, development, and successful release of high-performance data processing engine features for large production deployments.
- Good knowledge of the architecture of one or more of Apache Spark, Apache Flink, Presto/
- Trino.
- Exceptional programming skills in C, C++.
- Extensive development experience in Linux environments.
- Strong analytical and problem-solving skills with a passion for performance optimization.
- Rust experience preferred.