Job Description
Work with multiple cloud systems like AWS and GCP and its services; Design and implement highly scalable data-intensive processing systems; Collaborate with product, data/business analysts, data scientist, and engineering on the next generation of the industry-leading agriculture platform; Develop infrastructure to inform on key metrics, recommend changes and predict future results; Work with cutting edge open-source platforms such as Hadoop, Yarn, Flink, Spark etc; Ensure our codebase is continuously deliverable and deployable. Bachelor's degree in Computer Science or related technical field, or equivalent experience; Robust experience in a Software/Data Engineer or role, who has attained a Graduate degree in Computer Science, or another related field; Experience with Python, Scala and/or Java; Experience in architecting, designing, developing, and implementing cloud solutions on AWS/GCP cloud platforms; Experience working with relational and NoSQL datastores; Experience designing and implementing scalable Data Lakes, scalable lakehouse architecture and Medallion Architecture, enabling Skills & Qualifications: efficient data versioning, governance, and analytics across bronze, silver, and gold layers; Experience with DBT or equivalent ETL tool; Experience in Star schema and/or designing database schemas for complex and evolving data sets in data warehouse context; Experience with Big Data on GCP - BigQuery, Pub/Sub, Dataproc, Dataflow; Experience with Amazon Web Services (EC2, S3, RDS, SQS, EMR etc.) (Strong Plus); Experience in Spark (Scala/Python/Java). (Strong Plus); Experience in Delta Lake, Iceberg and/or Hudi. ( Strong Plus ); A successful history of manipulating, processing and extracting value from large, disconnected datasets; Proven ability to collaborate with multidisciplinary teams of business analysts, developers, data scientists, and subject-matter experts; Good understanding of cloud design considerations and limitations and it's impact on Pricing; Experience working in both AWS and GCP cloud systems; Experience in several of the following areas: database architecture, ETL, business intelligence, big data, feature engineering, advanced analytics; Experience building scalable backend services (REST APIs, microservices, designing and implementing efficient data processing algorithms, messaging paradigms, middleware, persistent store); Experience with data pipeline and workflow management tools: Azkaban, Luigi, Airflow, Argo etc; Fluency in Portuguese and English is required for working in a global team; Willingness to be based in São Paulo, with some weekly office days expected.