Senior Data Engineer (Python / Spark / Delta Lake / Elasticsearch)

Job description
Senior Data Engineer (Python / Spark / Delta Lake / Elasticsearch)

With headquarters in Australia and a regional office in Sri Lanka, 3Rive Technologies strives to set global standards. With a proven track record in delivering effective digital solutions for global organizations. We are looking for a few Senior Data Engineer (Python/Spark/Delta Lake/Elasticsearch) to join our team. Where the potential candidate will get an opportunity to collaborate with the corporate world while having personal career growth. Become a part of our inclusive team, where every individual is appreciated, respected, and experiences a sense of belonging.

Key Responsibilities:
• Develop custom software products that scale for enterprise demands, based on functional guidance from product management and technical guidance from engineering leadership.
• Develop code to support search engineering including ingestion and processing of data, optimizing the search algorithm and developing search related features using Python, PySpark, Scala, Spark, Java and related technologies.
• Utilize Elasticsearch for data storage and retrieval
• Develop and execute data loading procedures into the Delta Lake target environment, ensuring data integrity and accuracy.
• Contribute to, and occasionally lead, technical meetings on a regular basis to clearly articulate engineering development status and escalate issues with tasks assigned.
• Participate in research and development work to advance engineering and product organizational objectives.
• Ensure high code quality by following software engineering best practices.
• Work collaboratively in a cross functional team in an Agile delivery environment.
• Adhere to DevOps principles and be involved in projects throughout their full software development lifecycle: from development, QA and deployment to post-production support.
• Actively engage with internal training courses and offerings to improve technical and non-technical skills.

Skills and Qualifications:
• Bachelor’s degree in computer science, Information Technology, or a related field
• 3+ years of experience as a Software Engineer or related role
• Experience with large-scale search systems and big data technologies
• Strong programming skills using Java and Python is a must
• Experience with NoSQL datastores like Elasticsearch
• Spark, Snowflake, Google Vertex and Delta Lake experience .
• Experience with real-time data processing technologies (e.g., Apache Kafka, Apache Flink)
• Proven experience with SQL and writing complex queries
• Experience with GCP (Google Cloud) experience will be a nice to have
• Excellent problem-solving and analytical skills
• Strong communication and collaboration skills
• Ability to work independently and as part of a team
• Practical exposure to software scalability, performance, and resiliency optimization techniques.
• Experience working with Apache Iceberg will be an added advantage
• Experience with PyTorch, TensorFlow, R or similar technologies will be added advantage
• Familiarity with working in an agile development environment, along with exposure to DevOps tools and methodologies

What we can ensure for you,
As a team, we believe it takes great people to make a great company.
It’s the exceptional contribution of all individuals at 3rive that drives our success! Our mission is to do meaningful work and deliver projects that we’re proud and passionate about. We work smartly, and we have a great time doing it all together. Our open and collaborative work environment allows individuals to grow and flourish, as team members build and learn from each other’s unique capabilities.
• An opportunity to prove yourself in a global context
• Access to an array of large-scale ASX & NASDAQ listed client projects with the opportunity of collaborating in the data & analytics vertical
• Flexible working hours
• Hybrid work culture
• Family medical insurance with OPD & Spectacles with dental
• Yearly performance bonus
• Career guidance with access to many professional study programs via online learning and teaching platforms
• Flat organization structure

3rive tech is an equal opportunity employer. Our organization operates as an equal opportunity employer adhering to a policy that strictly prohibits any form of discrimination based on age, disability, sex, race, religion or belief, gender reassignment, marriage / civil partnership, pregnancy/maternity, or sexual orientation.

We embrace inclusivity and actively advocate for equal opportunities, recognizing the importance of diverse talents, skills, and potential. We encourage applications from a broad spectrum of candidates. Selection for positions will be solely based on individual merit.