Senior Software Engineer - Big Data

Key Skills

Ansible, Apache Flink, Apache Hive, Big Data Technology, Debugging, HBase, HDFS, Java Programming, NoSQL, Puppet, Python Programming, Scala Programming, Software Development, Spark Core, Terraform, YARN

Job Description

We are looking for senior or principal software engineers to join our agile team responsible for data processing and management for our client data stores and data provided by our real-time bidding solution on petabyte scale. Every day we ingest hundred of data streams from around the globe, process terabytes of data and provide them to our analytics and customer stores building the foundation for our Machine Learning, Analytics solutions within the company, as well as build analytics and insights into the data for our customers.

Essential Responsibilities

• Build fault-tolerant, scalable batch and real-time distributed data processing systems
• Daily use technologies such as YARN, HDFS, Spark, Flink, Kafka, Hive, HBase, OpenTSDB, Vertica, SQL DB
• Build solutions with advanced data pipeline programming models like Apache Beam
• Orchestrate generic deployments following the “build once run everywhere” approach – on premise or cloud
• Selection and use of adequate cloud technologies to fulfill scalability and performance requirements
• Participate in architecture discussions, influence the road map, take ownership and responsibility over new projects
• Optimize performance and resource utilization on large production clusters
• Maintain and support existing platforms and applications, evolve them to newer tech stacks and architectures
• Contribute to open source projects

Desired Characteristics

• Proven long term experience and enthusiasm for distributed data processing at scale, eagerness to learn new things
• Expertise in designing and architecting distributed low latency and scalable solutions in a hybrid environment – cloud and on-premise
• Exposure to the whole software development lifecycle from inception to production and monitoring
• Fluency in Java or solid experience in Scala, Python
• Expert in usage of services like Spark, Hdfs, Hive, Hbase
• Experience in adequate usage of cloud services (aws) at scale
• Experience in agile software development processes
• Excellent interpersonal and communication skills

Nice To Have

• Experience with large scale / multi-tenant distributed systems
• Experience with columnar / NoSQL databases – Vertica, Snowflake, Hbase, Scylla, Couchbase
• Experience in real team streaming frameworks – Flink, Storm
• Experience with configuration management tools such as Terraform/Puppet, Salt, Ansible
• Experience with debugging and tuning JVM garbage collection and memory problems