Big Data Engineer

Key Skills

Python Programming, Scala Programming, C++, Java Programming, Machine learning techniques, Data science techniques, SQL, Apache Hadoop, MapReduce, TensorFlow, PyTorch, R Programming

Job Description

We are a hard-working, diverse product organization building a SaaS cloud security platform to deliver the future of cloud security and governance. We live and breathe distributed systems, horizontally scaled microservice architectures, graph data models, security-focused data analytics and data science, cloud services, and high availability designs all with integrated instrumentation and logging. We are seeking capable problem solvers and critical thinkers with growth mindsets to join our team. While we have a high bar for commitment and performance, we also expect our work to be balanced, fun and fulfilling.

We're looking to add a distributed systems software engineer to focus on big data ingestion services and features. Not only will you add features to existing collection and ingestion systems, you will also help put major new components in place within our data collection pipeline. You will help make expert decisions on pivotal technical and design choices based on operational considerations, constraints of our current product and customer-facing product roadmap. Subsequently you must be able to design, code, deploy and operationalize microservices into a public cloud and Kubernetes-based SaaS. Your work will involve both horizontal scale and data ingestion and storage challenges. You will have opportunities both to learn from senior engineers and also take a technical leadership role, while working to promote and sustain a culture of engineering and operational excellence. This position presents a long-term opportunity to lead a team dedicated to this ongoing statement of work.

Duties and Responsibilities:

• Architect, design and implement new distributed systems for collecting and storing log data, scaling parallel ingestion and analysis, supporting machine learning models, etc.
• Enhance existing backend services and pipelines that collect, store, analyze and correlate public cloud inventory, log and event data
• Analyze fault-tolerance and high availability issues, performance and scale challenges; and discuss and propose ideas to solve them
• Pinpoint problems, instrument relevant components as needed, and ultimately implement solutions
• Learn, understand and apply trade-offs between consistency, durability and costs to build solutions which can meet the demands of rapidly growing services
• Deploy and ensure operational readiness of the services, define and meet commitments to our customers regarding availability and performance
• Lead and mentor junior developers working on these services

Requirements and Qualifications:

• 3-5+ years industry experience designing, building and supporting large scale systems in production
• Experience building distributed systems microservices to support Big Data and Data Lake applications
• Familiarity with technologies like Spark, Flink, Kafka used to build high scale data pipelines
• Excellent understanding of low-level operating systems concepts including multi-threading, memory management, networking and storage, performance and scale
• Strong CS fundamentals including data structures, algorithms, and distributed systems
• Systems programming skills including multi-threading, concurrency, etc. in a compiled language like Golang, C++ or Java
• Experience with cloud infrastructure - AWS, Azure or Google Cloud
• Track record of identifying and implementing creative solutions
• Ability to influence decisions with excellent verbal and written communication skills

Category : Engineering and Technology

Subcategory: Software Engineering

Experience: Manager and Professional

Full Time/ Part Time: Full Time

Posted Date: 2021-05-06

VMware Company Overview: At VMware, we believe that software has the power to unlock new opportunities for people and our planet. We look beyond the barriers of compromise to engineer new ways to make technologies work together seamlessly. Our cloud, mobility, and security software form a flexible, consistent digital foundation for securely delivering the apps, services and experiences that are transforming business innovation around the globe. At the core of what we do are our people who deeply value execution, passion, integrity, customers, and community. Shape what’s possible today.