SRE - 2 (Big Data)

Key Skills

Apache Hive, Bash scripting, Big Data Technology, Grafana, HDFS, Kerberos, Linux Operating system, MapReduce, Perl Programming, Prometheus, Python Programming, SPARK Programming, YARN

Job Description

As a Site Reliability Engineer specializing in Data Platform OnPremise, you will play a pivotal role in deploying, maintaining, and optimizing the reliability, scalability, and performance of our Cloudera Data Platform (CDP) infrastructure. You will collaborate with cross-functional teams to design and implement robust systems that support our data-driven initiatives. The ideal candidate will possess deep knowledge of Data Platform systems, strong troubleshooting skills, and a proactive approach to automation and optimization. Your work will ensure the smooth functioning, security, and high performance of a large, high-density Cloudera-based infrastructure.

Roles and Responsibilities:

• Cloudera Data Platform Management: Implement and manage Cloudera Data Platform on-premises, including planning, installation, configuration, and integration with existing systems.
• Infrastructure Management: Ensure optimal performance, high availability, and scalability of Cloudera-based infrastructure. Regularly monitor system health and perform maintenance tasks.
• Troubleshooting: Utilize strong operational expertise to resolve issues related to system capacity, memory, CPU, storage, networking, and other infrastructure components.
• Automation: Develop and automate runbooks using scripting tools like Shell, Python, etc., and leverage configuration management tools such as Terraform, Ansible, or SALT.
• Data Security & Compliance: Implement and enforce security best practices to maintain data integrity and confidentiality. Ensure compliance with regulations such as GDPR, HIPAA, and DPR.
• Performance Optimization: Continuously optimize the Cloudera infrastructure to enhance efficiency and reduce costs. Identify bottlenecks, tune configurations, and implement best practices for resource utilization.
• Capacity Planning: Monitor resource utilization trends and plan for future capacity needs. Proactively identify potential constraints and propose solutions.
• Collaboration: Work with infrastructure, network, database, application, and business intelligence teams to ensure high data quality and availability, and optimize the PhonePe Hadoop ecosystem.
• Backup & Disaster Recovery: Implement and maintain robust backup and disaster recovery strategies. Regularly test and update backup and recovery procedures.
• Patches & Upgrades: Apply patches and perform rolling upgrades according to Cloudera's advisory and security compliance requirements.
• Documentation & Knowledge Sharing: Create clear, comprehensive documentation for configurations, processes, and procedures. Share knowledge and best practices with team members to foster continuous improvement.
• Collaboration & Communication: Communicate project status, issues, and resolutions clearly with cross-functional teams.

Skills Required:

• Educational Background: Bachelor’s degree in Computer Science, Engineering, or a related field.
• Technical Expertise: Proficiency in Linux system administration, shell scripting, networking concepts (including IPtables, IPsec).
• Experience: 3-5 years of experience in designing, setting up, and managing large-scale Hadoop clusters, ensuring high availability and performance optimization.
• Hadoop Ecosystem: Strong understanding of Hadoop ecosystem technologies (HDFS, MapReduce, YARN, Hive, Spark, etc.).
• Data Security: Experience with Kerberos, LDAP, and ensuring data security within distributed systems.
• Databases: Strong knowledge of relational (MySQL, SQL Server) and NoSQL databases.
• Configuration Management: Hands-on experience with tools like Salt, Ansible, Puppet, or Chef.
• Scripting Skills: Strong proficiency in scripting languages such as Perl, Python, and Bash for automation and troubleshooting.
• Monitoring & Logging: Experience with monitoring and logging solutions like Prometheus, Grafana, and the ELK stack.
• Networking Knowledge: Solid understanding of networking principles and protocols (TCP/IP, UDP, DNS, DHCP, etc.).
• Unix/Linux Expertise: Expertise in managing *nix systems (Ubuntu, Redhat, Fedora, etc.) and proficiency with Unix tools and programs.
• Communication Skills: Strong communication skills and the ability to collaborate effectively with cross-functional teams.
• Problem-Solving: Excellent analytical and troubleshooting skills with the ability to manage multiple priorities in high-pressure environments.

Good to Have:

• Certifications: Cloudera Certified Administrator (CCA) or Cloudera Certified Professional (CCP) certification.
• Cloudera CDP Experience: Minimum 2 years of experience in managing Cloudera Data Platform (CDP) and administering large Hadoop-based environments (>100 machines).
• Open Data Lake Technologies: Familiarity with Open Data Lake components like Ozone, Iceberg, Spark, and Flink.
• Containerization & Orchestration: Familiarity with Docker, Kubernetes, and OpenShift is a plus.
• Airflow: Experience with designing, developing, and maintaining Airflow DAGs for automating business-as-usual processes.

PhonePe Full-Time Employee Benefits (For Full-Time Roles Only):

• Insurance: Medical, Critical Illness, Accidental, and Life Insurance
• Wellness Program: Employee Assistance Program, Onsite Medical Center, Emergency Support System
• Parental Support: Maternity & Paternity Benefits, Adoption Assistance, Day-care Support
• Mobility Benefits: Relocation Assistance, Transfer Support, Travel Policy
• Retirement Benefits: Employee PF Contribution, Flexible PF Contribution, Gratuity, NPS, Leave Encashment
• Other Benefits: Higher Education Assistance, Car Lease, Salary Advance Policy