Big-data Engineer and Administrator

Key Skills

Java Programming, Environmental, Python Programming, SQL, API, Scrum Agile Methodology, AWS, Big Data Technology, Apache Hadoop, Apache Kafka, Apache Hive, MapReduce, Amazon EMR, Shell Scripting language, SPARK Programming, Storm, HBase, SFTP, Amazon Simple Storage Service (S3)

Job Description

We represent Hitachi Vantara to enterprise clients across industries, establishing business relationships to understand customer challenges so that we can deliver profitable business for Hitachi products, services and solutions. We collaborate as a team and cross-functionally to ensure the success of our customers; success that is celebrated and shared. Our solutions bring value to every line of business and we need people like you to build those deep relationships and to passionately articulate our value proposition.

What you bring to the team

• 3+ years in a Big-data Engineer role;
• Sound knowledge of Hadoop Architecture and distributed data environments;
• Excellent functional knowledge and experience with big data components such as Kafka, Spark,Hadoop, Hbase, Hive;
• Experience in optimizing hadoop/hive settings for maximum throughput (includes compaction; hive metadata; TeZ Containerization;);
• Experience in shell scripting, have knowledge of scripting languages such as Python and be exposed to programming languages such as Java;
• Knowledge of Table definitions, file formats, UDF, Data Layout (Partitions and Buckets);
• Monitor Cluster performance, resource usage, backup and mirroring, provisioning and automation (Administration);
• ProdOps experience on Storm/Kafka.
• Experience in analyzing MapReduce jobs logs and error handling;
• AWS/Cloud: Sound knowledge of Cloud basics: S3, EMR. Need experience in identifying opportunities to improve processes, operations and automation using IaC;
• Experience ingesting data from multiple data sources such as REST API, SFTP flat files, Streaming data using Apache Storm, Kafka Streams or others etc.

• Ability to work under US East Coast schedule

Will be a plus:

• Degree in Computer Science, IT, or similar field; a Master's is a plus;
• Data engineering certification (e.g AWS Certified Big Data - Specialty) is a plus;
• Experience with Agile methodology

Our Company

Hitachi Vantara is part of the Global Hitachi family. We balance innovation with an open, friendly culture and the backing of a long-established parent company, known for its ethical reputation. We guide customers from what's now to what's next by unlocking the value of their data and applications to solve their digital challenges, achieving outcomes that benefit both business and society.

Our people are our biggest asset, they drive our innovation advantage and we strive to offer a flexible and collaborative workplace where they can thrive. Diversity of thought is welcomed and our employee base is represented by several active Employee Resource Group communities. We offer industry leading benefits packages (flexible working, generous pension and private healthcare) and promote a creative and inclusive culture. If driving real change gives you a sense of pride and you are passionate about powering social good, we'd love to hear from you.