Senior/Principal Data Engineer

Key Skills

Java Programming, Python Programming, SQL, Git, Azure, Apache Kafka, Apache Airflow, Dagger2, SPARK Programming, Databricks, Artifactory, MLOps tools, Data Lake

Job Description

Hitachi Solutions is a global Microsoft solutions integrator passionate about developing and delivering industry-focused solutions that support our clients to deliver on their business transformation goals. Our industry focus, expertise, and intellectual property is what truly sets us apart. We have earned, and continue to maintain, a strategic relationship with Microsoft. Recognized for our achievements - teaming with our clients to deliver innovative digital solutions and services - is how we have achieved year after year recognition.
As their trusted advisor, we support our clients to deliver on their strategic business initiatives as they unify, automate, and modernize their data and operations to increase efficiency, reduce costs, and enhance their customer's experience. Our over 3,000 team members across 14 countries, and our 18 years of 100% focus on Microsoft technologies and business applications, is how we deliver excellence through expert services and industry-focused cloud solutions.
A part of Hitachi, Ltd., our company has a long and rich history of innovation, financial strength, and international presence of one of the world's largest companies. Since 1910, Hitachi, Ltd. has been a leader in manufacturing innovative products and solutions that support industry and social infrastructure around the globe supported by 303,000 employees in over 100 countries and across 864 companies.

Job Description

NEW PRODUCT DEVELOPMENT AND INNOVATIONS TEAM
This position in our company is housed in our New Product Development team formed in 2021. Joining this team represents an opportunity to fast-track your career and to work with a team of fun and nerdy colleagues in a disruptive startup atmosphere: focused on hypergrowth, moving quickly, and making mistakes in the furtherance of innovation and sound engineering.
Armed with an existing book of business, and a stable financial parent - it is the goal of this group to transform our company into a billion-dollar product company, by focusing on engineering excellence and making the cloud easier for our customers.

SENIOR/PRINCIPAL DATA ENGINEER (PYTHON, SPARK, DATABRICKS, AZURE)
Hitachi is seeking Data Engineers at multiple levels. Interested candidates are invited to apply.

This is a full-time role in our product organization for an expert in big data systems design with considerable skill and expertise in data architecture, especially in big data systems (Spark and other EDW technology).

Individuals in this role will assist in the design, development, enhancement, and maintenance of complex data pipelines products that manage business critical operations, and large-scale analytics pipelines. Qualified applicants will have a demonstrated capability to learn new concepts quickly, have a strong data engineering background, and/or have robust software engineering expertise

Qualifications

• 4+ years building conceptual, logical and/or physical database designs, or the pipelines required to build them
• 2+ years of hands-on Python experience (especially WHL and packed code)
• Expert level knowledge of using SQL to write complex, highly-optimized queries across large volumes of data.
• 2+ years of Kimball (Dimensional Modeling) Expertise
• 2+ years of Spark experience (especially Databricks Spark and Delta Lake)
• Experience with source control (git) on the command line
• Strong hands-on experience implementing big-data solutions in the Azure Data Lake ecosystem (Azure Data Lake, Databricks).
• Ability to work independently and provide guidance to junior data engineers.
• (preferred) Experience with big data pipelines or DAG Tools (Dbt, Data Factory, Airflow, or similar)
• (preferred) Experience with Kafka or other streaming technology (or a willingness to learn)
• (preferred) Experience with database deployment pipelines (i.e. dacpac's or similar technology)
• (preferred) Once or more unit testing or data quality frameworks
• (bonus) Experience with MLFlow and other MLOps pipeline technology

Additional Information

We are an equal opportunity employer. All applicants will be considered for employment without attention to age, race, color, religion, sex, sexual orientation, gender identity, national origin, veteran or disability status