Software Engineer / Data Platform

Key Skills

Java Programming, Python Programming, C++, C Programming, SQL, Cloud computing, Scala Programming, Machine learning techniques, Data science techniques, MATLAB Programming, PyTorch, TensorFlow, R Programming

Job Description

The Data Platform Team provides platforms that support data analytic activities for all services offered on the LINE app.

We are currently looking for software engineers in the following three Parts: Ingestion Part (collects and aggregates data), Platform Dev Part (develops applications related to data analysis), and Hadoop Part (develops proprietary Hadoop clusters).

Our Mission

Ingestion Part

• Develop next-gen data processing and pipeline for the LINE messaging platform
• Design and build a distributed data processing system, such as Kafka, Elasticsearch, Hadoop, etc.
• Build a large-scale data pipeline for real-time data analysis
• Design and build an automated and reliable system in order to provide highly available services

Platform Dev Part

• Work closely with other teams (e.g. Ingestion Part and Hadoop Part) to develop tools for distributed systems in order to support the tasks of internal planners and data scientists
• Develop API servers to connect systems
• Revamp and operate the above tools and services

Hadoop Part

• Develop a proprietary Hadoop distribution and its ecosystem in order to operate Vanilla Hadoop clusters consisting of a few thousand servers in a stable and effective manner
• Develop a security system to safely maintain confidential data
• Automate operation and build monitoring systems to further refine the platform, instead of using Hadoop out-of-box

About the Team

The Data Platform Team is responsible to collect data from LINE services, format data for analysis, and provide formatted data as a tool on its distributed system. Members of this team are expected to communicate and work closely with various departments, including service developers, Hadoop engineers, and data scientists. This team also communicates frequently with colleagues in both domestic and overseas offices. The team consists of members with different nationalities where communication is mainly in Japanese with occasional use of English. As with other departments, English is used in documentations and chat conversations. This team develops a wide range of tools for distributed systems (e.g. Spark and Presto) as well as more traditional tools for batch processing, admin screens, etc. Refer to the following for specific applications and peripheral systems developed by the Data Platform Team.