Lead Data Engineer Job Description
Lead Data Engineer Duties & Responsibilities
To write an effective lead data engineer job description, begin by listing detailed duties, responsibilities and expectations. We have included lead data engineer job description templates that you can modify and use.
Sample responsibilities for this position include:
Lead Data Engineer Qualifications
Qualifications for a job description may include education, certification, and experience.
Licensing or Certifications for Lead Data Engineer
List any licenses or certifications required by the position: AWS, SQL, ETL, MS, BI, CCNP, CISSP, CISA, CISM, MCTS
Education for Lead Data Engineer
Typically a job would require a certain level of education.
Employers hiring for the lead data engineer job most commonly would prefer for their future employee to have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Engineering, Statistics, Education, Technical, Information Technology, Information Systems, Mathematics, Computer Engineering, Management
Skills for Lead Data Engineer
Desired skills for lead data engineer include:
Desired experience for lead data engineer includes:
Lead Data Engineer Examples
Lead Data Engineer Job Description
- Possess analytical skills to evaluate, understand and interpret credit bureau data from the three national credit bureau agencies
- Writes specifications to capture specific consumer behaviors from data assets
- Must be able to communicate ideas and analysis results effectively both verbally and in writing to both internal and external clients
- Responsible for executing analytical projects, leading project development for data analysis and decision support tools
- Supervising the Jakarta data engineering team to provide robust data/analytics platforms to the entire organization
- Coordinating closely with the Business Intelligence and DWH team for data integration and alignment
- Designing and supervising strategic internal data tools
- Reviewing code before delivery
- Exploring new data sources to provide additional data to power business strategy decisions
- Build, scale and maintain data pipelines to process billions of daily events into our Hadoop and RDBMS data warehouses
- Experience with new generation technologies, such as Hadoop, HBase, Hive, Cassandra, MongoDB
- Expert experience in data warehouse development utilizing Microsoft SQL Server 2012
- Knowledge and experience working with Linux build environments
- Experience with Hardware/Software monitoring
- In-depth understanding of the Hadoop eco-system
- 3+ years with Agile engineering practices
Lead Data Engineer Job Description
- Provide support for deployed data applications and analytical models by being a trusted advisor to Data Scientists and other data consumers by identifying data problems and guiding issue resolution with partner Data Engineers and source data providers
- Leading a team of mission data engineers to operation and maintain data shadowing for flight test events
- Interfacing with program responsible engineers to collect data display, data recording, data archiving and intercom system requirements
- Design and implementation of data telemetry hardware systems including network backbone
- Implementing a real-time display system for test engineers to monitor data during tests
- Maintenance of master parameter list and derived parameters
- Implementing/developing data post processing toolsets and supporting test engineer post processing requirements
- Implementing data archival system
- Organizing and tasking a team of engineers
- Instrumentation and telemetry principles
- Knowledge and exposure to Big Data technologies including
- Bachelor’s degree in Programming/Systems or Computer Science or other related field
- Application programming, analysis experience or related IT experience
- Technical ability – pro-actively keep their own specialist skills and knowledge at leading edge standard in order to fully fulfil their role
- Delivery Capability– ability to work in an agile environment with evolving demands and increasing expectations
- Interpersonal Skills - expert problem solving skills with the ability to handle stressful situations with perseverance and professionalism
Lead Data Engineer Job Description
- Real-time display system hardware/tools
- Pulse Code Modulated (PCM) databus
- 1553, 1394, Ethernet, serial databus protocols
- Intercomm hardware systems
- Basic networking principles and configuration
- Resolve issues involving map reduce, yarn, sqoop job failures
- Work with data delivery teams to setup new Hadoop users
- Participate in new data product or new technology evaluations
- Assist in the decision-making process related to the selection of software architecture solutions
- Implement architectures to handle web-scale data and its organization
- 1.5+ years of hands on experience with Apache Spark
- 3+ years of experience in implementing Hadoop solutions in data analytics space
- Advanced degree in Computer Science, Engineer, Economics, Statistics or related field
- Strong knowledge of credit data
- Experience in programming and high-throughput analysis of data, including integration of large-scale data sets, design and implementation of data processing pipelines, and analysis
- Strong knowledge of statistical analysis software is a plus
Lead Data Engineer Job Description
- Assist in creating documents that ensure consistency in development across the online organization
- Implement and support a platform that can provide ad-hoc access to large datasets
- Full lifecycle application development related to middleware and enterprise data movement
- Advance thecloud architecture for data stores
- Work closely with product managers and engineers to design, implement, test and continually improve scalable web applications and services running on AWS
- Mentor and assit other engineers in or out of your areas of ownership and expertise
- Investigate, evaluate and present new emerging technologies for use with web applications and services
- Lead and participate in architecture and design efforts
- Develop scalable production ready data integration and processing solutions
- Convert large volumes of structured and unstructured customer data
- Experience programming in SAS, UNIX, Perl, Python, C++, SQL, Excel VB, and other languages
- Designing and building statistical analysis models, machine learning models, other analytical modeling using these technologies on large data sets
- 8 or more years of hands-on experience developing software with Ab-Initio ETL tool and at least 2 years of technical lead
- 2 or more years experience developing in a UNIX/Teradata Data Warehouse environment required
- Would require working with business users, understanding requirements, and training them on data retrieval from Teradata
- Experience with Informatica Powercenter desired
Lead Data Engineer Job Description
- Lead in architecture and design activities for Digital Marketing
- Build applications from conception to production
- Establishes research and proof-of-concept initiatives in new and emerging technology spaces
- Provides the team with thought leadership to promote re-use and develop consistent scalable patterns
- Drive evaluation, adoption and learn new tools and technologies to keep technology stack modern as needed for the Product solution
- Work heavily within the Hadoop ecosystem and migrate data from Teradata to Hadoop
- Identify opportunities, understand technical needs of projects and convert them into enterprise requirements
- Strategic Design and implementation decisions to ensure quality and efficiency
- Seek alternative technology solutions to problems
- Creates, supports, and administers relational databases
- 2+ years experience building production data pipelines (using Hadoop, Hive, Pig, Spark, ) on web-scale datasets
- Experience processing, storing, and using large amounts of structured and unstructured data
- Ability to conduct requirement gathering
- At least 3 years of experience with Unix and shell scripting
- Develop real-time and batch data ingest pipelines to be used for analysis, machine learning, dashboards, alerts and visualizations
- Develop new systems and tools to enable data scientists to consume and analyse data faster and more efficiently