Data Engineer Job Description
Data engineer
provides code reviews to assess overall code quality and flexibility, and the application of good object-orientated programming techniques.
Data Engineer Duties & Responsibilities
To write an effective data engineer job description, begin by listing detailed duties, responsibilities and expectations. We have included data engineer job description templates that you can modify and use.
Sample responsibilities for this position include:
Work closely with the data science team to build a state-of-the-art analytical platform to facilitate analysis and A/B testing of massive search and user data
Gather and process raw data at scale (including writing scripts, web scraping, calling APIs, write SQL queries, writing applications)
Familiarity with SQL, AWS Stack (S3, Redshift, RDS, MySQL)
Data structure model and design, data (de)normalization, data indexing, data reporting
Creating product metrics, ad hoc queries, and business needs documentation
Query performance and tuning
Solve the challenges of scaling and optimization of the internal data flow
Ensure consistency and accuracy of the data, and help design and develop data models for reporting and aggregation frameworks
Analyze Finance data models, create and optimize data ingestion processes for the Data Lake through ETL technologies like data replication and scripting with Greenplum, Talend
Assists in developing project plan and cost estimations
Data Engineer Qualifications
Qualifications for a job description may include education, certification, and experience.
Licensing or Certifications for Data Engineer
List any licenses or certifications required by the position: AWS, CCNP, CCNA, ITIL, II, IAT, GCP, AZURE, PMP, SQL
Education for Data Engineer
Typically a job would require a certain level of education.
Employers hiring for the data engineer job most commonly would prefer for their future employee to have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Engineering, Technical, Statistics, Mathematics, Information Systems, Education, Business, Math, Science
Skills for Data Engineer
Desired skills for data engineer include:
Testing
SQL
Python
Code reviews
Operations
Build processes
Implementation
Debugging
Applications programming and ensure application design adheres to the overall architecture blueprint
How areas of business
Desired experience for data engineer includes:
Experience working in a Linux environment or similar
Has worked with web-scale data sets, preferably in Hadoop
Experience with Cassandra, Kafka, Pig, Hive, Storm and other tools are a great plus
Experience with machine learning or related fields is another great plus
Bachelor's degree in a quantitative field (e.g., Mathematical, Statistics )
Proficiency with ETL development and tools (experience with Informatica a plus)
Data Engineer Examples
1
Data Engineer Job Description
Job Description Example
Our innovative and growing company is looking to fill the role of data engineer. If you are looking for an exciting place to work, please take a look at the list of qualifications below.
Responsibilities for data engineer
- Handle and report from, large complex datasets in collaboration with domain experts
- Data extraction through RESTful services
- Design end-to-end big data solutions by understanding business requirements and translating them into technical components as building blocks of the to-be solution
- Analyse data with sourcing, capturing, transformation and preparation techniques
- Work with new technologies such as Kafka, Flink, Storm, Spark, Hive, Sqoop, Oozie, Impala, Akka, Elastic search…
- Become a real data architect
- Build, deploy and manage big data solutions that can adequately handle the needs of a rapidly growing data driven company
- Streamline data access and security to enable data scientists and analysts to easily access to data whenever they need to
- Build out scalable and reliable ETL pipelines and processes to ingest data from a large number and variety of data sources
- Maintain and optimize the performance of our data analytics infrastructure to ensure accurate, reliable and timely delivery of key insights for decision making
Qualifications for data engineer
- Extensive background extracting and transforming complex data sets (ETL process design and administration)
- Slow data loads and data bugs result in keyboard facerolls
- Previous experience with the Hadoop eco-system (HDFS, Map Reduce, Pig, Hive)
- Passionate video gamer and in-depth knowledge of Blizzard games, products, and services
- Database and data warehousing experience, both in RDBMS and NoSQL environments
- Experience with Unix / Linux bash
2
Data Engineer Job Description
Job Description Example
Our company is searching for experienced candidates for the position of data engineer. Thank you in advance for taking a look at the list of responsibilities and qualifications. We look forward to reviewing your resume.
Responsibilities for data engineer
- Influence and evolve Big Data security models
- Capture, maintain and integrate technical metadata in a Big Data ecosystem and external metadata repositories
- Assist and enable the integration of business metadata in a Big Data ecosystem and external metadata repositories
- Participate in PoC/PoT efforts to integrate new Big Data management technologies, software engineering tools, and new patterns into existing structures
- Create custom software components
- Create Big Data warehouses that can be used for reporting or analysis by data scientists
- Influence/recommend ways to improve data reliability, efficiency and quality
- Collaborate with other data management and IT team members on project goals
- Document detailed Big Data design solutions conformant to enterprise standards, architecture and technologies
- Oversee handover to operational teams
Qualifications for data engineer
- Experience in remote software installation
- Bachelor’s Degree in Computer Science, Information Systems, Mathematics, Statistics, Finance, Business, related field or equivalent working experience
- Excellent verbal/written communication & data presentation skills, including ability to succinctly summarize key findings and effectively communicate with both business and technical teams
- Industry experience as a Data Engineer or related specialty
- Experience communicating with senior management (top level decision makers) with colleagues from engineering, analytics, and business backgrounds
- Experience with Business Intelligence tools and systems
3
Data Engineer Job Description
Job Description Example
Our growing company is looking to fill the role of data engineer. Thank you in advance for taking a look at the list of responsibilities and qualifications. We look forward to reviewing your resume.
Responsibilities for data engineer
- Implement best practices around database operations
- Programming in Python, Go, Java (and learning other languages as needed) to process large amounts of structured and unstructured data
- Understanding emerging trends in the constantly evolving and growing Big Data field
- Delivering systems and features with the best performance and scaled to our constantly growing business needs
- Build data expertise and own data quality for the pipelines you build
- Architect, build and launch new data models and data marts that provide intuitive analytics to your customers
- Design and develop new systems and tools to enable folks to consume and understand data faster
- Have a clear understanding of the reports/analyses/insights to be driven by data and build data solutions to optimally support the analytics needs
- Integrate third party data to enrich our data environment and enable new analytic perspectives
- Work across multiple teams in high visibility roles and own solutions end-to-end
Qualifications for data engineer
- 3+ years development experience with notable BI reporting tools like Oracle BI Enterprise Edition (OBIEE), Tableau
- Experience working with Distributed databases and AWS or other cloud technologies would be a plus
- Use business intelligence and visualization software (e.g., OBIEE, Tableau Server, ) to empower non-technical, internal customers to drive their own analytics and reporting
- 1+ years of Demonstrable Experience in developing natural language processing models, corpus and models
- Experience with NLP tools such as NLTK, OpenNLP, Stanford CoreNLP and similar open source solutions
- Experience with NLP tagging methods and techniques such as CCG, Penn TreeBank
4
Data Engineer Job Description
Job Description Example
Our company is growing rapidly and is looking for a data engineer. Thank you in advance for taking a look at the list of responsibilities and qualifications. We look forward to reviewing your resume.
Responsibilities for data engineer
- Managing AWS resources including EC2, RDS, Redshift and
- Design, develop, implement, and support our emerging big data analytics capabilities through the development and maintenance of advanced data ingestion, processing, modeling, and reporting capabilities
- Partner effectively with both WM and Enterprise IT teams, with key business stakeholders and help apply leading edge technology to solve complex business problems
- Design and implement new or enhanced data processes, tools or models
- Working with stakeholders, provide operational and process execution and ongoing user support
- Write a query to do almost anything
- Process unstructured data into a form suitable for analysis – and then do the analysis if needed
- We use AWS extensively, so experience with RDS and other web services will help you hit the ground running
- Define and develop an approach to database upgrades that allows for always on services
- Assisting with solutions by helping to define database physical structure and functional capabilities, database security, data back-up, and recovery specifications
Qualifications for data engineer
- Ability to apply combinations of classifiers, LDA, Naïve Bayes, Decision Tree, k-NN, and SVM
- Experience with Machine Learning applications and tools such as Mahout, Mallet, PyBrain
- Experience developing and applying machine learning/deep learning methods using tools such as Scikit, R or similar languages
- Experience in analyzing vast data stores and uncovering insights
- 3 years’ experience as a Database Engineer with exposure to Big Data solutions
- Candidate must have a deep understanding of logical and physical data modeling for OLTP and OLAP systems
5
Data Engineer Job Description
Job Description Example
Our growing company is looking to fill the role of data engineer. If you are looking for an exciting place to work, please take a look at the list of qualifications below.
Responsibilities for data engineer
- Track success criteria, communicate results with senior leaders, support adhoc requests, bridge technical and business program metrics
- Work with bag data technologies like Cassandra, Accumulo, HBase, Spark, Hadoop, HDFS, AVRO, MongoDB, or Zookeeper
- Profile data to measure quality, integrity, accuracy, and completeness
- Develop and implement tools, scripts, queries, and applications for ETL
- Increase “speed to deliver” by implementing workload/workflow automation solutions
- Collaborate with other analysts and audit teams to identify opportunities to design and implement solutions maximizing productivity and revenue
- Convert & cleanse “big data”
- Contribute to group knowledge sharing platforms and best practices
- Restore client data from various external sources to our file system
- Manage technical day to day with 3rd party data providers and demand side platforms, troubleshooting any issues that may arise
Qualifications for data engineer
- Self-starter, able to tackle problems alone, and also to work well with internal and external development partners
- Must integrate well into a multi-cultural team environment
- Ability to travel to destinations in the United States to Europe
- Experience in a data warehouse development and/or data engineering role
- BS/MS in Computer Science, Engineering
- Experience in handling large data sets (multiple TBs)