Senior Data Engineer Job Description
Senior Data Engineer Duties & Responsibilities
To write an effective senior data engineer job description, begin by listing detailed duties, responsibilities and expectations. We have included senior data engineer job description templates that you can modify and use.
Sample responsibilities for this position include:
Senior Data Engineer Qualifications
Qualifications for a job description may include education, certification, and experience.
Licensing or Certifications for Senior Data Engineer
List any licenses or certifications required by the position: AWS, ETL, BI, RHCE, CDMP, ITIL, GCP, BASE, SAS, CFA
Education for Senior Data Engineer
Typically a job would require a certain level of education.
Employers hiring for the senior data engineer job most commonly would prefer for their future employee to have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Engineering, Technical, Statistics, Math, Business, Design, Mathematics, Management, General Management
Skills for Senior Data Engineer
Desired skills for senior data engineer include:
Desired experience for senior data engineer includes:
Senior Data Engineer Examples
Senior Data Engineer Job Description
- Implement new mechanisms for modeling and predicting reader behavior
- Design and implement experiments and models to answer targeted questions
- Lead technical efforts to modernize and replace legacy reporting solutions
- Lead daily activities of globally distributed development team
- Collaborate with Analysts and Business SMEs to perform detailed requirement analysis for enhancing and creating new reporting and analytics solutions
- Imagine and implement creative and innovative reporting solution designs while adhering to architectural requirements of maintainability and scalability
- Individual will be hands-on and contributing to project deliverables
- Drive system problem resolution and root cause analysis
- Ensure and maintain integrity of code base during concurrent development cycles
- Collaborate with experts in a variety of technologies to come up with the best overall solutions
- A passion for thoroughness, execution and completion
- Ability to communicate in a clear, articulate, customer-friendly manner to all levels of an organization to present findings and recommendations to team members and senior management
- Experience in summarizing analysis results using PowerPoint and explaining analysis results to senior management levels
- Experience in working with virtual groups across the organization and across the world required
- Approachable, personable and team player comfortable working in an Agile environment
- Experience in managing multiple initiatives in parallel
Senior Data Engineer Job Description
- Performance diagnostics (SQL, Hardware)
- Perform on-call off-hours support for critical systems
- Ability to travel globally when needed
- Help data scientists optimize productionized Hive queries
- Ensure data quality and help in achieving data governance
- Communicate results and impact to business stakeholders
- Collaborate with other teams to tailor data products accordingly
- Responsible for aspects of new and existing data development and data integration using content-based data products
- Builds and maintains data warehouse databases
- Coordinates with other IT departments to ensure implementation and monitoring of databases
- Broad background with BI environments, tools and development encompassing relational databases, data warehousing and data mining practices
- At least 1 year of professional experience in implementing customer facing Html based Tableau reports
- Experience in implementing the Tableau Reports using Live Access model
- Understanding of Web concepts (html, css, urls, frames ) to the level appropriate to develop Tableau reports in Html according to UX designs
- Experience in configuring Tableau Reporting system in appliance deployment mode for various business needs
- Being able to articulate and communicate to various types of audiences (developers to managers to business owners)
Senior Data Engineer Job Description
- Work with client- and server-side engineers to ingest new data and properly surface for regular consumption
- Develop robust end-to-end data solutions for structured and unstructured data including, but not limited to, ingestion, parsing, integration, auditing, logging, aggregation, normalization, modeling, and error handling
- Collaborate with cross functional team to resolve data quality and operational issues
- Participate in an on-call rotation to support the Information Management cluster
- Interact directly with end users to gather requirements and consult on data integration solutions
- Identify opportunities for team standardization in coding, deployments, documentation and other related areas and create said standards
- Create and maintain key metadata artifacts including, but not limited to, data lineage, data dictionaries, and Entity Relationship Diagrams (ERD)
- Educate the team in emerging related technologies and identify value add opportunities for their implementation
- Develop Interactive Dashboards and supporting Services that will provide customer intelligence insights
- Help developing a high-profile, innovative product that will revolutionize the way how businesses attribute success to their marketing tech stacks
- Must be effective in working both independently and in a customer setting
- At least 4 years of hands-on experience working with large data sets
- Minimum of a BS in a quantitative discipline such as Statistics, Engineering, Computer Science, Applied Mathematics
- Bachelor’s Degree in Computer Science, Mathematics or Machine Learning
- 6 years experience with C#/VB, HTML/CSS, JavaScript or C++
- Familiarity with reporting or BI tools
Senior Data Engineer Job Description
- Designing scalable implementations of the models
- Hands-on programming based on TDD, usually in a pair programming environment
- Advising clients on the usage of different distributed storage and computing technologies from the plethora of options available in the ecosystem
- Support ongoing data Integration efforts for all development and production environments including Big Data Design Patterns
- Design and develop applications for all data warehousing components, including Real-Time Data Ingestion techniques, Transformations, aggregations, and related data quality strategy
- Design and implementation of multi-source data channels and ETL processes
- Design, enhance and implement ETL/data ingestion platform on the cloud
- Clear documentation of process flow diagrams and best practices
- Designing and implementing solutions to provide stakeholders self-service access to timely, accurate and actionable data that serve their distinct use-cases
- Improve the efficiency, scalability, and stability of our Vertica and Hadoop data warehouse
- A Bachelor’s degree and a minimum of 5 years’ relevant experience as a data engineer
- Work as hands-on data modeler and ad-hoc data analyst when needed
- Work with DBAs and DevOps to ensure proper scaling and monitoring of various data stores
- At least 2 years of experience in C, Java, Go, Scala, or Python
- 2+ years of experience working with Cloud platforms and services such as AWS
- 2+ years of experience working with automated build and continuous integration systems such as Ansible, Terraform , Chef and Jenkins
Senior Data Engineer Job Description
- Design and develop ETL (extract-transform-load) processes to validate and transform data, calculate metrics and attributes, populate data models , using HADOOP, Spark, SQL, and other technologies
- Lead by example, demonstrating best practices for code development and optimization, unit testing, CI/CD, performance testing, capacity planning, documentation, monitoring, alerting, and incident response in order to ensure data availability, data quality, usability and required performance
- Use programming languages such as SAS, R, Python, and SQL to create automated processes for data gathering, cleansing, reporting, and visualization
- Implement systems for tracking data quality, usage, and consistency
- Design and develop new data products, using languages, such as Ruby
- Monitor and maintain system health and security
- Oversee administration and improvements to source control and deployment process
- Prepare unit tests for all work to be released to our live environment (including data validation scripts for data sets releases or changes)
- Implement performance tuning on the databases based on monitoring
- Design and implement data products using Hadoop technologies
- Hands on deployment experience with Hadoop/Spark, Scala, MySQL, Redshift and Amazon AWS or other cloud base systems
- Comfortable writing code in python, ruby, perl, or equivalent scripting language
- Experience with Cosmos/Scope, SQL, or Hadoop
- At least 3 years of professional work experience programming in Python, Java or Scala
- Master's Degree in Computer Science, Computer Engineering, Data Science or related discipline
- 2+ years of Distributed Computing frameworks such as Apache Spark, Hadoop