Senior Big Data Engineer Job Description
Senior big data engineer
provides innovative design and deployment approaches that leverage the best of innovations in in-memory processing, agile delivery, automated testing, containerisation etc.
Senior Big Data Engineer Duties & Responsibilities
To write an effective senior big data engineer job description, begin by listing detailed duties, responsibilities and expectations. We have included senior big data engineer job description templates that you can modify and use.
Sample responsibilities for this position include:
Performing hands-on development, leading code reviews and testing, creating automation tools, and creating proofs of concept
Build services (e.g, Inviso and Genie) to integrate various open source data processing engines like Hadoop, Presto and Spark into our infrastructure
Ability to work along with operations team during any production issues related to platform
Best practices like agile methodologies, design thinking and continuous deployment that will allow you to innovate really fast
Architect, design, prototype, and deliver solutions across Big Data applications to support business strategies and deliver business value utilizing agile methodologies
Operations and Tuning - Investigate problems and resolve as required, including working with various internal teams and vendors
Quickly generate and update HTML prototypes from UI concepts for testing and team feedback
Manage cloud deployment & configuration of our clusters and gateways
Build tool/automation to make deployment and monitoring production environment more repeatable
Work with the AWS team to help integrate our tools with the AWS Hadoop control plane
Senior Big Data Engineer Qualifications
Qualifications for a job description may include education, certification, and experience.
Licensing or Certifications for Senior Big Data Engineer
List any licenses or certifications required by the position: AWS, GCIA, CISSP, ITIL, PCF, CV
Education for Senior Big Data Engineer
Typically a job would require a certain level of education.
Employers hiring for the senior big data engineer job most commonly would prefer for their future employee to have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Engineering, Technical, Computer Engineering, Mathematics, Software Development, Design, Education, Information Systems, Software Engineering
Skills for Senior Big Data Engineer
Desired skills for senior big data engineer include:
SQL
Build processes
Code reviews
Operations
Spark
Testing
Collections
JVM memory model
Multi-threading
Agile methodologies
Desired experience for senior big data engineer includes:
4+ years solid Java or Python development experience in a non-web oriented, production data platform environment, at Big Data scale
2 years hands on experience with extracting and loading many TBs of data rapidly from a variety of SMP and MPP databases
2 years hands on Hadoop experience (Hive, Pig, Hbase, Oozie, ), including solution design, writing MR jobs, standing up and configuring environments, and debugging workflows
2 years hands on experience building solutions in Amazon Web Services using the AWS CLI and boto
Solid understanding of EMR, S3, EC2, Vertica, and noSQL databases required
Solid understanding of Linux and Hadoop internals required
Senior Big Data Engineer Examples
1
Senior Big Data Engineer Job Description
Job Description Example
Our company is searching for experienced candidates for the position of senior big data engineer. Please review the list of responsibilities and qualifications. While this is our ideal list, we will consider candidates that do not necessarily have all of the qualifications, but have sufficient experience and talent.
Responsibilities for senior big data engineer
- Help to code our second generation data warehouse on Cloudera Hadoop
- Build transactional data capabilities centered on MongoDB
- Design and develop scalable analytic solutions processing massive amounts of real-time data
- Develop complex MapReduce jobs to compute important metrics that are used daily by millions of users
- Work with others on the Big Data team to solve complex computational and analytical problems
- Work on scalability, performance and functional problems
- Design and execute with elegance and economy
- Solve end-to-end problems independently by writing requirements, designing, and implementing software
- Responsible for the design, development and maintenance of high quality code for complex production components in given production
- Look at projects holistically and understands the impact of changes end to end
Qualifications for senior big data engineer
- Successful track record in deploying and managing large-scale distributed systems
- Expert in Linux and network configuration
- Experience in automating production systems with Python or equivalent languages
- 4+ years contributing to R&D and production deployments of large backend data processing and analysis systems
- Experience in building tools/automation in Python or equivalent
- 5+ years of software development experience in distributed systems
2
Senior Big Data Engineer Job Description
Job Description Example
Our growing company is hiring for a senior big data engineer. We appreciate you taking the time to review the list of qualifications and to apply for the position. If you don’t fill all of the qualifications, you may still be considered depending on your level of experience.
Responsibilities for senior big data engineer
- Build strong relationships with Business & Technology Partners and provide leadership, direction, best practices, and coaching to technology development teams
- Owning all Technical aspects of Software Development and architectural design for assigned applications
- Driving continuous improvement in applications, through use of consistent development practices and tools, and through ongoing design and code refactoring
- Collaborating with stakeholders through ongoing product and platform releases, to solve existing needs, identify exciting opportunities, and predict future challenges
- Working closely with product managers and architects to prioritize and manage features, technical requirements, known defects, and issues
- Managing and appropriately escalating delivery impediments, risks, issues, and changes tied to the product development initiatives
- Mentoring other engineers, and educating colleagues on industry trends and new technologies
- Data Integration – Design and develop new source system integrations from a variety of formats including files, database extracts and APIs
- Data Pipelines – Design and develop highly scalable Data Pipelines that incorporate complex transformations and efficient code
- Data Delivery – Design and develop solutions for delivering data that meets SLAs and is of high quality to various WB divisions for marketing and reporting external vendors
Qualifications for senior big data engineer
- Experience with Amazon Web Services (AWS) is a plus
- 3+ years of experience standing up and automating the deployment of solutions in AWS using the AWS CLI, with a focus on EMR, EC2, S3, EBS, Redshift, Dynamo, VPCs
- HP will not sponsor individuals for immigration benefits for this position
- Demonstrable skills in software application and services development
- Proven proficiency in Java or related programming language
- Experience with AWS and other services, such as Redshift, Kafka, Apache Storm, Lambda, S3, Kinesis
3
Senior Big Data Engineer Job Description
Job Description Example
Our company is growing rapidly and is hiring for a senior big data engineer. Please review the list of responsibilities and qualifications. While this is our ideal list, we will consider candidates that do not necessarily have all of the qualifications, but have sufficient experience and talent.
Responsibilities for senior big data engineer
- Experience with distributive computing tools (Spark, Hive, etc)
- Deals well with ambiguous/under-defined problems
- Ability to enable business capabilities through innovation is a plus Preferred Qualifications
- Delivering business value using creative and effective approaches
- Extensive proven experience working with UI technologies such as HTML, JavaScript, front-end UI design and development covering new features and capabilities
- Understand business requirements, define the data structure, collect data from various source systems and transform them into the format that data scientists can pick up to build predictive models
- Design, develop, debug, and operate resilient distributed systems that run across large amount of compute nodes
- Conduct performance analysis across all technical layers
- Build tools/automation to manage the big data landscape
- Investigate new technology and potentially help to investigate and implement container based architecture
Qualifications for senior big data engineer
- Development experience with at least one NoSQL database development (2 years/skilled)
- Experience with Hadoop/HiveQL/MapReduce
- Proficiency with build, test, and deployment automation systems
- Knowledge of security or cloud/serving technology a plus
- Bachelor’s Degree or above in Computer Science or equivalent experience (4+ years)
- 7+ years of hands on software engineering experience with good analytical and problem solving skills
4
Senior Big Data Engineer Job Description
Job Description Example
Our company is searching for experienced candidates for the position of senior big data engineer. To join our growing team, please review the list of responsibilities and qualifications.
Responsibilities for senior big data engineer
- Design, development, and support delivery of solutions on Hadoop and Aster Big Data platforms
- Help formulate use cases from potentially ambiguous business requirements
- Contribute significantly toward developing a roadmap for Big Data within the existing data warehousing architecture
- Provide technical direction to a world class engineering team
- Collaborate with Sales and Client Partners to develop new business and manage existing client relationships
- Must be adept at sizing and estimating new projects, as well supporting new business development
- Act as a focal point for the local field organizations, by providing leadership both internally and towards the customer with respect to Big Data analytics (for example, by developing new solution offers and technology demonstrations, documenting best-practice and “rolling-up their sleeves” to support successful PoC and PoV delivery)
- Work with local Sales and Professional Services management to build, manage, and close a pipeline of Big Data opportunities
- Prepare and conduct presentations and client work shops
- Create concepts and implement analytical solutions, conduct data mining, create predictive models and conduct ad hoc analytics
Qualifications for senior big data engineer
- Proficient in Object Oriented Programming and/or Functional Programming concepts
- Proficient in using Java 7 or above
- Experience in multiple modules of Spring framework (IOC/DI)
- Persistence framework (EJB, JPA, Hibernate or equivalent)
- Proficient in building REST/Soap web services and related frameworks like CXF, AXIS2, or other equivalent
- Web application development using JEE, JavaScript, jQuery, angularjs
5
Senior Big Data Engineer Job Description
Job Description Example
Our innovative and growing company is searching for experienced candidates for the position of senior big data engineer. Thank you in advance for taking a look at the list of responsibilities and qualifications. We look forward to reviewing your resume.
Responsibilities for senior big data engineer
- Monitor, analyze and improve performance of operations and build CI/CD processes
- Ensures that all defined development policies, procedures and standards are followed
- Leads a data management queue for metadata management, data ingestion, data organization and automated balance, quality and controls
- Performs hands-on architecture, design, and development of systems owning all technical aspects of the software development
- Support large-scale Hadoop clusters across production and dev environments
- Develop automation programs/scripts, setup CI/CD, graphs, dashboards, alerts, logs, and manage, scale and optimize our services
- Help setup new services in our production environments
- Manage replication links between clusters to maintain high availability, setup backups
- Develop high-volume, low-latency, data-driven solutions in the cloud utilizing current and next generation technologies to meet evolving business needs
- Develop custom batch-oriented and real-time streaming data pipelines (cloud and data-center), working within the MapReduce ecosystem, migrating flows from ELT to ETL
Qualifications for senior big data engineer
- JMS or other Messaging Technologies
- Knowledge in NoSQL databases like HBase, Redis, MongoDB and Cassandra
- Distributed and component based architecture
- MS Degree or BS with equivalent experience
- Strong demonstrated development experience in Big Data and distributed systems, including data storage and management, searching, scheduling, logging, serialization, and modeling technologies
- Experience with UNIX based systems including the ability to create and maintain Shell scripts