Hadoop Developer Job Description
Hadoop Developer Duties & Responsibilities
To write an effective hadoop developer job description, begin by listing detailed duties, responsibilities and expectations. We have included hadoop developer job description templates that you can modify and use.
Sample responsibilities for this position include:
Hadoop Developer Qualifications
Qualifications for a job description may include education, certification, and experience.
Licensing or Certifications for Hadoop Developer
List any licenses or certifications required by the position: AWS, GCP
Education for Hadoop Developer
Typically a job would require a certain level of education.
Employers hiring for the hadoop developer job most commonly would prefer for their future employee to have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Engineering, Education, Information Technology, Technical, Information Systems, Mathematics, Business, Science, Software Engineering
Skills for Hadoop Developer
Desired skills for hadoop developer include:
Desired experience for hadoop developer includes:
Hadoop Developer Examples
Hadoop Developer Job Description
- Build SQL/HIVE databases and tables
- Developing distributed computing Big Data applications using Spark, Elastic Search on HDP Hadoop
- Identify, capture and communicate the issues and risks promptly
- Coding various Services, Components, and participating in the development of Java and Big Data/Hadoop processes
- Conduct/Lead technical discussions with internal and external stakeholders
- Coordinate with cross-functional teams , testing teams and drive resolution of open items and issues
- End-to-end technology delivery for key strategic and regulatory projects
- Understand business requirements through user stories and user centred design, and identification of appropriate solutions
- Production of solution designs that meet the business requirements and are deliverable and supportable
- Experimentation with minimum viable products to verify best technology choices and solution design
- Experience using J2EE (EJB, JMS, JDBC, JSP, JSF, Struts, Tag Libraries, AJAX), XML, WebServices, Oracle (database modeling) would be a major plus
- Able to build complex queries based on large data sets aggregated across various client applications
- Excellent understanding of relational query languages
- Experience using Talend Studio 5.2 and above or another ETL tool eg
- Experience using NoSQL Databases such as MongoDB or Cassandra would be a plus
- Communicate with direct an Offshore team
Hadoop Developer Job Description
- Test possible solutions with customers as early as possible
- Manage big data infrastructure including data processes
- Execute end-to-end unit testing (Extraction, transformation, load)
- Execute data load and support data reconciliation in scope with data migration
- Participate in defect analysis and resolution
- Provide Enterprise Hadoop Development expertise that will be supporting the business functions
- Work with the team to create and maintain design documents and participate in cross training with other team members
- Support existing solutions and assist in the resolution of hard to solve data problems
- Structure and design framework including applications design
- Administration, maintenance, control, and optimization of Hadoop capacity, security, configuration, process scheduling, and errors
- Must have excellent communication skills and be able to present a coherent picture to people with technical and finance backgrounds
- Implement complex algorithms efficiently using PIG Latin language and user defined function in Java/Python
- Create and optimize workflow using Oozie scripts
- Research into Hadoop ecosystem for efficient and optimized solutions
- Create solutions involving disparate data sources viz
- Ability to architect and design optimized solutions for handling large data in an efficient and time bound manner
Hadoop Developer Job Description
- Coding various Services, Components, and participating in the development of Java and Big Data/Hadoop processes and Platform build outs
- Develop applications using open source or in house frameworks
- Evaluating existing and proposed data models and how to best access and query them existing and proposed data interfaces and how to clearly document them, including specification of data flow models, data flow timing, data mapping, and data transformation rules including data validations and controls
- Write PIG scripts, develop data integration and ingestion processes
- Architect data pipelines and integrations with internal and external sources
- Build job automation and scheduling using Control-M, unit and system testing
- Designing, developing, testing, tuning and building a large-scale data processing system, for Data Ingestion and Data products that allow the Client to improve quality, velocity and monetization of our data assets for both Operational Applications and Analytical needs
- Install, configuration, management and monitoring of various Hadoop and database systems
- Perform upgrades, scripting, task automation, backups/recovery
- Documentation of the installation and upgrade processes
- Total experience of 4-6 years with at least 3+ years on Hadoop technologies
- Skilled in algorithm design is desirable
- Understanding of RDBMS technologies and SQL language
- Experience with Spark, HBase, Java (MapReduce), Python (linux shell like scripts) development
- 3 years proven experience in ETL (Syncsort DMX-h, Ab Initio, IBM - InfoSphere Data Replication, ), mainframe skills, JCL
- Proficiency with Java, Python, Scala, HBase, Hive, Map Reduce, ETL, Kafka, Mongo, PostgreSQL, Visualization technologies
Hadoop Developer Job Description
- Loading from disparate data sets using Hadoop stack of ingestion and workflow tools
- Configure and implementation of Data Marts in Hadoop platform
- Design and develop efficient systems that involve Big Data
- Integrate 3rd party API's (REST, JSONP, OAuth)
- Join a tightly knit team solving hard problems the right way
- Own meaningful parts of our service, have an impact, grow with the company
- Engineer will be responsible for documenting detailed system specifications, design, development, testing using development tools & continuous integration processes
- Engineer has the capability to handle ad-hoc requests
- Engineers lead efforts, oversee work results, provide training and serve as a technical resource for other Developers
- Responsibility for the Development and Implementation of data-stream components for Industrial Iot Applications
- Proficiency with Java, Python, Scala, HBase, Hive, Map Reduce, ETL, Kafka, Mongo, Postgres, Visualization technologies
- BS in Computer Science or similar technical degree or specialized training/equivalent work experience
- Must be hands-on, strong team player and be extremely flexible and able to work in deadline driven, fast paced environment adopting to constantly changing business priorities
- Be self-starter, ability to work independently or with others and achieve goals in a matrixed org setting
- New and emerging technologies excite you, but you are mindful enough to employ the right solutions for addressing the business problems
- Above 4 years of working experience
Hadoop Developer Job Description
- Identification of new technological toolchain/implementation trends in Big Data
- Align with the data science team regarding machine learning efforts in the cloud
- Participate in development of data store, risk margin and client reporting applications as a developer
- Work with large systems that comprise of several applications
- Participate or lead in design of large scale analytical systems
- Participate or develop large scale systems
- Spend up to 25% of their time doing administration and maintenance activities during critical data gathering seasons
- Spends as much time as possible learning the data types of our science domain
- Candidates need to have and advanced knowledge of technologies such as technologies such as J2EE, Spring, Spark, Hadoop ,Vertica
- Expertise in Bigdata technologies is a plus
- Bachelor of Science in Computer Science, Information Technology or equivalent experience
- 2+ years of hands on development experience with Hadoop ecosystem
- ETL workflow development and scheduling experience
- Proficiency in Java and/or Scala development, in a Linux based environment