Big Data-Hadoop Job Description
Big Data-Hadoop Duties & Responsibilities
To write an effective big data-hadoop job description, begin by listing detailed duties, responsibilities and expectations. We have included big data-hadoop job description templates that you can modify and use.
Sample responsibilities for this position include:
Big Data-Hadoop Qualifications
Qualifications for a job description may include education, certification, and experience.
Licensing or Certifications for Big Data-Hadoop
List any licenses or certifications required by the position: DOD, AWS, SSL, STIBO, HDPCA, TOGAF
Education for Big Data-Hadoop
Typically a job would require a certain level of education.
Employers hiring for the big data-hadoop job most commonly would prefer for their future employee to have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Engineering, Technical, Education, Technology, Information Technology, Business, Information Systems, Mathematics, Computer Engineering
Skills for Big Data-Hadoop
Desired skills for big data-hadoop include:
Desired experience for big data-hadoop includes:
Big Data-Hadoop Examples
Big Data-Hadoop Job Description
- Big Data Sr
- Attitude for simple, elegant design
- Build high-performance and reliable data transport mechanisms
- Implement and optimize API-level integrations with Hadoop and Hadoop technologies, or other relevant technologies
- For seamless/effortless/performant orchestration of data flow and data processing
- Leverage a variety of open source technologies to achieve an optimally flexible architecture, and provide guidance to ensure customer success on best-of-breed deployments
- Define and publish best practices
- Performs as subject matter advisor and may serve as point of escalation and become actively involved, as required, to meet schedules and resolve problems highly complex in nature
- Establishes credibility with partners while managing expectations
- May develop budget for functional area
- Bachelor's degree in a technical field such as Computer Science, or equivalent work experience, -5+
- Interpret Hadoop service and role logs (HDFS)
- Proficiency across the full range of database and business intelligence tools
- Linux command line proficiency, bash shell scripting, proficiency in one of Java or Python
- Working knowledge of common Hadoop tools including Flume, Sqoop, Oozie, PIG, MapReduce, HDFS, and YARN
- Hive and Hbase databases, Parquet and Avro file formats
Big Data-Hadoop Job Description
- Participating in full lifecycle implementation of the technical components of solutions, including tool selection, data architecture, database administration, ETL (extract, transform and load), reporting and integration
- Troubleshoot issues with Hadoop
- Based on functional specifications, develop test plans, and test cases for functional, regression, and system level test activities to ensure the accuracy and stability of the software applications
- Take ownership for the manual, automation, performance and security testing for our web applications
- Design and build automation frameworks to automate as much of the testing as possible for the web applications
- Prepare test documentation based on requirements
- Track and analyze test results, create report, draw conclusion, and make necessary recommendations for improvements
- Complete full system regression testing for releases
- Collaborate with development teams to understand how development changes affect ongoing Automation efforts
- Identify the components of the system to be automated based on business priority and benefits of automated testing
- Spark Streaming, Storm & Kafka
- At least 2 years exp in Hadoop, MapReduce, Pig, Hive, Oozie and HCatalog
- At least 2 years exp in HBase, Cassandra or MongoDB or any other NoSQL DB
- At least 5 years exp in programming in either core Java or C or C++ or Scala
- At least 2 years exp in scripting in either Python or Perl or Shell scripting
- At least 3 years exp in Representational State Transfer (REST) Application Program Interface (API) and Java Messaging Service (JMS) and WebServices
Big Data-Hadoop Job Description
- Extensive usage and experience with Data Frames with in Spark
- HBase implementation for low latency requirements
- Experience in full lifecycle application development, including requirements gathering, design, coding, testing, QA and documentation and production support
- Write and review QA documentation
- Provide support for large scale Hadoop clusters across production and development environments
- Focus on automation to improve availability and scalability in our production environments
- Help architect existing and new services in our production environments
- Manage replication links between clusters to maintain high availability
- Work with SQL Server and Big Data platforms such as Splunk and Hadoop, for a variety of projects
- Handle development, optimization, testing, migration, security, tracing, logical/physical design and maintenance
- At least 4 years exp in JUnit and JMeter and MRUnit or other testing tools
- At least 3 years exp in distributed computing design patterns, algorithms, data structures, and security standards
- Basic knowledge in Kafka, Spark, and Storm
- At least 3 years exp in demonstrated effective verbal and written communication skills
- 5 + years of application development experience in Data warehouse environment in any RDBMS/ appliance data base environment
- 3 + years of application development experience in Java/ J2EE
Big Data-Hadoop Job Description
- Participate in the 24×7 on-call rotation schedule, when ready, in order to support the data platforms that support the clinical staff in the enterprise and are available 24×7
- Implement ETL processes and document ETL processes
- Define data retention policies and assist in enforcement
- Design and manage the migration of data from legacy systems to new data solutions
- Monitor and analyze information and data systems and evaluate their performance to discover ways of enhancing them (new technologies, upgrades )
- Utilizes existing methods and procedures to create designs within the proposed solution to solve business problems
- Contributes to design of solution, executes development of design, and seeks guidance on complex technical challenges where necessary
- Understands methods and procedures on new assignments and executes deliverables with guidance as needed
- Develop new applications and data flows based on functional requirements
- Unit testing and debugging
- Understanding of Business Intelligence and Data Warehousing principles including staging source data, master data management and conformed dimensions, data stewardship, and consumption data models
- Experience working with large data sets, experience working with distributed computing a plus (Map/Reduce, Hadoop, Hive, Apache Spark)
- Programming experience in Java, Scala and Python
- Ideally familiar with the concepts of Hadoop and HANA Vora, Spark, HDFS, Java Script
- Must be proficient in Java development
- Must have exposure and knowledge of messaging frameworks
Big Data-Hadoop Job Description
- Hands-on development or testing experience in Java, Scala, Python and Big Data technologies required
- Lead small teams of developers and coordinate development activities
- Mentor junior members of the team in software development best practices
- Loading from disparate data sets in to Hadoop Data Lake
- Analyzing Big Data Analytic technologies and applications in both business intelligence analysis and new service
- Offerings, adopting and implementing these insights and standard methodologies
- 3+ years’ experience working with SQL and relational database structures
- Programming experience in VBA or other programming languages
- Write SQL codes
- Ability to work well with other team members to solve problems
- Exposure and or experience with Rules based frameworks
- Experience developing Map Reduce/YARN applications
- Experience and or exposure to Spark, Scala is a big plus
- Experience with Hbase, HIVE and/or Impala is a big plus
- SQL application development experience
- Very strong analytical and problem solving skills, great communication and team player skills