Difference between revisions of "BSc: Introduction To Big Data"

From IU
Jump to navigation Jump to search
 
(One intermediate revision by one other user not shown)
Line 1: Line 1:
  +
 
= Introduction to Big Data =
  
= Introduction to Big Data =
  +
* '''Course name''': Introduction to Big Data
  +
* '''Code discipline''': N/A
  +
* '''Subject area''':
   
  +
== Short Description ==
−
* <span>'''Course name:'''</span> Introduction to Big Data
  
  +
This course covers the following concepts: Distributed data organization; Distributed data processing.
−
* <span>'''Course number:'''</span> N/A
  
   
−
== Course Characteristics ==
 +
== Prerequisites ==
   
−
=== Key concepts of the class ===
 +
=== Prerequisite subjects ===
   
−
* Distributed data organization
  
−
* Distributed data processing
  
   
  +
=== Prerequisite topics ===
−
=== What is the purpose of this course? ===
  
  +
  +
  +
== Course Topics ==
  +
{| class="wikitable"
  +
|+ Course Sections and Topics
  +
|-
  +
! Section !! Topics within the section
  +
|-
  +
| Introduction ||
  +
# What is Big Data
  +
# Characteristics of Big Data
  +
# Data Structures
  +
# Types of Analytics
  +
|-
  +
| Hadoop ||
  +
# Data storage
  +
# Clustering
  +
# Design decisions
  +
# Scaling
  +
# Distributed systems
  +
# The ecosystem
  +
|-
  +
| HDFS ||
  +
# Distributed storage
  +
# Types of nodes
  +
# Files and blocks
  +
# Replication
  +
# Memory usage
  +
|-
  +
| MapReduce ||
  +
# Distributed processing
  +
# MapReduce model
  +
# Applications
  +
# Tasks management
  +
# Patterns
  +
|-
  +
| YARN ||
  +
# Resource manager
  +
# Components
  +
# Run an application
  +
# Schedules
  +
|-
  +
| Optimizing Data Processing ||
  +
# CAP theorem
  +
# Distributed storage and computation
  +
# Batch Processing
  +
# Stream Processing
  +
# Usage patterns
  +
# NoSQL databases
  +
|-
  +
| Spark ||
  +
# Architecture
  +
# Use cases
  +
# Job scheduling
  +
# Data types
  +
# SparkML
  +
# GraphX
  +
|}
  +
== Intended Learning Outcomes (ILOs) ==
   
  +
=== What is the main purpose of this course? ===
 
Software systems are increasingly based on large amount of data that come from a wide range of sources (e.g., logs, sensors, user-generated content, etc.). However, data are useful only if it can be analyzed properly to extract meaningful information can be used (e.g., to take decisions, to make predictions, etc.). This course provides an overview of the state-of-the-art technologies, tools, architectures, and systems constituting the big data computing solutions landscape. Particular attention will be given to the Hadoop ecosystem that is widely adopted in the industry.
  
Software systems are increasingly based on large amount of data that come from a wide range of sources (e.g., logs, sensors, user-generated content, etc.). However, data are useful only if it can be analyzed properly to extract meaningful information can be used (e.g., to take decisions, to make predictions, etc.). This course provides an overview of the state-of-the-art technologies, tools, architectures, and systems constituting the big data computing solutions landscape. Particular attention will be given to the Hadoop ecosystem that is widely adopted in the industry.
   
−
=== Course objectives based on Bloom’s taxonomy ===
 +
=== ILOs defined at three levels ===
−  
−
=== - What should a student remember at the end of the course? ===
  
   
  +
==== Level 1: What concepts should a student know/remember/explain? ====
  +
By the end of the course, the students should be able to ...
 
* The most common structures of distributed storage.
  
* The most common structures of distributed storage.
 
* Batch processing techniques
  
* Batch processing techniques
Line 25: Line 87:
 
* Basic tools to address specific processing needs
  
* Basic tools to address specific processing needs
   
−
=== - What should a student be able to understand at the end of the course? ===
 +
==== Level 2: What basic practical skills should a student be able to perform? ====
−  
 
By the end of the course, the students should be able to ...
  
By the end of the course, the students should be able to ...
−  
 
* The basis of the CAP theorem
  
* The basis of the CAP theorem
 
* The structure of the MapReduce
  
* The structure of the MapReduce
Line 35: Line 95:
 
* The characteristics of a NoSQL database
  
* The characteristics of a NoSQL database
   
−
=== - What should a student be able to apply at the end of the course? ===
 +
==== Level 3: What complex comprehensive skills should a student be able to apply in real-life scenarios? ====
−  
 
By the end of the course, the students should be able to ...
  
By the end of the course, the students should be able to ...
−  
 
* Use a NoSQL database
  
* Use a NoSQL database
 
* Write a program for batch processing
  
* Write a program for batch processing
−
* Write a program for stream processing
 +
* Write a program for stream processing
  +
== Grading ==
   
−
=== Course evaluation ===
 +
=== Course grading range ===
  +
{| class="wikitable"
−  
−
{|
 +
|+
−
|+ Course grade breakdown
  
−
!
  
−
!
  
−
!align="center"| '''Proposed points'''
  
 
|-
  
|-
  +
! Grade !! Range !! Description of performance
−
| Labs/seminar classes
  
−
| 20
  
−
|align="center"| 30
  
 
|-
  
|-
  +
| A. Excellent || 90-100 || -
−
| Interim performance assessment
  
−
| 30
  
−
|align="center"| 30
  
 
|-
  
|-
  +
| B. Good || 75-89 || -
−
| Exams
  
−
| 50
 +
|-
  +
| C. Satisfactory || 60-74 || -
−
|align="center"| 40
  
  +
|-
  +
| D. Poor || 0-59 || -
 
|}
  
|}
   
  +
=== Course activities and grading breakdown ===
−
If necessary, please indicate freely your course’s features in terms of students’ performance assessment.
  
  +
{| class="wikitable"
−  
  +
|+
−
=== Grades range ===
  
−  
−
{|
  
−
|+ Course grading range
  
−
!
  
−
!
  
−
!align="center"| '''Proposed range'''
  
 
|-
  
|-
  +
! Activity Type !! Percentage of the overall course grade
−
| A. Excellent
  
−
| 90-100
  
−
|align="center"|
  
 
|-
  
|-
  +
| Labs/seminar classes || 30
−
| B. Good
  
−
| 75-89
  
−
|align="center"|
  
 
|-
  
|-
  +
| Interim performance assessment || 30
−
| C. Satisfactory
  
−
| 60-74
  
−
|align="center"|
  
 
|-
  
|-
−
| D. Poor
 +
| Exams || 40
−
| 0-59
  
−
|align="center"|
  
 
|}
  
|}
   
  +
=== Recommendations for students on how to succeed in the course ===
−
If necessary, please indicate freely your course’s grading features.
  
  +
   
−
=== Resources and reference material ===
 +
== Resources, literature and reference materials ==
   
  +
=== Open access resources ===
 
* Slides and material provided during the course.
  
* Slides and material provided during the course.
 
* Vignesh Prajapati. Big Data Analytics with R and Hadoop. Packt Publishing, 2013
  
* Vignesh Prajapati. Big Data Analytics with R and Hadoop. Packt Publishing, 2013
 
* Jules J. Berman. Principles of Big Data: Preparing, Sharing, and Analyzing Complex Information. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2013
  
* Jules J. Berman. Principles of Big Data: Preparing, Sharing, and Analyzing Complex Information. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2013
   
−
== Course Sections ==
 +
=== Closed access resources ===
   
−
The main sections of the course and approximate hour distribution between them is as follows:
  
   
  +
=== Software and tools used within the course ===
−
{|
  
  +
−
|+ Course Sections
  
  +
= Teaching Methodology: Methods, techniques, & activities =
−
!align="center"| '''Section'''
  
  +
−
! '''Section Title'''
  
−
!align="center"| '''Teaching Hours'''
 +
== Activities and Teaching Methods ==
  +
{| class="wikitable"
  +
|+ Activities within each section
 
|-
  
|-
  +
! Learning Activities !! Section 1 !! Section 2 !! Section 3 !! Section 4 !! Section 5 !! Section 6 !! Section 7
−
|align="center"| 1
  
−
| Introduction
  
−
|align="center"| 2
  
 
|-
  
|-
  +
| Testing (written or computer based) || 1 || 1 || 1 || 1 || 1 || 1 || 1
−
|align="center"| 2
  
−
| Hadoop
  
−
|align="center"| 4
  
 
|-
  
|-
  +
| Discussions || 1 || 1 || 1 || 1 || 1 || 1 || 1
−
|align="center"| 3
  
−
| HDFS
  
−
|align="center"| 4
  
 
|-
  
|-
  +
| Development of individual parts of software product code || 0 || 0 || 1 || 1 || 1 || 1 || 1
−
|align="center"| 4
  
−
| MapReduce
  
−
|align="center"| 4
  
 
|-
  
|-
  +
| Homework and group projects || 0 || 0 || 1 || 1 || 1 || 1 || 1
−
|align="center"| 5
  
−
| YARN
  
−
|align="center"| 4
  
 
|-
  
|-
  +
| Midterm evaluation || 0 || 0 || 1 || 1 || 1 || 1 || 1
−
|align="center"| 6
  
  +
|}
−
| Optimizing Data Processing
  
  +
== Formative Assessment and Course Activities ==
−
|align="center"| 6
  
−
|-
  
−
|align="center"| 7
  
−
| Spark
  
−
|align="center"| 6
  
−
|}
  
   
−
=== Section 1 ===
 +
=== Ongoing performance assessment ===
−  
−
==== Section title: ====
  
−  
−
Introduction
  
−  
−
=== Topics covered in this section: ===
  
−  
−
* What is Big Data
  
−
* Characteristics of Big Data
  
−
* Data Structures
  
−
* Types of Analytics
  
−  
−
=== What forms of evaluation were used to test students’ performance in this section? ===
  
−  
−
<div class="tabular">
  
−  
−
<span>|a|c|</span> &amp; '''Yes/No'''<br />
  
−
Development of individual parts of software product code &amp; 0<br />
  
−
Homework and group projects &amp; 0<br />
  
−
Midterm evaluation &amp; 0<br />
  
−
Testing (written or computer based) &amp; 1<br />
  
−
Reports &amp; 0<br />
  
−
Essays &amp; 0<br />
  
−
Oral polls &amp; 0<br />
  
−
Discussions &amp; 1<br />
  
−  
−  
−  
−
</div>
  
−
=== Typical questions for ongoing performance evaluation within this section ===
  
−  
−
# Describe the 6 Vs
  
−
# Describe the types of analytics
  
−  
−
=== Typical questions for seminar classes (labs) within this section ===
  
   
  +
==== Section 1 ====
  +
{| class="wikitable"
  +
|+
  +
|-
  +
! Activity Type !! Content !! Is Graded?
  +
|-
  +
| Question || Describe the 6 Vs || 1
  +
|-
  +
| Question || Describe the types of analytics || 1
  +
|-
  +
| Question || Design the structure of a DB to address a specific analytics type || 0
  +
|-
  +
| Question || Give examples of the 6 Vs in real systems || 0
  +
|}
  +
==== Section 2 ====
  +
{| class="wikitable"
  +
|+
  +
|-
  +
! Activity Type !! Content !! Is Graded?
  +
|-
  +
| Question || Describe the Hadoop ecosystem || 1
  +
|-
  +
| Question || Structure of an Hadoop cluster || 1
  +
|-
  +
| Question || Describe the scaling techniques || 1
  +
|-
  +
| Question || Configure a basic Hadoop node || 0
  +
|-
  +
| Question || Configure a basic Hadoop cluster || 0
  +
|}
  +
==== Section 3 ====
  +
{| class="wikitable"
  +
|+
  +
|-
  +
! Activity Type !! Content !! Is Graded?
  +
|-
  +
| Question || Describe the characteristics of the different nodes || 1
  +
|-
  +
| Question || How files and blocks are managed || 1
  +
|-
  +
| Question || How memory is managed || 1
  +
|-
  +
| Question || How replication works || 1
  +
|-
  +
| Question || Configure a HDFS cluster || 0
  +
|-
  +
| Question || Configure different replication approaches || 0
  +
|-
  +
| Question || Build a HDFS client || 0
  +
|-
  +
| Question || Use a HDFS command line || 0
  +
|}
  +
==== Section 4 ====
  +
{| class="wikitable"
  +
|+
  +
|-
  +
! Activity Type !! Content !! Is Graded?
  +
|-
  +
| Question || Describe the MapReduce model || 1
  +
|-
  +
| Question || Describe tasks management || 1
  +
|-
  +
| Question || Describe patterns of usage || 1
  +
|-
  +
| Question || Solve with MapReduce a specific problem || 0
  +
|-
  +
| Question || Implement a usage pattern || 0
  +
|}
  +
==== Section 5 ====
  +
{| class="wikitable"
  +
|+
  +
|-
  +
! Activity Type !! Content !! Is Graded?
  +
|-
  +
| Question || Describe the resource manager || 1
  +
|-
  +
| Question || Describe the lifecycle of an application || 1
  +
|-
  +
| Question || Describe and compare the scheduling approaches || 1
  +
|-
  +
| Question || Compare the performance of the different schedules in different load conditions || 0
  +
|-
  +
| Question || Configure YARN || 0
  +
|-
  +
| Question || Evaluate the overall performance of YARN || 0
  +
|}
  +
==== Section 6 ====
  +
{| class="wikitable"
  +
|+
  +
|-
  +
! Activity Type !! Content !! Is Graded?
  +
|-
  +
| Question || Analyze the CAP theorem || 1
  +
|-
  +
| Question || Define the kinds of data storage available || 1
  +
|-
  +
| Question || Characteristics of batch processing || 1
  +
|-
  +
| Question || Characteristics of stream processing || 1
  +
|-
  +
| Question || Describe the usage patterns || 1
  +
|-
  +
| Question || Compare NoSQL databases || 1
  +
|-
  +
| Question || Build a program to solve a problem with batch processing || 0
  +
|-
  +
| Question || Build a program to solve a problem with stream processing || 0
  +
|-
  +
| Question || Interact with a NoSQL database || 0
  +
|}
  +
==== Section 7 ====
  +
{| class="wikitable"
  +
|+
  +
|-
  +
! Activity Type !! Content !! Is Graded?
  +
|-
  +
| Question || Describe the architecture of Spark || 1
  +
|-
  +
| Question || Describe the types of schedulers || 1
  +
|-
  +
| Question || Different characteristics of the data types || 1
  +
|-
  +
| Question || Features of SparkML || 1
  +
|-
  +
| Question || Features of GraphX || 1
  +
|-
  +
| Question || Analyze the performance of different schedulers || 0
  +
|-
  +
| Question || Write a program exploiting the features of each data type || 0
  +
|-
  +
| Question || Write a program using SparkML || 0
  +
|-
  +
| Question || Write a program using GraphX || 0
  +
|}
  +
=== Final assessment ===
  +
'''Section 1'''
 
# Design the structure of a DB to address a specific analytics type
  
# Design the structure of a DB to address a specific analytics type
 
# Give examples of the 6 Vs in real systems
  
# Give examples of the 6 Vs in real systems
  +
'''Section 2'''
−  
−
=== Test questions for final assessment in this section ===
  
−  
−
# Design the structure of a DB to address a specific analytics type
  
−
# Give examples of the 6 Vs in real systems
  
−  
−
=== Section 2 ===
  
−  
−
==== Section title: ====
  
−  
−
Hadoop
  
−  
−
=== Topics covered in this section: ===
  
−  
−
* Data storage
  
−
* Clustering
  
−
* Design decisions
  
−
* Scaling
  
−
* Distributed systems
  
−
* The ecosystem
  
−  
−
=== What forms of evaluation were used to test students’ performance in this section? ===
  
−  
−
<div class="tabular">
  
−  
−
<span>|a|c|</span> &amp; '''Yes/No'''<br />
  
−
Development of individual parts of software product code &amp; 0<br />
  
−
Homework and group projects &amp; 0<br />
  
−
Midterm evaluation &amp; 0<br />
  
−
Testing (written or computer based) &amp; 1<br />
  
−
Reports &amp; 0<br />
  
−
Essays &amp; 0<br />
  
−
Oral polls &amp; 0<br />
  
−
Discussions &amp; 1<br />
  
−  
−  
−  
−
</div>
  
−
=== Typical questions for ongoing performance evaluation within this section ===
  
−  
−
# Describe the Hadoop ecosystem
  
−
# Structure of an Hadoop cluster
  
−
# Describe the scaling techniques
  
−  
−
=== Typical questions for seminar classes (labs) within this section ===
  
−  
−
# Configure a basic Hadoop node
  
−
# Configure a basic Hadoop cluster
  
−  
−
=== Test questions for final assessment in this section ===
  
−  
 
# Identify the Hadoop components useful to address a specific problem.
  
# Identify the Hadoop components useful to address a specific problem.
 
# Configure an multi-node Hadoop system.
  
# Configure an multi-node Hadoop system.
  +
'''Section 3'''
−  
−
=== Section 3 ===
  
−  
−
==== Section title: ====
  
−  
−
HDFS
  
−  
−
==== Topics covered in this section: ====
  
−  
−
* Distributed storage
  
−
* Types of nodes
  
−
* Files and blocks
  
−
* Replication
  
−
* Memory usage
  
−  
−
=== What forms of evaluation were used to test students’ performance in this section? ===
  
−  
−
<div class="tabular">
  
−  
−
<span>|a|c|</span> &amp; '''Yes/No'''<br />
  
−
Development of individual parts of software product code &amp; 1<br />
  
−
Homework and group projects &amp; 1<br />
  
−
Midterm evaluation &amp; 1<br />
  
−
Testing (written or computer based) &amp; 1<br />
  
−
Reports &amp; 0<br />
  
−
Essays &amp; 0<br />
  
−
Oral polls &amp; 0<br />
  
−
Discussions &amp; 1<br />
  
−  
−  
−  
−
</div>
  
−
=== Typical questions for ongoing performance evaluation within this section ===
  
−  
−
# Describe the characteristics of the different nodes
  
−
# How files and blocks are managed
  
−
# How memory is managed
  
−
# How replication works
  
−  
−
==== Typical questions for seminar classes (labs) within this section ====
  
−  
−
# Configure a HDFS cluster
  
−
# Configure different replication approaches
  
−
# Build a HDFS client
  
−
# Use a HDFS command line
  
−  
−
==== Test questions for final assessment in this section ====
  
−  
 
# Configure a HDFS cluster with some specific replication approaches
  
# Configure a HDFS cluster with some specific replication approaches
 
# Build a HDFS client
  
# Build a HDFS client
  +
'''Section 4'''
−  
−
=== Section 4 ===
  
−  
−
==== Section title: ====
  
−  
−
MapReduce
  
−  
−
==== Topics covered in this section: ====
  
−  
−
* Distributed processing
  
−
* MapReduce model
  
−
* Applications
  
−
* Tasks management
  
−
* Patterns
  
−  
−
=== What forms of evaluation were used to test students’ performance in this section? ===
  
−  
−
<div class="tabular">
  
−  
−
<span>|a|c|</span> &amp; '''Yes/No'''<br />
  
−
Development of individual parts of software product code &amp; 1<br />
  
−
Homework and group projects &amp; 1<br />
  
−
Midterm evaluation &amp; 1<br />
  
−
Testing (written or computer based) &amp; 1<br />
  
−
Reports &amp; 0<br />
  
−
Essays &amp; 0<br />
  
−
Oral polls &amp; 0<br />
  
−
Discussions &amp; 1<br />
  
−  
−  
−  
−
</div>
  
−
=== Typical questions for ongoing performance evaluation within this section ===
  
−  
−
# Describe the MapReduce model
  
−
# Describe tasks management
  
−
# Describe patterns of usage
  
−  
−
==== Typical questions for seminar classes (labs) within this section ====
  
−  
−
# Solve with MapReduce a specific problem
  
−
# Implement a usage pattern
  
−  
−
==== Test questions for final assessment in this section ====
  
−  
 
# Describe the advantages and disadvantages of the MapReduce model
  
# Describe the advantages and disadvantages of the MapReduce model
 
# Solve a task designing the solution using MapReduce
  
# Solve a task designing the solution using MapReduce
 
# Solve a task designing the solution using a composition of usage patterns
  
# Solve a task designing the solution using a composition of usage patterns
  +
'''Section 5'''
−  
−
=== Section 5 ===
  
−  
−
==== Section title: ====
  
−  
−
YARN
  
−  
−
==== Topics covered in this section: ====
  
−  
−
* Resource manager
  
−
* Components
  
−
* Run an application
  
−
* Schedules
  
−  
−
=== What forms of evaluation were used to test students’ performance in this section? ===
  
−  
−
<div class="tabular">
  
−  
−
<span>|a|c|</span> &amp; '''Yes/No'''<br />
  
−
Development of individual parts of software product code &amp; 1<br />
  
−
Homework and group projects &amp; 1<br />
  
−
Midterm evaluation &amp; 1<br />
  
−
Testing (written or computer based) &amp; 1<br />
  
−
Reports &amp; 0<br />
  
−
Essays &amp; 0<br />
  
−
Oral polls &amp; 0<br />
  
−
Discussions &amp; 1<br />
  
−  
−  
−  
−
</div>
  
−
=== Typical questions for ongoing performance evaluation within this section ===
  
−  
−
# Describe the resource manager
  
−
# Describe the lifecycle of an application
  
−
# Describe and compare the scheduling approaches
  
−  
−
==== Typical questions for seminar classes (labs) within this section ====
  
−  
−
# Compare the performance of the different schedules in different load conditions
  
−
# Configure YARN
  
−
# Evaluate the overall performance of YARN
  
−  
−
==== Test questions for final assessment in this section ====
  
−  
 
# Evaluate the performance of a specific configuration
  
# Evaluate the performance of a specific configuration
 
# Compare the different schedules
  
# Compare the different schedules
  +
'''Section 6'''
−  
−
=== Section 6 ===
  
−  
−
==== Section title: ====
  
−  
−
Optimizing Data Processing
  
−  
−
==== Topics covered in this section: ====
  
−  
−
* CAP theorem
  
−
* Distributed storage and computation
  
−
* Batch Processing
  
−
* Stream Processing
  
−
* Usage patterns
  
−
* NoSQL databases
  
−  
−
=== What forms of evaluation were used to test students’ performance in this section? ===
  
−  
−
<div class="tabular">
  
−  
−
<span>|a|c|</span> &amp; '''Yes/No'''<br />
  
−
Development of individual parts of software product code &amp; 1<br />
  
−
Homework and group projects &amp; 1<br />
  
−
Midterm evaluation &amp; 1<br />
  
−
Testing (written or computer based) &amp; 1<br />
  
−
Reports &amp; 0<br />
  
−
Essays &amp; 0<br />
  
−
Oral polls &amp; 0<br />
  
−
Discussions &amp; 1<br />
  
−  
−  
−  
−
</div>
  
−
=== Typical questions for ongoing performance evaluation within this section ===
  
−  
−
# Analyze the CAP theorem
  
−
# Define the kinds of data storage available
  
−
# Characteristics of batch processing
  
−
# Characteristics of stream processing
  
−
# Describe the usage patterns
  
−
# Compare NoSQL databases
  
−  
−
==== Typical questions for seminar classes (labs) within this section ====
  
−  
−
# Build a program to solve a problem with batch processing
  
−
# Build a program to solve a problem with stream processing
  
−
# Interact with a NoSQL database
  
−  
−
==== Test questions for final assessment in this section ====
  
−  
 
# Identify problems and solutions related to the CAP theorem
  
# Identify problems and solutions related to the CAP theorem
 
# Compare solutions with batch and stream processing approaches
  
# Compare solutions with batch and stream processing approaches
 
# Design a system using a NoSQL database
  
# Design a system using a NoSQL database
  +
'''Section 7'''
  +
# Compare the performance of different schedules with different loads
  +
# Extend the SparkML library with a custom algorithm
  +
# Extend the GraphX library with a custom algorithm
   
−
=== Section 7 ===
 +
=== The retake exam ===
  +
'''Section 1'''
   
−
==== Section title: ====
 +
'''Section 2'''
   
  +
'''Section 3'''
−
Spark
  
   
  +
'''Section 4'''
−
==== Topics covered in this section: ====
  
   
  +
'''Section 5'''
−
* Architecture
  
−
* Use cases
  
−
* Job scheduling
  
−
* Data types
  
−
* SparkML
  
−
* GraphX
  
   
  +
'''Section 6'''
−
=== What forms of evaluation were used to test students’ performance in this section? ===
  
   
  +
'''Section 7'''
−
<div class="tabular">
  
−  
−
<span>|a|c|</span> &amp; '''Yes/No'''<br />
  
−
Development of individual parts of software product code &amp; 1<br />
  
−
Homework and group projects &amp; 1<br />
  
−
Midterm evaluation &amp; 1<br />
  
−
Testing (written or computer based) &amp; 1<br />
  
−
Reports &amp; 0<br />
  
−
Essays &amp; 0<br />
  
−
Oral polls &amp; 0<br />
  
−
Discussions &amp; 1<br />
  
−  
−  
−  
−
</div>
  
−
=== Typical questions for ongoing performance evaluation within this section ===
  
−  
−
# Describe the architecture of Spark
  
−
# Describe the types of schedulers
  
−
# Different characteristics of the data types
  
−
# Features of SparkML
  
−
# Features of GraphX
  
−  
−
==== Typical questions for seminar classes (labs) within this section ====
  
−  
−
# Analyze the performance of different schedulers
  
−
# Write a program exploiting the features of each data type
  
−
# Write a program using SparkML
  
−
# Write a program using GraphX
  
−  
−
==== Test questions for final assessment in this section ====
  
−  
−
# Compare the performance of different schedules with different loads
  
−
# Extend the SparkML library with a custom algorithm
  
−
# Extend the GraphX library with a custom algorithm
  

Latest revision as of 12:57, 12 July 2022

Introduction to Big Data

  • Course name: Introduction to Big Data
  • Code discipline: N/A
  • Subject area:

Short Description

This course covers the following concepts: Distributed data organization; Distributed data processing.

Prerequisites

Prerequisite subjects

Prerequisite topics

Course Topics

Course Sections and Topics
Section Topics within the section
Introduction
  1. What is Big Data
  2. Characteristics of Big Data
  3. Data Structures
  4. Types of Analytics
Hadoop
  1. Data storage
  2. Clustering
  3. Design decisions
  4. Scaling
  5. Distributed systems
  6. The ecosystem
HDFS
  1. Distributed storage
  2. Types of nodes
  3. Files and blocks
  4. Replication
  5. Memory usage
MapReduce
  1. Distributed processing
  2. MapReduce model
  3. Applications
  4. Tasks management
  5. Patterns
YARN
  1. Resource manager
  2. Components
  3. Run an application
  4. Schedules
Optimizing Data Processing
  1. CAP theorem
  2. Distributed storage and computation
  3. Batch Processing
  4. Stream Processing
  5. Usage patterns
  6. NoSQL databases
Spark
  1. Architecture
  2. Use cases
  3. Job scheduling
  4. Data types
  5. SparkML
  6. GraphX

Intended Learning Outcomes (ILOs)

What is the main purpose of this course?

Software systems are increasingly based on large amount of data that come from a wide range of sources (e.g., logs, sensors, user-generated content, etc.). However, data are useful only if it can be analyzed properly to extract meaningful information can be used (e.g., to take decisions, to make predictions, etc.). This course provides an overview of the state-of-the-art technologies, tools, architectures, and systems constituting the big data computing solutions landscape. Particular attention will be given to the Hadoop ecosystem that is widely adopted in the industry.

ILOs defined at three levels

Level 1: What concepts should a student know/remember/explain?

By the end of the course, the students should be able to ...

  • The most common structures of distributed storage.
  • Batch processing techniques
  • Stream processing techniques
  • Basic distributed data processing algorithms
  • Basic tools to address specific processing needs

Level 2: What basic practical skills should a student be able to perform?

By the end of the course, the students should be able to ...

  • The basis of the CAP theorem
  • The structure of the MapReduce
  • How to process batch data
  • How to process stream data
  • The characteristics of a NoSQL database

Level 3: What complex comprehensive skills should a student be able to apply in real-life scenarios?

By the end of the course, the students should be able to ...

  • Use a NoSQL database
  • Write a program for batch processing
  • Write a program for stream processing

Grading

Course grading range

Grade Range Description of performance
A. Excellent 90-100 -
B. Good 75-89 -
C. Satisfactory 60-74 -
D. Poor 0-59 -

Course activities and grading breakdown

Activity Type Percentage of the overall course grade
Labs/seminar classes 30
Interim performance assessment 30
Exams 40

Recommendations for students on how to succeed in the course

Resources, literature and reference materials

Open access resources

  • Slides and material provided during the course.
  • Vignesh Prajapati. Big Data Analytics with R and Hadoop. Packt Publishing, 2013
  • Jules J. Berman. Principles of Big Data: Preparing, Sharing, and Analyzing Complex Information. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2013

Closed access resources

Software and tools used within the course

Teaching Methodology: Methods, techniques, & activities

Activities and Teaching Methods

Activities within each section
Learning Activities Section 1 Section 2 Section 3 Section 4 Section 5 Section 6 Section 7
Testing (written or computer based) 1 1 1 1 1 1 1
Discussions 1 1 1 1 1 1 1
Development of individual parts of software product code 0 0 1 1 1 1 1
Homework and group projects 0 0 1 1 1 1 1
Midterm evaluation 0 0 1 1 1 1 1

Formative Assessment and Course Activities

Ongoing performance assessment

Section 1

Activity Type Content Is Graded?
Question Describe the 6 Vs 1
Question Describe the types of analytics 1
Question Design the structure of a DB to address a specific analytics type 0
Question Give examples of the 6 Vs in real systems 0

Section 2

Activity Type Content Is Graded?
Question Describe the Hadoop ecosystem 1
Question Structure of an Hadoop cluster 1
Question Describe the scaling techniques 1
Question Configure a basic Hadoop node 0
Question Configure a basic Hadoop cluster 0

Section 3

Activity Type Content Is Graded?
Question Describe the characteristics of the different nodes 1
Question How files and blocks are managed 1
Question How memory is managed 1
Question How replication works 1
Question Configure a HDFS cluster 0
Question Configure different replication approaches 0
Question Build a HDFS client 0
Question Use a HDFS command line 0

Section 4

Activity Type Content Is Graded?
Question Describe the MapReduce model 1
Question Describe tasks management 1
Question Describe patterns of usage 1
Question Solve with MapReduce a specific problem 0
Question Implement a usage pattern 0

Section 5

Activity Type Content Is Graded?
Question Describe the resource manager 1
Question Describe the lifecycle of an application 1
Question Describe and compare the scheduling approaches 1
Question Compare the performance of the different schedules in different load conditions 0
Question Configure YARN 0
Question Evaluate the overall performance of YARN 0

Section 6

Activity Type Content Is Graded?
Question Analyze the CAP theorem 1
Question Define the kinds of data storage available 1
Question Characteristics of batch processing 1
Question Characteristics of stream processing 1
Question Describe the usage patterns 1
Question Compare NoSQL databases 1
Question Build a program to solve a problem with batch processing 0
Question Build a program to solve a problem with stream processing 0
Question Interact with a NoSQL database 0

Section 7

Activity Type Content Is Graded?
Question Describe the architecture of Spark 1
Question Describe the types of schedulers 1
Question Different characteristics of the data types 1
Question Features of SparkML 1
Question Features of GraphX 1
Question Analyze the performance of different schedulers 0
Question Write a program exploiting the features of each data type 0
Question Write a program using SparkML 0
Question Write a program using GraphX 0

Final assessment

Section 1

  1. Design the structure of a DB to address a specific analytics type
  2. Give examples of the 6 Vs in real systems

Section 2

  1. Identify the Hadoop components useful to address a specific problem.
  2. Configure an multi-node Hadoop system.

Section 3

  1. Configure a HDFS cluster with some specific replication approaches
  2. Build a HDFS client

Section 4

  1. Describe the advantages and disadvantages of the MapReduce model
  2. Solve a task designing the solution using MapReduce
  3. Solve a task designing the solution using a composition of usage patterns

Section 5

  1. Evaluate the performance of a specific configuration
  2. Compare the different schedules

Section 6

  1. Identify problems and solutions related to the CAP theorem
  2. Compare solutions with batch and stream processing approaches
  3. Design a system using a NoSQL database

Section 7

  1. Compare the performance of different schedules with different loads
  2. Extend the SparkML library with a custom algorithm
  3. Extend the GraphX library with a custom algorithm

The retake exam

Section 1

Section 2

Section 3

Section 4

Section 5

Section 6

Section 7

Retrieved from "https://eduwiki.innopolis.university/index.php?title=BSc:_Introduction_To_Big_Data&oldid=7205"