Difference between revisions of "BSc: Introduction To Big Data"
m (M.petrishchev moved page BSc:IntroductionToBigData to BSc: Introduction To Big Data) |
|||
| Line 15: | Line 15: | ||
Software systems are increasingly based on large amount of data that come from a wide range of sources (e.g., logs, sensors, user-generated content, etc.). However, data are useful only if it can be analyzed properly to extract meaningful information can be used (e.g., to take decisions, to make predictions, etc.). This course provides an overview of the state-of-the-art technologies, tools, architectures, and systems constituting the big data computing solutions landscape. Particular attention will be given to the Hadoop ecosystem that is widely adopted in the industry. |
Software systems are increasingly based on large amount of data that come from a wide range of sources (e.g., logs, sensors, user-generated content, etc.). However, data are useful only if it can be analyzed properly to extract meaningful information can be used (e.g., to take decisions, to make predictions, etc.). This course provides an overview of the state-of-the-art technologies, tools, architectures, and systems constituting the big data computing solutions landscape. Particular attention will be given to the Hadoop ecosystem that is widely adopted in the industry. |
||
| − | + | == Course Objectives Based on Bloom’s Taxonomy == |
|
| − | === |
+ | === What should a student remember at the end of the course? === |
* The most common structures of distributed storage. |
* The most common structures of distributed storage. |
||
| Line 25: | Line 25: | ||
* Basic tools to address specific processing needs |
* Basic tools to address specific processing needs |
||
| − | === |
+ | === What should a student be able to understand at the end of the course? === |
By the end of the course, the students should be able to ... |
By the end of the course, the students should be able to ... |
||
| Line 35: | Line 35: | ||
* The characteristics of a NoSQL database |
* The characteristics of a NoSQL database |
||
| − | === |
+ | === What should a student be able to apply at the end of the course? === |
By the end of the course, the students should be able to ... |
By the end of the course, the students should be able to ... |
||
Revision as of 15:57, 26 April 2022
Introduction to Big Data
- Course name: Introduction to Big Data
- Course number: N/A
Course Characteristics
Key concepts of the class
- Distributed data organization
- Distributed data processing
What is the purpose of this course?
Software systems are increasingly based on large amount of data that come from a wide range of sources (e.g., logs, sensors, user-generated content, etc.). However, data are useful only if it can be analyzed properly to extract meaningful information can be used (e.g., to take decisions, to make predictions, etc.). This course provides an overview of the state-of-the-art technologies, tools, architectures, and systems constituting the big data computing solutions landscape. Particular attention will be given to the Hadoop ecosystem that is widely adopted in the industry.
Course Objectives Based on Bloom’s Taxonomy
What should a student remember at the end of the course?
- The most common structures of distributed storage.
- Batch processing techniques
- Stream processing techniques
- Basic distributed data processing algorithms
- Basic tools to address specific processing needs
What should a student be able to understand at the end of the course?
By the end of the course, the students should be able to ...
- The basis of the CAP theorem
- The structure of the MapReduce
- How to process batch data
- How to process stream data
- The characteristics of a NoSQL database
What should a student be able to apply at the end of the course?
By the end of the course, the students should be able to ...
- Use a NoSQL database
- Write a program for batch processing
- Write a program for stream processing
Course evaluation
| Proposed points | ||
|---|---|---|
| Labs/seminar classes | 20 | 30 |
| Interim performance assessment | 30 | 30 |
| Exams | 50 | 40 |
If necessary, please indicate freely your course’s features in terms of students’ performance assessment.
Grades range
| Proposed range | ||
|---|---|---|
| A. Excellent | 90-100 | |
| B. Good | 75-89 | |
| C. Satisfactory | 60-74 | |
| D. Poor | 0-59 |
If necessary, please indicate freely your course’s grading features.
Resources and reference material
- Slides and material provided during the course.
- Vignesh Prajapati. Big Data Analytics with R and Hadoop. Packt Publishing, 2013
- Jules J. Berman. Principles of Big Data: Preparing, Sharing, and Analyzing Complex Information. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2013
Course Sections
The main sections of the course and approximate hour distribution between them is as follows:
| Section | Section Title | Teaching Hours |
|---|---|---|
| 1 | Introduction | 2 |
| 2 | Hadoop | 4 |
| 3 | HDFS | 4 |
| 4 | MapReduce | 4 |
| 5 | YARN | 4 |
| 6 | Optimizing Data Processing | 6 |
| 7 | Spark | 6 |
Section 1
Section title:
Introduction
Topics covered in this section:
- What is Big Data
- Characteristics of Big Data
- Data Structures
- Types of Analytics
What forms of evaluation were used to test students’ performance in this section?
|a|c| & Yes/No
Development of individual parts of software product code & 0
Homework and group projects & 0
Midterm evaluation & 0
Testing (written or computer based) & 1
Reports & 0
Essays & 0
Oral polls & 0
Discussions & 1
Typical questions for ongoing performance evaluation within this section
- Describe the 6 Vs
- Describe the types of analytics
Typical questions for seminar classes (labs) within this section
- Design the structure of a DB to address a specific analytics type
- Give examples of the 6 Vs in real systems
Test questions for final assessment in this section
- Design the structure of a DB to address a specific analytics type
- Give examples of the 6 Vs in real systems
Section 2
Section title:
Hadoop
Topics covered in this section:
- Data storage
- Clustering
- Design decisions
- Scaling
- Distributed systems
- The ecosystem
What forms of evaluation were used to test students’ performance in this section?
|a|c| & Yes/No
Development of individual parts of software product code & 0
Homework and group projects & 0
Midterm evaluation & 0
Testing (written or computer based) & 1
Reports & 0
Essays & 0
Oral polls & 0
Discussions & 1
Typical questions for ongoing performance evaluation within this section
- Describe the Hadoop ecosystem
- Structure of an Hadoop cluster
- Describe the scaling techniques
Typical questions for seminar classes (labs) within this section
- Configure a basic Hadoop node
- Configure a basic Hadoop cluster
Test questions for final assessment in this section
- Identify the Hadoop components useful to address a specific problem.
- Configure an multi-node Hadoop system.
Section 3
Section title:
HDFS
Topics covered in this section:
- Distributed storage
- Types of nodes
- Files and blocks
- Replication
- Memory usage
What forms of evaluation were used to test students’ performance in this section?
|a|c| & Yes/No
Development of individual parts of software product code & 1
Homework and group projects & 1
Midterm evaluation & 1
Testing (written or computer based) & 1
Reports & 0
Essays & 0
Oral polls & 0
Discussions & 1
Typical questions for ongoing performance evaluation within this section
- Describe the characteristics of the different nodes
- How files and blocks are managed
- How memory is managed
- How replication works
Typical questions for seminar classes (labs) within this section
- Configure a HDFS cluster
- Configure different replication approaches
- Build a HDFS client
- Use a HDFS command line
Test questions for final assessment in this section
- Configure a HDFS cluster with some specific replication approaches
- Build a HDFS client
Section 4
Section title:
MapReduce
Topics covered in this section:
- Distributed processing
- MapReduce model
- Applications
- Tasks management
- Patterns
What forms of evaluation were used to test students’ performance in this section?
|a|c| & Yes/No
Development of individual parts of software product code & 1
Homework and group projects & 1
Midterm evaluation & 1
Testing (written or computer based) & 1
Reports & 0
Essays & 0
Oral polls & 0
Discussions & 1
Typical questions for ongoing performance evaluation within this section
- Describe the MapReduce model
- Describe tasks management
- Describe patterns of usage
Typical questions for seminar classes (labs) within this section
- Solve with MapReduce a specific problem
- Implement a usage pattern
Test questions for final assessment in this section
- Describe the advantages and disadvantages of the MapReduce model
- Solve a task designing the solution using MapReduce
- Solve a task designing the solution using a composition of usage patterns
Section 5
Section title:
YARN
Topics covered in this section:
- Resource manager
- Components
- Run an application
- Schedules
What forms of evaluation were used to test students’ performance in this section?
|a|c| & Yes/No
Development of individual parts of software product code & 1
Homework and group projects & 1
Midterm evaluation & 1
Testing (written or computer based) & 1
Reports & 0
Essays & 0
Oral polls & 0
Discussions & 1
Typical questions for ongoing performance evaluation within this section
- Describe the resource manager
- Describe the lifecycle of an application
- Describe and compare the scheduling approaches
Typical questions for seminar classes (labs) within this section
- Compare the performance of the different schedules in different load conditions
- Configure YARN
- Evaluate the overall performance of YARN
Test questions for final assessment in this section
- Evaluate the performance of a specific configuration
- Compare the different schedules
Section 6
Section title:
Optimizing Data Processing
Topics covered in this section:
- CAP theorem
- Distributed storage and computation
- Batch Processing
- Stream Processing
- Usage patterns
- NoSQL databases
What forms of evaluation were used to test students’ performance in this section?
|a|c| & Yes/No
Development of individual parts of software product code & 1
Homework and group projects & 1
Midterm evaluation & 1
Testing (written or computer based) & 1
Reports & 0
Essays & 0
Oral polls & 0
Discussions & 1
Typical questions for ongoing performance evaluation within this section
- Analyze the CAP theorem
- Define the kinds of data storage available
- Characteristics of batch processing
- Characteristics of stream processing
- Describe the usage patterns
- Compare NoSQL databases
Typical questions for seminar classes (labs) within this section
- Build a program to solve a problem with batch processing
- Build a program to solve a problem with stream processing
- Interact with a NoSQL database
Test questions for final assessment in this section
- Identify problems and solutions related to the CAP theorem
- Compare solutions with batch and stream processing approaches
- Design a system using a NoSQL database
Section 7
Section title:
Spark
Topics covered in this section:
- Architecture
- Use cases
- Job scheduling
- Data types
- SparkML
- GraphX
What forms of evaluation were used to test students’ performance in this section?
|a|c| & Yes/No
Development of individual parts of software product code & 1
Homework and group projects & 1
Midterm evaluation & 1
Testing (written or computer based) & 1
Reports & 0
Essays & 0
Oral polls & 0
Discussions & 1
Typical questions for ongoing performance evaluation within this section
- Describe the architecture of Spark
- Describe the types of schedulers
- Different characteristics of the data types
- Features of SparkML
- Features of GraphX
Typical questions for seminar classes (labs) within this section
- Analyze the performance of different schedulers
- Write a program exploiting the features of each data type
- Write a program using SparkML
- Write a program using GraphX
Test questions for final assessment in this section
- Compare the performance of different schedules with different loads
- Extend the SparkML library with a custom algorithm
- Extend the GraphX library with a custom algorithm