Big Data Technologies and Analytics
- Course name: Big Data Technologies and Analytics
- Code discipline: N/A
- Subject area:
Short Description
This course covers the following concepts: Advanced distributed data organization; Advanced distributed data processing.
Prerequisites
Prerequisite subjects
Prerequisite topics
Course Topics
Course Sections and Topics
| Section |
Topics within the section
|
| Introduction |
- What is Big Data
- Characteristics of Big Data
- Technologies
- Virtualization and cloud computing
|
| File systems and resource managers |
- HDFS
- YARN
|
| Batch Processing |
- Distributed batch processing
- MapReduce model
- Applications
- Tasks management
- Patterns
|
| Stream Processing |
- CAP theorem
- Distributed storage and computation
- Distributed Stream Processing
- Usage patterns
|
| Analytics |
- Architecture
- Use cases
- SparkML
- GraphX
|
Intended Learning Outcomes (ILOs)
What is the main purpose of this course?
Nowadays companies need to manage vast amounts of data on a daily basis. Storing, sorting, accessing and analyzing obtaining synthetic information is considered one of the great challenges of the 21st century and and being effective in this may make the difference between success and failure. In order to gain a competitive advantage, Big Data and Analytics professionals are able to extract useful information from data and increase the Return Of Investments. In this course, students will be exposed to the key technologies and techniques, including R and Apache Spark, in order to analyze large-scale data sets and uncover valuable business information.
ILOs defined at three levels
Level 1: What concepts should a student know/remember/explain?
By the end of the course, the students should be able to ...
- Understanding of big data applications.
- Algorithms for the statistical analysis of big data
- Fundamental principles of predictive analytics
Level 2: What basic practical skills should a student be able to perform?
By the end of the course, the students should be able to ...
- How to process batch data
- How to process stream data
- Advanced design of distributed architectures
- Advanced design of distributed algorithms
Level 3: What complex comprehensive skills should a student be able to apply in real-life scenarios?
By the end of the course, the students should be able to ...
- Write a program for batch processing
- Write a program for stream processing
- Design distributed processing pipelines
- Desing distributed algorithms
Grading
Course grading range
| Grade |
Range |
Description of performance
|
| A. Excellent |
90-100 |
-
|
| B. Good |
75-89 |
-
|
| C. Satisfactory |
60-74 |
-
|
| D. Poor |
0-59 |
-
|
Course activities and grading breakdown
| Activity Type |
Percentage of the overall course grade
|
| Labs/seminar classes |
30
|
| Interim performance assessment |
30
|
| Exams |
40
|
Recommendations for students on how to succeed in the course
Resources, literature and reference materials
Open access resources
- Slides and material provided during the course.
- F. Provost and T. Fawcett. Data Science for Business. O’Reilly, 2013
- Matthew North. Data Mining for the Masses, Second Edition: with implementations in RapidMiner and R. CreateSpace Independent Publishing Platform, 2012
- Tom White. Hadoop: The Definitive Guide. O’Reilly Media, Inc., 2012
- Seema Acharya and Subhashini Chellappan. Big data and analytics. WileyIndia, 2016
Closed access resources
Software and tools used within the course
Teaching Methodology: Methods, techniques, & activities
Activities and Teaching Methods
Activities within each section
| Learning Activities |
Section 1 |
Section 2 |
Section 3 |
Section 4 |
Section 5
|
| Testing (written or computer based) |
1 |
1 |
1 |
1 |
1
|
| Discussions |
1 |
1 |
1 |
1 |
1
|
| Development of individual parts of software product code |
0 |
1 |
1 |
1 |
1
|
| Homework and group projects |
0 |
1 |
1 |
1 |
1
|
| Midterm evaluation |
0 |
1 |
1 |
1 |
1
|
Formative Assessment and Course Activities
Ongoing performance assessment
Section 1
| Activity Type |
Content |
Is Graded?
|
| Question |
Describe the 6 Vs |
1
|
| Question |
Describe the technologies to support big data |
1
|
| Question |
Design the structure of a cloud architecture for big data |
0
|
| Question |
Give examples of the 6 Vs in real systems |
0
|
Section 2
| Activity Type |
Content |
Is Graded?
|
| Question |
Describe the characteristics of the different nodes of HDFS |
1
|
| Question |
How files and blocks are managed |
1
|
| Question |
Describe the resource manager |
1
|
| Question |
Describe the lifecycle of an application |
1
|
| Question |
Describe and compare the scheduling approaches |
1
|
| Question |
Configure a HDFS cluster |
0
|
| Question |
Build a HDFS client |
0
|
| Question |
Use a HDFS command line |
0
|
| Question |
Configure YARN |
0
|
| Question |
Evaluate the overall performance of YARN |
0
|
Section 3
| Activity Type |
Content |
Is Graded?
|
| Question |
Describe the MapReduce model |
1
|
| Question |
Describe tasks management |
1
|
| Question |
Describe patterns of usage |
1
|
| Question |
Solve with MapReduce a specific problem |
0
|
| Question |
Implement a usage pattern |
0
|
Section 4
| Activity Type |
Content |
Is Graded?
|
| Question |
Analyze the CAP theorem |
1
|
| Question |
Define the kinds of data storage available |
1
|
| Question |
Characteristics of stream processing |
1
|
| Question |
Describe the usage patterns |
1
|
| Question |
Build a program to solve a problem with stream processing |
0
|
| Question |
Interact with a NoSQL database |
0
|
Section 5
| Activity Type |
Content |
Is Graded?
|
| Question |
Features of SparkML |
1
|
| Question |
Features of GraphX |
1
|
| Question |
Write a program using SparkML |
0
|
| Question |
Write a program using GraphX |
0
|
Final assessment
Section 1
- Design the structure of a cloud architecture for a specific analytics type
- Give examples of the 6 Vs in real systems
Section 2
- Configure a HDFS cluster with some specific replication approaches
- Build a HDFS client
- Evaluate the performance of a specific configuration
- Compare the different schedules
Section 3
- Describe the advantages and disadvantages of the MapReduce model
- Solve a task designing the solution using MapReduce
- Solve a task designing the solution using a composition of usage patterns
Section 4
- Identify problems and solutions related to the CAP theorem
- Compare solutions with batch and stream processing approaches
- Design a system using a NoSQL database
Section 5
- Extend the SparkML library with a custom algorithm
- Extend the GraphX library with a custom algorithm
The retake exam
Section 1
Section 2
Section 3
Section 4
Section 5