BSc: Natural Language Processing - Revision history

V.ivanov: /* Section 1 */

2023-01-10T16:09:19Z

Section 1

← Older revision		Revision as of 16:09, 10 January 2023
Line 173:		Line 173:
	\| Question \|\| What is perplexity in language modeling? \|\| 1		\| Question \|\| What is perplexity in language modeling? \|\| 1
	\|-		\|-
−	\| Question \|\| What is ~~anti-aliasing~~ and why is ~~anti-aliasing~~ used in language modeling? \|\| 0	+	\| Question \|\| What is smoothing and why is smoothing used in language modeling? \|\| 0
−	\|}	+	\|}
		+
	==== Section 2 ====		==== Section 2 ====
	{\| class="wikitable"		{\| class="wikitable"

V.ivanov: /* The retake exam */

2023-01-10T16:08:42Z

The retake exam

← Older revision		Revision as of 16:08, 10 January 2023
Line 232:		Line 232:
	# What is self-attention?		# What is self-attention?
	# Explain the Transformer architecture.		# Explain the Transformer architecture.
−
−	=== The retake exam ===
−	'''Section 1'''
−
−	'''Section 2'''
−
−	'''Section 3'''
−
−	'''Section 4'''

V.ivanov: /* Course activities and grading breakdown */

2023-01-10T16:06:03Z

Course activities and grading breakdown

← Older revision		Revision as of 16:06, 10 January 2023
Line 101:		Line 101:
	! Activity Type !! Percentage of the overall course grade		! Activity Type !! Percentage of the overall course grade
	\|-		\|-
−	\| ~~Midterm~~ \|\| 30	+	\| Final Exam \|\| 30
	\|-		\|-
−	\| Final project \|\| 40	+	\| Final project \|\| 30
	\|-		\|-
−	\| Assignments \|\| 20	+	\| Assignments \|\| 30
	\|-		\|-
	\| Lab Participation / Quizzes \|\| 10		\| Lab Participation / Quizzes \|\| 10

Ai.valeev: /* Level 1: What concepts should a student know/remember/explain? */

2022-12-29T17:25:30Z

Level 1: What concepts should a student know/remember/explain?

← Older revision		Revision as of 17:25, 29 December 2022
Line 58:		Line 58:

	==== Level 1: What concepts should a student know/remember/explain? ====		==== Level 1: What concepts should a student know/remember/explain? ====
−	By the end of the course, the students should know:	+	By the end of the course, the students should know ...
−	* Fundamental approaches to text analysis	+	* Fundamental approaches to text analysis;
	* Various natural language processing algorithms;		* Various natural language processing algorithms;
	* Ways to measure the performance of NLP systems;		* Ways to measure the performance of NLP systems;

Ai.valeev at 17:22, 29 December 2022

2022-12-29T17:22:41Z

← Older revision		Revision as of 17:22, 29 December 2022
Line 9:		Line 9:

	== Prerequisites ==		== Prerequisites ==
		+
		+	As an undergraduate level course, the students are expected to have basic understanding of
		+	probability, linear algebra, programming in python and basics of machine learning.

	=== Prerequisite subjects ===		=== Prerequisite subjects ===
		+	* CSE101 — Introduction to Programming I
		+	* CSE202 — Analytical Geometry and Linear Algebra I
		+	* CSE206 — Probability And Statistics
	* CSE302 — Introduction to Machine Learning		* CSE302 — Introduction to Machine Learning
		+
		+	=== Prerequisite topics ===

	== Course Topics ==		== Course Topics ==
Line 115:		Line 123:

	=== Open access resources ===		=== Open access resources ===
		+	* Handbook Of Natural Language Processing, Second Edition Chapman & Hall Crc Machine Learning & Pattern Recognition 2010
	* Clark, Alexander, Chris Fox, and Shalom Lappin, eds. The handbook of computational linguistics and natural language processing. John Wiley & Sons, 2013.		* Clark, Alexander, Chris Fox, and Shalom Lappin, eds. The handbook of computational linguistics and natural language processing. John Wiley & Sons, 2013.
	* Dan Jurafsky and James H. Martin. Speech and Language Processing (3rd ed.)		* Dan Jurafsky and James H. Martin. Speech and Language Processing (3rd ed.)
Line 158:		Line 167:
	! Activity Type !! Content !! Is Graded?		! Activity Type !! Content !! Is Graded?
	\|-		\|-
−	\| Question \|\| ? \|\| 1	+	\| Question \|\| Explain the Chomsky hierarchy \|\| 1
	\|-		\|-
−	\| Question \|\| ? \|\| 1	+	\| Question \|\| What is a language model? \|\| 1
		+	\|-
		+	\| Question \|\| What is perplexity in language modeling? \|\| 1
		+	\|-
		+	\| Question \|\| What is anti-aliasing and why is anti-aliasing used in language modeling? \|\| 0
	\|}		\|}
	==== Section 2 ====		==== Section 2 ====
Line 168:		Line 181:
	! Activity Type !! Content !! Is Graded?		! Activity Type !! Content !! Is Graded?
	\|-		\|-
−	\| Question \|\| ? \|\| 1	+	\| Question \|\| Explain the MaxEnt principle in a modeling language \|\| 1
	\|-		\|-
		+	\| Question \|\| Describe examples of NLP applications and approaches to assessing their quality \|\| 1
−	\| Question \|\| ? \|\| 1
	\|}		\|}
	==== Section 3 ====		==== Section 3 ====
Line 178:		Line 191:
	! Activity Type !! Content !! Is Graded?		! Activity Type !! Content !! Is Graded?
	\|-		\|-
		+	\| Question \|\| Why use an encoder-decoder model rather than a regular RNN for automatic translation? \|\| 1
−	\| Question \|\| ? \|\| 1
		+	\|-
		+	\| Question \|\| How to handle variable length input sequences with RNN? \|\| 1
		+	\|-
		+	\| Question \|\| What is beam search and why is it used? \|\| 1
		+	\|-
		+	\| Question \|\| Describe the components of the encoder-decoder model \|\| 1
		+	\|}
		+	==== Section 4 ====
		+	{\| class="wikitable"
		+	\|+
		+	\|-
		+	! Activity Type !! Content !! Is Graded?
		+	\|-
		+	\| Question \|\| What is the most important layer in the Transformer architecture? What is its purpose? \|\| 1
		+	\|-
		+	\| Question \|\| Describe the architecture of BERT and the training process for this model. \|\| 1
	\|-		\|-
		+	\| Question \|\| Describe the architecture of GPT (version 2 or 3) and the training process for this model. \|\| 1
−	\| Question \|\| ? \|\| 1
	\|}		\|}
	=== Final assessment ===		=== Final assessment ===
	'''Section 1'''		'''Section 1'''
		+	# Suppose you run a dependency parser (shift-reduce) with a standard arc system for a sentence of length n. How many shift operations are needed?
−	# ?
		+	# Describe the main differences between formal languages (such as logic or programming languages) and natural languages (such as Russian).
		+	# Give an equation to find the most likely sequence of part-of-speech (POS) tags that can be used by a stochastic POS tagger. Assuming a bigram model.
		+	# Explain what is meant by the terms smoothing and backoff in the context of the stochastic POS tagger model.
	'''Section 2'''		'''Section 2'''
		+	# Suppose you classify text based on a bag of words in a document. The raw input is a single line containing the text of the entire document. Describe in one or two sentences the pipeline from the raw input to the feature vector.
−	# ?
		+	# Suppose you have a neural network that matches the training data exactly. Describe two ways to solve this problem.
		+	# Compare different methods for text processing (decision trees, hidden markov models, support vector machines).
	'''Section 3'''		'''Section 3'''
		+	# How would you classify different RNN architectures?
−	# ?
		+	# You train a neural network with the Adam optimizer and observe the negative log probability on the training set over epochs. Instead of decreasing, it seems to fluctuate around where it started. What could you add to your training routine to fix this?
		+	# Describe the reason why the negative sampling skipgram model learns faster than the base skipgram model.
	'''Section 4'''		'''Section 4'''
		+	# Compare BERT architectures with GPT (version 2 or 3). Explain the disadvantages and advantages of each.
−	# ?
		+	# What contextual representations do you know for words?
		+	# What is model distillation? How is it performed?
		+	# Explain the attention mechanism.
		+	# What is self-attention?
		+	# Explain the Transformer architecture.

	=== The retake exam ===		=== The retake exam ===

Ai.valeev at 16:56, 29 December 2022

2022-12-29T16:56:59Z

Show changes

N.askarbekuly: Blanked the page

2022-04-20T20:53:52Z

Blanked the page

Show changes

M.petrishchev: M.petrishchev moved page BSc:NaturalLanguageProcessing to BSc: Natural Language Processing

2022-04-18T08:54:17Z

M.petrishchev moved page BSc:NaturalLanguageProcessing to BSc: Natural Language Processing

← Older revision	Revision as of 08:54, 18 April 2022
(No difference)

M.petrishchev: M.petrishchev moved page BSc:NaturalLanguageProcessing.S22 to BSc:NaturalLanguageProcessing

2022-04-03T13:57:26Z

M.petrishchev moved page BSc:NaturalLanguageProcessing.S22 to BSc:NaturalLanguageProcessing

← Older revision	Revision as of 13:57, 3 April 2022
(No difference)

M.petrishchev: Created page with "= Natural Language Processing = * '''Course name:''' Natural Language Processing * '''Course number:''' XYZ * '''Subject area:''' Progr..."

2022-03-03T08:13:17Z

Created page with "= Natural Language Processing = * '''Course name:''' Natural Language Processing * '''Course number:''' XYZ * '''Subject area:''' Progr..."

New page

= Natural Language Processing =

* '''Course name:''' Natural Language Processing
* '''Course number:''' XYZ
* '''Subject area:''' Programming Languages and Software Engineering
* '''Version:''' Spring 2022

== Course characteristics ==

=== Key concepts of the class ===

*
*
*
*
*
*
*
*
*

=== What is the purpose of this course? ===

The Natural Language Processing course teaches

=== - What should a student remember at the end of the course? ===

*
*

=== - What should a student be able to understand at the end of the course? ===

* How to create high quality software using mainstream concepts of programming.
* What is object-oriented programming and its main advantages
* How to increase the level of abstraction with help of genericity.
* How to create concurrent programs and what are the main issues related to this kind of programming

=== - What should a student be able to apply at the end of the course? ===

* To be able to create quality programs in Java.

=== Course evaluation ===

{|
|+ Course grade breakdown
!
!
!align="center"| '''Proposed points'''
|-
| Labs/seminar classes
| 40
|align="center"| 40
|-
| Interim performance assessment
| 30
|align="center"| 30
|-
| Exams
| 30
|align="center"| 30
|}

If necessary, please indicate freely your course’s features in terms of students’ performance assessment:

==== Labs/seminar classes: ====

* In-class participation 1 point for each individual contribution in a class but not more than 1 point a week (i.e. 14 points in total for 14 study weeks),
* overall course contribution (to accumulate extra-class activities valuable to the course progress, e.g. a short presentation, book review, very active in-class participation, etc.) up to 6 points.

==== Interim performance assessment: ====

* in-class tests up to 10 points for each test (i.e. up to 40 points in total for 2 theory and 2 practice tests),
* computational practicum assignment up to 10 points for each task (i.e. up to 30 points for 3 tasks).

==== Exams: ====

* mid-term exam up to 30 points,
* final examination up to 30 points.

==== Overall score: ====

100 points (100%).

=== Grades range ===

{|
|+ Course grading range
!
!
!align="center"| '''Proposed range'''
|-
| A. Excellent
| 85-100
|align="center"| 85-100
|-
| B. Good
| 75-84
|align="center"| 75-84
|-
| C. Satisfactory
| 60-75
|align="center"| 60-75
|-
| D. Poor
| 0-59
|align="center"| 0-59
|}

If necessary, please indicate freely your course’s grading features:

* A: more than 85 of the overall score;
* B: at least 85 of the overall score;
* C: at least 75 of the overall score;
* D: less than 60 of the overall score.

=== Resources and reference material ===

==== Textbook: ====

*
*

== Course Sections ==

The main sections of the course and approximate hour distribution between them is as follows:

{|
|+ Course Sections
|align="center"| '''Section'''
| '''Section Title'''
|align="center"| '''Lectures'''
|align="center"| '''Seminars'''
|align="center"| '''Self-study'''
|align="center"| '''Knowledge'''
|-
|align="center"| '''Number'''
|
|align="center"| '''(hours)'''
|align="center"| '''(labs)'''
|align="center"|
|align="center"| '''evaluation'''
|-
|align="center"| 1
| Introduction to programming
|align="center"| 12
|align="center"| 6
|align="center"| 12
|align="center"| 2
|-
|align="center"| 2
| Introduction to object-oriented programming
|align="center"| 8
|align="center"| 4
|align="center"| 8
|align="center"| 1
|-
|align="center"| 3
| Introduction to generics, exception handling and programming by contract (C)
|align="center"| 8
|align="center"| 4
|align="center"| 8
|align="center"| 1
|-
|align="center"| 4
| Introduction to programming environments
|align="center"| 12
|align="center"| 6
|align="center"| 12
|align="center"| 2
|-
|align="center"| 5
| Introduction to concurrent and functional programming
|align="center"| 8
|align="center"| 4
|align="center"| 8
|align="center"| 1
|-
|align="center"| Final examination
|
|align="center"|
|align="center"|
|align="center"|
|align="center"| 2
|}

=== Section 1 ===

==== Section title: ====

Introduction to programming

=== Topics covered in this section: ===

* Basic definitions – algorithm, program, computer, von Neumann architecture, CPU lifecycle.
* Programming languages history and overview. Imperative (procedural) and functional approaches.
* Translation – compilation vs. interpretation. JIT, AOT. Hybrid modes.
* Introduction to typification. Static and dynamic typing. Type inference. Basic types – integer, real, character, boolean, bit. Arrays and strings. Records-structures.
* Programming – basic concepts. Statements and expressions. 3 atomic statements - assignment, if-check, goto. Control structures – conditional, assignment, goto, case-switch-inspect, loops.
* Variables and constants.
* Routines – procedures and functions.

=== What forms of evaluation were used to test students’ performance in this section? ===

<div class="tabular">

|a|c| & '''Yes/No''' 
Development of individual parts of software product code & 0 
Homework and group projects & 1 
Midterm evaluation & 1 
Testing (written or computer based) & 1 
Reports & 0 
Essays & 0 
Oral polls & 1 
Discussions & 1 

</div>
=== Typical questions for ongoing performance evaluation within this section ===

# What is the difference between compiler and interpreter?
# What is the difference between type and variable?
# What is the background of structured programming?

=== Typical questions for seminar classes (labs) within this section ===

# How to compile a program?
# How to run a program?
# How to debug a program?

=== Test questions for final assessment in this section ===

# What are the basic control structure of structured programming?
# What is the difference between statements and expressions?
# What are the benefits of type inference?

=== Section 2 ===

==== Section title: ====

Introduction to object-oriented programming

=== Topics covered in this section: ===

* Key principles of object-oriented programming
* Overloading is not overriding
* Concepts of class and object
* How objects can be created?
* Single and multiple inheritance

=== What forms of evaluation were used to test students’ performance in this section? ===

<div class="tabular">

|a|c| & '''Yes/No''' 
Development of individual parts of software product code & 1 
Homework and group projects & 1 
Midterm evaluation & 1 
Testing (written or computer based) & 0 
Reports & 1 
Essays & 0 
Oral polls & 1 
Discussions & 1 

</div>
=== Typical questions for ongoing performance evaluation within this section ===

# What is the meaning of polymorphism?
# How to check the dynamic type of an object?
# What are the limitations of single inheritance?
# What are the issues related with multiple inheritance?

=== Typical questions for seminar classes (labs) within this section ===

# How to handle array of objects of some class type?
# How to implement the class which logically has to have 2 constructors with the same signature but with different semantics?

=== Test questions for final assessment in this section ===

# Name all principles of object-oriented programming?
# Explain what conformance means?
# Explain why cycles are prohibited in the inheritance graph?

=== Section 3 ===

==== Section title: ====

Introduction to generics, exception handling and programming by contract (C)

=== Topics covered in this section: ===

* Introduction to generics
* Introduction to exception handling
* Introduction to programming by contract (C)

=== What forms of evaluation were used to test students’ performance in this section? ===

<div class="tabular">

|a|c| & '''Yes/No''' 
Development of individual parts of software product code & 0 
Homework and group projects & 1 
Midterm evaluation & 1 
Testing (written or computer based) & 0 
Reports & 0 
Essays & 0 
Oral polls & 1 
Discussions & 1 

</div>
=== Typical questions for ongoing performance evaluation within this section ===

# What is constrained genericity?
# What is exception?
# What is assertion?

=== Typical questions for seminar classes (labs) within this section ===

# How constrained genericity may be used for sorting of objects?
# In which order catch blocks are being processed?
# Where is the problem when precondition is violated?

=== Test questions for final assessment in this section ===

# Can array be treated as generic class?
# What is the difference between throw and throws in Java?
# What is purpose of the class invariant?

=== Section 4 ===

==== Section title: ====

Introduction to programming environments

=== Topics covered in this section: ===

* Concept of libraries as the basis for reuse.
* Concept of interfaces/API. Separate compilation.
* Approaches to software documentation.
* Persistence. Files.
* How to building a program. Recompilation problem. Name clashes, name spaces

=== What forms of evaluation were used to test students’ performance in this section? ===

<div class="tabular">

|a|c| & '''Yes/No''' 
Development of individual parts of software product code & 0 
Homework and group projects & 1 
Midterm evaluation & 0 
Testing (written or computer based) & 1 
Reports & 0 
Essays & 0 
Oral polls & 1 
Discussions & 1 

</div>
=== Typical questions for ongoing performance evaluation within this section ===

# How reuse helps to develop software?
# How concept of libraries and separate compilation co-relate?
# What are the benefits of integrating documentation into the source code?
# Why is it essential to have persistent data structures?

=== Typical questions for seminar classes (labs) within this section ===

# What is to be done to design and develop a library?
# How to add documenting comments into the source code?
# What ways exists in Java to support persistence ?

=== Test questions for final assessment in this section ===

# How to deal with name clashes?
# What is the main task of the recompilation module?
# What are the differences between different formats of persistence files?

=== Section 5 ===

==== Section title: ====

Introduction to concurrent and functional programming

=== Topics covered in this section: ===

* Concurrent programming.
* Functional programming within imperative programming languages.

=== What forms of evaluation were used to test students’ performance in this section? ===

<div class="tabular">

|a|c| & '''Yes/No''' 
Development of individual parts of software product code & 0 
Homework and group projects & 1 
Midterm evaluation & 0 
Testing (written or computer based) & 1 
Reports & 0 
Essays & 0 
Oral polls & 1 
Discussions & 1 

</div>
=== Typical questions for ongoing performance evaluation within this section ===

# Explain the key differences parallelism and concurrency
# What are the key issues related to parallel execution?
# What are the models of parallel execution?
# What is the difference between function and object?

=== Typical questions for seminar classes (labs) within this section ===

# Which Java construction support concurrency?
# What is a thread?
# What is in-line lambda function?

=== Test questions for final assessment in this section ===

# What is the meaning of SIMD and MIMD?
# What are the implications of the Amdahl’s law?
# What model of concurrency Java relies on?
# Which function can be considered as pure?
# How to declare a function to accept a functional object as its argument?
# How Java supports high-order functions?
# How capturing variables works in Java?