Natural Language Processing II (EN)

Description

In this course, we will follow up on the basic Natural Language Processing course with more advanced topics.

We will mainly focus on data preprocessing and the latest applications of deep learning in text processing. It will mainly be about the architecture of neural networks built on the so-called Transformers. Using the transfer learning method, we will show how large pre-trained neural networks can be used for various practical applications.

Contents

• Character encoding and unicode normalization
• Traditional tokenization (simple methods, Spacy, Moses)
• Subword tokenization (byte-pair encoding, wordpiece, sentencepiece)
• Data cleaning (deduplication, removal of text ballast)

• General principles
• Implementation of the skip-gram model

• LSTM and GRU memory cells
• Implementation of machine translation using recurrent networks

• Attention is all you need
• Transformer architecture
• GPT2
• BERT
• XLNET

• Classification of texts
• Recognizing named entities
• Question answering

Prerequisites

• Basic knowledge of Python programming
• High School Mathematics
• Course-level knowledge of machine learning Introduction to Machine Learning
• Course-level knowledge Natural Language Processing