Media Informatics Systems

Week 1:

Basic definitions. Media content management. Main topics of multimedia (image, voice, video) processing.

Week 2:

Processing audio contents. Short Time Fourier Spectrum, windowing, spectrogram computation. The fundamentals of signal detection.

Week 3:

Music recognition. Challenges in real-time pattern matching: additive noise, linear and non-linear distortions. Audio fingerprinting. Case study: Shazam.

Week 4:

Recognition of variable sound and image signals. Statistics based general classification. Probability density function, likelihood, training and test. Bayes' theorem.

Week 5:

Multimedia classification tasks based on multi-variate Gaussian Mixture Models.

Week 6:

Maximum Likelihood vs. Discriminative Training – theoretical and practical issues regarding audio and image data. The application of Multi Layer Perceptrons on media informatics tasks.

Week 7:

Processing time-varying signals: Dynamic Time Warping, Hidden Markov-Models and their practical implementations. Fundamentals of speech recognition.

Week 8:

Contemporary speech recognition technologies. Acoustic, pronunciation and language models. Subtitling methods and standards. Case studies: BBC and the Hungarian National Television subtitling approaches.

Week 9:

Deep learning: deep feed-forward neural nets, convolutional nets and requrrent networks - and their applications on automatic annotation of media contents (text, sound, image and video).

Week 10:

Complex tasks based on multimedia technologies: shape detection in images, object tracking in videos. Face detection and recognition solutions. Video processing methods in practice: hand gesture recognition in videos.

Week 11:

Metadata: semantic and desriptive metadata. Multimedia metadata standards. EBU/SMPTE metadata, Dublin Core, Material Exchange Format (MXF).

Week 12:

Multimedia databases. Multimedia information retrieval. Search modes, types, algorithms. Quality measurement of an information retrieval system.

Week 13:

Digital Media Management Systems (DMMS) / Multimedia Asset Management. Structure of the systems: gathering, storing, displaying subsystems. Lifecycle attributes. Integration tools. Architecture of the media content management systems, and types of it: DAM, DM, KM, Web CMS, ECM. Overall model of systems.

Week 14:

Digital archiving: tasks, approaches, technics. Archiving strategies: On-line, near-line, off-line, off-site accessibility, levels. Record management.

In the practice sessions, the following topics (not exculsively and not fully) will be discussed: audio feature extraction, audio fingerprinting, GMM, automatic speech recognition, language modeling, applied deep neural nets in Keras, image annotation tasks, multimedia retrieval, visualization tools, video processing.