SLT 2022 Notes

Attending an Offline Conference Again 


I was fortunate to have an opportunity to attend the Spoken Language Technology workshop 2022, held from 9th to 12th January in Doha, Qatar. I presented my final PhD work at the conference. Going back to an (almost) in-person conference was so refreshing and I learned so many things.

In this post, I'll summarise my takeaways from the SLT 2022 conference. The photo below is the conference venue, Marsa Malaz Kempinski. 

Human Speech Processing and Neural Networks

As a linguistics student (back in 2012), I've been always keen to learn studies on relevance of human language processing to computer speech processing. On this topic, I remember one of the keynote presentations and one poster.

The first keynote talk by Nima Mesgarani was interesting, mentioning that the shallow layers of an RNN-T encoder learns acoustic and phonetic information occurring in short time frames while the later layers capture lexical and semantic information which can happen across longer time frames. This apparently corresponds to human brain processing of speech. Coincidentally, the poster I saw, Phoneme Segmentation using Self-supervised Speech Models by Strgar and Harwath, also had a similar observation that more attention weights go to the shallow layers of wav2vec2.0 and HuBERT for phone processing. 

Multimodality

It was nice to see many works on multimodality. I found the following works very interesting.
Speech and NLP models are becoming better and better thanks to a humongous amount of data but still lacking understanding of cross-modality unlike humans. I am wondering about the real potential of multimodal processing if more amount of data is available. 

Data Domain Shift and ASR

ASR systems are often trained and evaluated on the same data domain but real world ASR applications can face "unexpected" including accent, background noise and other obstacles which ASR is not trained on. 

Relating to that, this is my favourite poster at the conference "How does pre-trained wav2vec2.0 perform on domain-shifted ASR? An extensive benchmark on air traffic control communications" by Zuluaga-Gomez et al. They investigated the effects of data domain-shift on the Kaldi-based system and the wav2vec2.0-CTC system. They found that just about 10 hours of speech of the air traffic control domain was sufficient to fine-tune the wav2vec2.0 system for decent quality transcripts. 

Doha


The SLT conference happened right after the World Cup 2022 in Qatar and Doha was ready to accommodate visitors. I had no problem finding English speaking people for help. The place I'd love to visit again is Souq Waqif, the local market which has literally every thing one can imagine. 

 
The left photo was taken in the West Bay district and the right was the Museum of Islamic Art. 

Comments

Post a Comment

Popular posts from this blog

How wav2vec2.0 takes input audio data

Image
Introduction Building an automatic speech recognition (ASR) system has become easier than ever thanks to the availability of deep learning and ASR toolkits. These toolkits magically pre-process an audio file and a neural model can seamlessly take processed audio features, letting users needing not to know much details. This is my motivation to write a summary of how audio files are processed for a neural model. In this post, I will focus on  wav2vec2.0  proposed by Baevski et al. which learns speech representations in a self-supervised manner. This  post  from Hugging Face is an excellent tutorial showing how to fine-tune wav2vec2.0 for the ASR task. I would like to provide information focusing only on input structure of wav2vec2.0 to compliment the tutorial. Fig1: Audio input processing of wav2vec2.0 discussed in this post is the part circled by a red line. This figure is taken from Baevski et
Read more

Setup git pre-push hook

Motivation I often forget to run tests before pushing my commits. This is a bad practice. Someone might use my broken code in a collaborative environment. I was looking for a way to automatically run tests before creating a commit or pushing commits. The solution is Git Hooks:  Customizing Git Hooks . Git Hooks Many hooks can be setup in a Git repository. For example, Git Hooks can run before creating a commit ( pre-commit ), after creating a commit ( post-commit ) and before pushing commits to a remote branch ( pre-push ). All of these are client-side hooks and client-side hooks are not shared across local repositories. The other type of hooks are server-side hooks  that run after commits are pushed to a remote server and policy is imposed on all users of the repository. I have setup  pre-push  in a Git repository to avoid forgetting to run tests before sending updates to the remote server. Setup pre-push The way to setup  pre-push  is as follows:
Read more

Understanding kaldi lattice-prune and beam

Introduction For a long time, I have been thinking that Kaldi lattice-prune  uses a beam which keeps the specified number of candidates alive. How dare me. I finally took a time to properly understand this.  There are several excellent Kaldi notes (for example Josh Meyer's blog ), but I could not find information specifically about  lattice-prune . My intension of this blog post is to leave information about Kaldi lattice-prune and --beam (assuming that someone uses Kaldi or hybrid ASR systems in 2022). TL;DR  lattice-prune --beam=4  for example deletes all paths of a lattice that exceed <best_path_cost> + 4, as can be seen in this line of the source code . Explanation To understand how Kaldi beam works, I took one of the utterances in the LibriSpeech dev set: 1272-128104-0000 . The lattice of this utterance should be in  exp/chain_cleaned/tdnn_1d_sp/decode_dev_clean_tgsmall/lat.1.gz  once the script  kaldi/egs/librispeech/
Read more