publications

2022

1. ICLR

   

   Transformer-based Transform Coding

   Zhu, Yinhao*,  Yang, Yang*, and Cohen, Taco

   In International Conference on Learning Representations (ICLR) 2022

   TLDR Abs HTML

   Neural data compression based on nonlinear transform coding has made great progress over the last few years, mainly due to improvements in prior models, quantization methods and nonlinear transforms. A general trend in many recent works pushing the limit of rate-distortion performance is to use ever more expensive prior models that can lead to prohibitively slow decoding. Instead, we focus on more expressive transforms that result in a better rate-distortion-computation trade-off. Specifically, we show that nonlinear transforms built on Swin-transformers can achieve better compression efficiency than transforms built on convolutional neural networks (ConvNets), while requiring fewer parameters and shorter decoding time. Paired with a compute-efficient Channel-wise Auto-Regressive Model prior, our SwinT-ChARM model outperforms VTM-12.1 by 3.68 % in BD-rate on Kodak with comparable decoding speed. In P-frame video compression setting, we are able to outperform the popular ConvNet-based scale-space-flow model by 12.35 % in BD-rate on UVG. We provide model scaling studies to verify the computational efficiency of the proposed solutions and conduct several analyses to reveal the source of coding gain of transformers over ConvNets, including better spatial decorrelation, flexible effective receptive field, and more localized response of latent pixels during progressive decoding.

   A swin-transformer based auto-encoder structure is proposed that achieves SOTA in rate-distortion-complexity trade-off of image compression. It also outperforms SSF in video compression. AFAIK SwinT-ChARM is the first neural image codec that outperforms VTM in rate-distortion while with comparable decoding time on GPU.

2021

1. ICIP

   

   Progressive Neural Image Compression With Nested Quantization And Latent Ordering

   Lu, Yadong*, Zhu, Yinhao*,  Yang, Yang*, Said, Amir, and Cohen, Taco S

   In IEEE International Conference on Image Processing (ICIP) 2021

   TLDR Abs HTML

   We present PLONQ, a progressive neural image compression scheme which pushes the boundary of variable bitrate compression by allowing quality scalable coding with a single bitstream. In contrast to existing learned variable bitrate solutions which produce separate bitstreams for each quality, it enables easier rate-control and requires less storage. Leveraging the latent scaling based variable bitrate solution, we introduce nested quantization, a method that defines multiple quantization levels with nested quantization grids, and progressively refines all latents from the coarsest to the finest quantization level. To achieve finer progressiveness in between any two quantization levels, latent elements are incrementally refined with an importance ordering defined in the rate-distortion sense. To the best of our knowledge, PLONQ is the first learning-based progressive image coding scheme and it outperforms SPIHT, a well-known wavelet-based progressive image codec.

   A prefix-decodable bitstream (lower bitrate stream is embedded as a prefix of higher bitrate stream) is obtained with nested quantization and per-element sorting by prior stddev, based on the hyperprior model.

2. JSP

   

   Transform Network Architectures for Deep Learning Based End-to-End Image/Video Coding in Subsampled Color Spaces

   Egilmez, Hilmi, Singh, Ankitesh Kumar, Coban, Muhammed, Karczewicz, Marta, Zhu, Yinhao,  Yang, Yang, Said, Amir, and Cohen, Taco

   In IEEE Open Journal of Signal Processing 2021

   TLDR Abs PDF

   Most of the existing deep learning based end-to-end image/video coding (DLEC) architectures are designed for non-subsampled RGB color format. However, in order to achieve a superior coding performance, many state-of-the-art block-based compression standards such as High Efficiency Video Coding (HEVC/H.265) and Versatile Video Coding (VVC/H.266) are designed primarily for YUV 4:2:0 format, where U and V components are subsampled by considering the human visual system. This paper investigates various DLEC designs to support YUV 4:2:0 format by comparing their performance against the main profiles of HEVC and VVC standards under a common evaluation framework. Moreover, a new transform network architecture is proposed to improve the efficiency of coding YUV 4:2:0 data. The experimental results on YUV 4:2:0 datasets show that the proposed architecture significantly outperforms naive extensions of existing architectures designed for RGB format and achieves about 10% average BD-rate improvement over the intra-frame coding in HEVC.

   PReLU can replace GDN in the hyperprior model to compress YUV (and RGB!) images without loss of coding gain.

3. CVPR-W

   

   Phase Selective Convolution

   Lin, Jamie Menjay, Noorzad, Parham,  Yang, Yang, Kwak, Nojun, and Porikli, Fatih

   In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops on Embedded Vision 2021

   TLDR Abs HTML

   This paper introduces Phase Selective Convolution (PSC), an enhanced convolution for more deliberate utilization of activations in convolutional networks. Unlike conventional use of convolutions with activation functions, PSC preserves the full space of activations while supporting desirable model nonlinearity. Similar to several other network operations, e.g., the ReLU operation, at the time of their introduction, PSC may not execute as efficiently on platforms without hardware specialization support. As a first step in addressing the need for optimization, we propose a hardware acceleration scheme to enable the intended efficiency for PSC execution. Moreover, we propose a PSC deployment strategy, with which PSC is applied only to selected layers of the networks, to avoid excessive increase in the total model size. To evaluate the results, we apply PSC as a drop-in replacement for selected convolution layers in several networks without affecting their macro network architectures. In particular, PSC-enhanced ResNets achieve higher accuracies by 1.0-2.0% and 0.7-1.0% on CIFAR-100 and ImageNet, respectively, in Pareto efficiency. PSC-enhanced MobileNets (V2 and V3 Large) and MobileNetV3 (Small) achieve 0.9-1.0% and 1.8% accuracy gains, respectively, on ImageNet at little (0.2-0.7%) total model size increase.

   A simple and effective extension of Conv+ReLU that can achieve better pareto efficiency in image classification when compared with ResNet and MobileNets

2020

1. ICASSP

   

   Feedback Recurrent Autoencoder

   Yang, Yang, Sautière, Guillaume, Ryu, J. Jon, and Cohen, Taco S

   In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2020

   TLDR Abs HTML

   In this work, we propose a new recurrent autoencoder architecture, termed Feedback Recurrent AutoEncoder (FRAE), for online compression of sequential data with temporal dependency. The recurrent structure of FRAE is designed to efficiently extract the redundancy along the time dimension and allows a compact discrete representation of the data to be learned. We demonstrate its effectiveness in speech spectrogram compression. Specifically, we show that the FRAE, paired with a powerful neural vocoder, can produce high-quality speech waveforms at a low, fixed bitrate. We further show that by adding a learned prior for the latent space and using an entropy coder, we can achieve an even lower variable bitrate.

   Extend VAE to sequential setting for efficient representation learning of sequential data. In the application of speech compression, when paired with a neural vocoder, the proposed FRAE signifantly outperforms Opus, an open source speech codec.

2. ACCV

   

   Feedback Recurrent Autoencoder for Video Compression

   Golinski, Adam*, Pourreza, Reza*,  Yang, Yang*, Sautière, Guillaume, and Cohen, Taco S

   In Asian Conference on Computer Vision (ACCV) 2020

   TLDR Abs HTML

   Recent advances in deep generative modeling have enabled efficient modeling of high dimensional data distributions and opened up a new horizon for solving data compression problems. Specifically, autoencoder based learned image or video compression solutions are emerging as strong competitors to traditional approaches. In this work, We propose a new network architecture, based on common and well studied components, for learned video compression operating in low latency mode. Our method yields competitive MS-SSIM/rate performance on the high-resolution UVG dataset, among both learned video compression approaches and classical video compression methods (H.265 and H.264) in the rate range of interest for streaming applications. Additionally, we provide an analysis of existing approaches through the lens of their underlying probabilistic graphical models.Finally, we point out issues with temporal consistency and color shift observed in empirical evaluation, and suggest directions forward to alleviate those.

   One of the early works that propose neural network based video codecs. A recurrent layer is introduced in the decoder to capture long-range redundancy of video content, the state of which is fed back to encoder.

3. CVPR

   

   Guided Variational Autoencoder for Disentanglement Learning

   Ding, Zheng*, Xu, Yifan*, Xu, Weijian, Parmar, Gaurav,  Yang, Yang, Welling, Max, and Tu, Zhuowen

   In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2020

   TLDR Abs HTML

   We propose an algorithm, guided variational autoencoder (Guided-VAE), that is able to learn a controllable generative model by performing latent representation disentanglement learning. The learning objective is achieved by providing signal to the latent encoding/embedding in VAE without changing its main backbone architecture, hence retaining the desirable properties of the VAE. We design an unsupervised and a supervised strategy in Guided-VAE and observe enhanced modeling and controlling capability over the vanilla VAE. In the unsupervised strategy, we guide the VAE learning by introducing a lightweight decoder that learns latent geometric transformation and principal components; in the supervised strategy, we use an adversarial excitation and inhibition mechanism to encourage the disentanglement of the latent variables. Guided-VAE enjoys its transparency and simplicity for the general representation learning task, as well as disentanglement learning. On a number of experiments for representation learning, improved synthesis/sampling, better disentanglement for classification, and reduced classification errors in meta learning have been observed.

   An auxiliary decoder is used to improve the disentanglement of latent from a VAE.

4. ICML-W

   Deep Generative Video Compression with Temporal Autoregressive Transforms

   Yang, Ruihan, Yang, Yibo, Marino, Joseph,  Yang, Yang, and Mandt, Stephan

   In ICML 2020 Workshop on Invertible Neural Networks, Normalizing Flows, andExplicit Likelihood Models 2020

   PDF

2019

1. ICASSP

   

   Automatic Grammar Augmentation for Robust Voice Command Recognition

   Yang, Yang, Lalitha, Anusha, Lee, Jinwon, and Lott, Chris

   In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2019

   TLDR Abs HTML

   This paper proposes a novel pipeline for automatic grammar augmentation that provides a significant improvement in the voice command recognition accuracy for systems with small footprint acoustic model (AM). The improvement is achieved by augmenting the user-defined voice command set, also called grammar set, with alternate grammar expressions. For a given grammar set, a set of potential grammar expressions (candidate set) for augmentation is constructed from an AM-specific statistical pronunciation dictionary that captures the consistent patterns and errors in the decoding of AM induced by variations in pronunciation, pitch, tempo, accent, ambiguous spellings, and noise conditions. Using this candidate set, greedy optimization based and cross-entropy-method (CEM) based algorithms are considered to search for an augmented grammar set with improved recognition accuracy utilizing a command-specific dataset. Our experiments show that the proposed pipeline along with algorithms considered in this paper significantly reduce the mis-detection and mis-classification rate without increasing the false-alarm rate. Experiments also demonstrate the consistent superior performance of CEM method over greedy-based algorithms.

   Improve voice command recognition of a light-weight acoustic model by augmenting the target command to capture variations of inputs.

2. INFOCOM

   Joint Antenna Allocation and Link Scheduling in FlexRadio Networks

   Qian, Zhenzhi,  Yang, Yang, Srinivasan, Kannan, and Shroff, Ness B.

   In IEEE INFOCOM 2019

   HTML

2016

1. MobiHoc

   Anonymous-Query Based Rate Control for Wireless Multicast: Approaching Optimality with Constant Feedback

   Wu, Fei,  Yang, Yang, Zhang, Ouyang, Srinivasan, Kannan, and Shroff, Ness B.

   In 17th ACM International Symposium on Mobile Ad Hoc Networking and Computing (MobiHoc) 2016

   Abs HTML

   For a multicast group of n receivers, existing techniques either achieve high throughput at the cost of prohibitively large (e.g., O(n)) feedback overhead, or achieve low feedback overhead but without either optimal or near-optimal throughput guarantees. Simultaneously achieving good throughput guarantees and low feedback overhead has been an open problem and could be the key reason why wireless multicast has not been successfully deployed in practice. In this paper, we develop a novel anonymous-query based rate control, which approaches the optimal throughput with a constant feedback overhead independent of the number of receivers. In addition to our theoretical results, through implementation on a software-defined ratio platform, we show that the anonymous-query based algorithm achieves low-overhead and robustness in practice.

2015

1. JSAC

   Constant-Delay and Constant-Feedback Moving Window Network Coding for Wireless Multicast: Design and Asymptotic Analysis

   Wu, Fei, Sun, Yin,  Yang, Yang, Srinivasan, Kannan, and Shroff, Ness B.

   IEEE Journal on Selected Areas in Communications 2015

   HTML
2. INFOCOM

   Scheduling in wireless networks with full-duplex cut-through transmission

   Yang, Yang, and Shroff, Ness B.

   In IEEE INFOCOM 2015

   HTML
3. CDC

   Dynamic user association and energy control in cellular networks with renewable resources

   Yang, Yang, Liu, Jiashang, Sinha, Prasun, and Shroff, Ness B.

   In 54th IEEE Conference on Decision and Control (CDC) 2015

   HTML

2014

1. ToIT

   Delay Asymptotics With Retransmissions and Incremental Redundancy Codes Over Erasure Channels

   Yang, Yang, Tan, Jian, Shroff, Ness B., and Gamal, Hesham El

   IEEE Transactions on Information Theory 2014

   HTML
2. INFOCOM

   Characterizing the achievable throughput in wireless networks with two active RF chains

   Yang, Yang, Chen, Bo, Srinivasan, Kannan, and Shroff, Ness B.

   In IEEE INFOCOM 2014

   HTML
3. WiOpt

   A near-optimal randomized algorithm for uplink resource allocation in OFDMA systems

   Yang, Yang, Nam, Changwon, and Shroff, Ness B.

   In 12th International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt) 2014

   HTML

2012

1. MobiHoc

   Throughput of Rateless Codes over Broadcast Erasure Channels

   Yang, Yang, and Shroff, Ness

   In Thirteenth ACM International Symposium on Mobile Ad Hoc Networking and Computing (MobiHoc) 2012

   Abs HTML

   In this paper, we characterize the throughput of a broadcast network with n receivers using rateless codes with block size K. We assume that the underlying channel is a Markov modulated erasure channel that is i.i.d. across users, but can be correlated in time. We characterize the system throughput asymptotically in n. Specifically, we explicitly show how the throughput behaves for different values of the coding block size K as a function of n, as n approaches infinity. Under the more restrictive assumption of memoryless channels, we are able to provide a lower bound on the maximum achievable throughput for any finite values of K and n. Using simulations we show the tightness of the bound with respect to system parameters n and K, and find that its performance is significantly better than the previously known lower bound.

2011

1. INFOCOM

   Delay asymptotics with retransmissions and fixed rate codes over erasure channels

   Tan, Jian,  Yang, Yang, Shroff, Ness B., and El Gamal, Hesham

   In IEEE INFOCOM 2011

   HTML