Machine Learning and Signal Processing Applications of Fixed Point Theory
28 03 2007Danilo Mandic, Imperial College London and Isao Yamada, Tokyo Institute of Technology
Abstract
This tutorial is aimed at providing a unifying and rigorous basis for linking of different parts of signal processing and related disciplines under one umbrella. Case studies within this tutorial will address problems from several overlapping areas: nonlinear and linear adaptive filtering, image processing, estimation theory, communications, automatic control, and machine learning.
Topics which will be covered in this tutorial include:
Danilo Mandic:
- Background theory and motivation
- History of fixed point theory
- Modern fixed point theory
- Random contraction mapping theorems
- Scope and perspective
- Unifying framework for the analysis of linear adaptive filtering problems
- Contraction Mapping Theorem (CMT) and Fixed Point Iteration (FPI) as a tool for the unifying analysis and interpretation of the adaptive filtering problem
- Application in standard acoustic echo cancellation
- Applications in relaxation of autonomous systems
- Analysis of nonlinear problems within the framework of fixed point theory
- Feedback, nonlinearity, recurrence and adaptivity
- Operation and stability of nonlinear adaptive filters
- Nonlinear dynamics, modularity, nesting, bifurcations, and chaos
- Contraction mapping interpretation of learning algorithms in machine learning and adaptive filtering
- Data reusing and the trade-off between recursive and iterative learning
- Blind source separation and extraction
- Statistical signal processing: an interpretation of the EM algorithm
- Turbo decoding
- Two dimensional problems: Fixed point theory for image processing and pattern recognition
- Colour image retrieval and object recognition
- Projection on convex sets (POCS) and image restoration
- Tracking and stability issue in autonomous systems
- Trajectory tracking as an example of fixed point theory
- Learning dynamics, association and generalisation
Isao Yamada
- Fixed point theory of nonexpansive mappings
- Convexity and fixed point theory
- Properties of nonexpansive mappings and their use in practical applications
- Design of mappings having a desired fixed point set
- Overview of fixed point algorithms for nonexpansive and quasi-nonexpansive mappings
- Mann algorithm (POCS will be stated here as an example),
- Haugazeau’s algorithm and generalization
- Modern algorithms: Anchor method, Hybrid Steepest Descent method
- Adaptive filtering paradigm revisited: advanced concepts
- Monotone approximation property of NLMS, APA
- Unified view of Adaptive Projected Subgradient Method
- Applications to convexly constrained inverse problems
- Generalization of the Projected Landweber method
- Nonsmooth constrained problems
- Applications
- Interference suppression in CDMA systems
- Advanced concepts in acoustic echo cancellation
- Other machine learning applications
Targeted Audience
The tutorial will be self-contained and will enable researchers and engineers to apply the concept immediately to their own problems. The material in the tutorial is suitable for generally knowledgeable audience, with basic knowledge of linear algebra, adaptive filtering and spectrum estimation.
Speaker Biographies
Dr. Mandic (www.commsp.ee.ic.ac.uk/~mandic) received the Ph.D. degree in nonlinear adaptive signal processing in 1999 from Imperial College. He is a Reader with the Department of Electrical and Electronic Engineering, Imperial College London, U.K.
He has written over 150 publications on a variety of aspects of signal processing, and has been a Guest Professor at the Catholic University Leuven, Belgium, Westminster University UK, and at TUAT Japan, and Frontier Researcher at the Brain Science Institute RIKEN, Tokyo, Japan. Dr. Mandic has been a Member of the IEEE Signal Processing Society Technical Committee on Machine Learning for Signal Processing, Associate Editor for IEEE Transactions on Circuits and Systems II, and Associate Editor for International Journal of Mathematical Modeling and Algorithms. He has won awards for his papers and for the products coming from his collaboration with industry.
His work on the applications of fixed point theory in engineering has been widely published, and some of the concepts relevant to this tutorial can be found in his research monographs Recurrent Neural Networks for Prediction (with J. Chambers), Wiley 2001, and Complex Valued Nonlinear Adaptive Filters (with S. L. Goh), Wiley 2007.
Dr. Yamada (www.comm.ss.titech.ac.jp/~isao/index.html) obtained his PhD degree in nonlinear signal processing in 1990 from Tokyo Institute of Technology, Tokyo, Japan. He is now an Associate Professor with the Department of Communications and Integrated Systems, Tokyo Institute of Technology, Tokyo, Japan. He has written over 150 publications on multidimensional, adaptive and array signal processing, optimization theory, and nonlinear inverse problems. Dr Yamada has been a Visiting Associate Professor at Pennsylvania State University, and has received awards for his research papers. He received the IEICE Young Researcher award in 1992, the ICF research award in 2004, and the DoCoMo mobile science award in 2005. Dr Yamada has served as an Associate Editor for IEEE Transactions on Circuits and Systems I, and for the International Journal on Multidimensional Systems and Signal Processing.
His pioneering “Hybrid Steepest Descent” fixed point algorithm has been included in mathematical monographs Inherently Parallel Algorithms for Feasibility and Optimization and Their Applications, Elsevier 2001 and Inverse Problem, Image Analysis and Medical Imaging, (Contemporary Mathematics 313) AMS 2002.
