Charalambos Charalambous - Profile on Academia.edu (original) (raw)

Papers by Charalambos Charalambous

Research paper thumbnail of Directed Information on Abstract Spaces: Properties and Variational Equalities

arXiv (Cornell University), Feb 16, 2013

Directed information or its variants are utilized extensively in the characterization of the capa... more Directed information or its variants are utilized extensively in the characterization of the capacity of channels with memory and feedback, nonanticipative lossy data compression, and their generalizations to networks. In this paper, we derive several functional and topological properties of directed information for general abstract alphabets (complete separable metric spaces) using the topology of weak convergence of probability measures. These include convexity of the set of consistent distributions, which uniquely define causally conditioned distributions, convexity and concavity of directed information with respect to the sets of consistent distributions, weak compactness of these sets of distributions, their joint distributions and their marginals. Furthermore, we show lower semicontinuity of directed information, and under certain conditions we also establish continuity of directed information. Finally, we derive variational equalities for directed information, including sequential versions. These may be viewed as the analogue of the variational equalities of mutual information (utilized in Blahut-Arimoto algorithm). In summary, we extend the basic functional and topological properties of mutual information to directed information. These properties are discussed in the context of extremum problems of directed information.

Research paper thumbnail of An upper bound to zero-delay rate distortion via Kalman filtering for vector Gaussian sources

We deal with zero-delay source coding of a vector Gaussian autoregressive (AR) source subject to ... more We deal with zero-delay source coding of a vector Gaussian autoregressive (AR) source subject to an average mean squared error (MSE) fidelity criterion. Toward this end, we consider the nonanticipative rate distortion function (NRDF) which is a lower bound to the causal and zero-delay rate distortion function (RDF). We use the realization scheme with feedback proposed in [1] to model the corresponding optimal "testchannel" of the NRDF, when considering vector Gaussian AR(1) sources subject to an average MSE distortion. We give conditions on the vector Gaussian AR(1) source to ensure asymptotic stationarity of the realization scheme (bounded performance). Then, we encode the vector innovations due to Kalman filtering via lattice quantization with subtractive dither and memoryless entropy coding. This coding scheme provides a tight upper bound to the zero-delay Gaussian RDF. We extend this result to vector Gaussian AR sources of any finite order. Further, we show that for infinite dimensional vector Gaussian AR sources of any finite order, the NRDF coincides with the zero-delay RDF. Our theoretical framework is corroborated with a simulation example.

Research paper thumbnail of Infinite Horizon Average Cost Dynamic Programming Subject to Total Variation Distance Ambiguity

arXiv (Cornell University), Dec 21, 2015

We analyze the infinite horizon minimax average cost Markov Control Model (MCM), for a class of c... more We analyze the infinite horizon minimax average cost Markov Control Model (MCM), for a class of controlled process conditional distributions, which belong to a ball, with respect to total variation distance metric, centered at a known nominal controlled conditional distribution with radius R ∈ [0, 2], in which the minimization is over the control strategies and the maximization is over conditional distributions. Upon performing the maximization, a dynamic programming equation is obtained which includes, in addition to the standard terms, the oscillator semi-norm of the cost-to-go. First, the dynamic programming equation is analyzed for finite state and control spaces. We show that if the nominal controlled process distribution is irreducible, then for every stationary Markov control policy the maximizing conditional distribution of the controlled process is also irreducible for R ∈ [0, Rmax]. Second, the generalized dynamic programming is analyzed for Borel spaces. We derive necessary and sufficient conditions for any control strategy to be optimal. Through our analysis, new dynamic programming equations and new policy iteration algorithms are derived. The main feature of the new policy iteration algorithms (which are applied for finite alphabet spaces) is that the policy evaluation and policy improvement steps are performed by using the maximizing conditional distribution, which is obtained via a water filling solution. Finally, the application of the new dynamic programming equations and the corresponding policy iteration algorithms are shown via illustrative examples.

Research paper thumbnail of Capacity Achieving Distributions & Information Lossless Randomized Strategies for Feedback Channels with Memory: The LQG Theory of Directed Information-Part II

arXiv (Cornell University), Apr 4, 2016

A methodology is developed to realized optimal channel input conditional distributions, which max... more A methodology is developed to realized optimal channel input conditional distributions, which maximize the finite-time horizon directed information, for channels with memory and feedback, by information lossless randomized strategies. The methodology is applied to general Time-Varying Multiple Input Multiple Output (MIMO) Gaussian Linear Channel Models (G-LCMs) with memory, subject to average transmission cost constraints of quadratic form. The realizations of optimal distributions by randomized strategies are shown to exhibit a decomposion into a deterministic part and a random part. The decomposition reveals the dual role of randomized strategies, to control the channel output process and to transmit new information over the channels. Moreover, a separation principle is shown between the computation of the optimal deterministic part and the random part of the randomized strategies. The dual role of randomized strategies generalizes the Linear-Quadratic-Gaussian (LQG) stochastic optimal control theory to directed information pay-offs. The characterizations of feedback capacity are obtained from the per unit time limits of finite-time horizon directed information, without imposingá priori assumptions, such as, stability of channel models or ergodicity of channel input and output processes. For time-invariant MIMO G-LCMs with memory, it is shown that whether feedback increases capacity, is directly related to the channel parameters and the transmission cost function, through the solutions of Riccati matrix equations, and moreover for unstable channels, feedback capacity is non-zero, provided the power exceeds a critical level.

Research paper thumbnail of A General Formula for Compound Channel Capacity

IEEE Transactions on Information Theory, Jul 1, 2016

A general formula for the capacity of arbitrary compound channels with the receiver channel state... more A general formula for the capacity of arbitrary compound channels with the receiver channel state information is obtained using the information density approach. No assumptions of ergodicity, stationarity or information stability are made and the channel state set is arbitrary. A direct (constructive) proof is given. To prove achievability, we generalize Feinstein Lemma to the compound channel setting, and to prove converse, we generalize Verdu-Han Lemma to the same compound setting. A notion of a uniform compound channel is introduced and the general formula is shown to reduce to the familiar sup − inf expression for such channels. As a by-product, the arbitrary varying channel capacity is established under maximum error probability and deterministic coding. Conditions are established under which the worst-case and compound channel capacities are equal so that the full channel state information at the transmitter brings in no advantage. The compound inf-information rate plays a prominent role in the general formula. Its properties are studied and a link between information-unstable and information-stable regimes of a compound channel is established. The results are extended to include ε-capacity of compound channels. Sufficient and necessary conditions for the strong converse to hold are given.

Research paper thumbnail of Information Structures of Extremum Problems of Feedback Capacity for General Channels with Memory

arXiv (Cornell University), Apr 4, 2016

For any class of channel conditional distributions, with finite memory dependence on channel inpu... more For any class of channel conditional distributions, with finite memory dependence on channel input RVs A n = {A i : i = 0,. .. , n} or channel output RVs B n = {B i : i = 0,. .. , n} or both, we characterize the subsets of channel input distributions P CI [0,n] ⊆ P [0,n] = P A i |A i−1 ,B i−1 : i = 1,. .. , n , which satisfy conditional independence on past information, and maximize directed information defined by I(A n → B n) = n ∑ i=0 I(A i ; B i |B i−1) and we derive the corresponding expressions, called "characterizations of Finite Transmission Feedback Information (FTFI) capacity". We derive similar characterizations, when general transmission cost constraints are imposed. Moreover, we also show that the structural properties apply to general nonlinear and linear autoregressive channel models defined by discrete-time recursions on general alphabet spaces, and driven by arbitrary distributed noise processes. We derive these structural properties by invoking stochastic optimal control theory and variational equalities of directed information, to identify tight upper bounds on I(A n → B n), which are achievable over subsets of conditional independence distributions P CI [0,n] ⊆ P [0,n] and specified by the dependence of channel distributions and transmission cost functions on inputs and output symbols. We apply the characterizations to recursive Multiple Input Multiple Output Gaussian Linear Channel Models with limited memory on channel input and output sequences. The structural properties of optimal channel input distributions, generalize the structural properties of Memoryless Channels with feedback, to any channel distribution with memory, and settle various long standing problems in information theory.

Research paper thumbnail of Variable Length Lossless Coding for Variational Distance Class: An Optimal Merging Algorithm

arXiv (Cornell University), Feb 1, 2012

In this paper we consider lossless source coding for a class of sources specified by the total va... more In this paper we consider lossless source coding for a class of sources specified by the total variational distance ball centred at a fixed nominal probability distribution. The objective is to find a minimax average length source code, where the minimizers are the codeword lengths-real numbers for arithmetic or Shannon codes-while the maximizers are the source distributions from the total variational distance ball. Firstly, we examine the maximization of the average codeword length by converting it into an equivalent optimization problem, and we give the optimal codeword lenghts via a waterfilling solution. Secondly, we show that the equivalent optimization problem can be solved via an optimal partition of the source alphabet, and re-normalization and merging of the fixed nominal probabilities. For the computation of the optimal codeword lengths we also develop a fast algorithm with a computational complexity of order O(n). I. INTRODUCTION Lossless fixed to variable length source codes are often categorized into problems of known source probability distribution and unknown source probability distribution. For known source T. Charalambous was with the

Research paper thumbnail of Causal Rate Distortion Function and Relations to Filtering Theory

arXiv (Cornell University), Apr 5, 2011

A causal rate distortion function (RDF) is defined, existence of extremum solution is described v... more A causal rate distortion function (RDF) is defined, existence of extremum solution is described via weak *-convergence, and its relation to filtering theory is discussed. The relation to filtering is obtained via a causal constraint imposed on the reconstruction kernel to be realizable while the extremum solution is given for the stationary case.

Research paper thumbnail of Single Letter Expression of Capacity for a Class of Channels with Memory

arXiv (Cornell University), Jan 4, 2017

We study finite alphabet channels with Unit Memory on the previous Channel Outputs called UMCO ch... more We study finite alphabet channels with Unit Memory on the previous Channel Outputs called UMCO channels. We identify necessary and sufficient conditions, to test whether the capacity achieving channel input distributions with feedback are time-invariant, and whether feedback capacity is characterized by single letter, expressions, similar to that of memoryless channels. The method is based on showing that a certain dynamic programming equation, which in general, is a nested optimization problem over the sequence of channel input distributions, reduces to a non-nested optimization problem. Moreover, for UMCO channels, we give a simple expression for the ML error exponent, and we identify sufficient conditions to test whether feedback does not increase capacity. We derive similar results, when transmission cost constraints are imposed. We apply the results to a special class of the UMCO channels, the Binary State Symmetric Channel (BSSC) with and without transmission cost constraints, to show that the optimization problem of feedback capacity is non-nested, the capacity achieving channel input distribution and the corresponding channel output transition probability distribution are time-invariant, and feedback capacity is characterized by a single letter formulae, precisely as Shannon's single letter characterization of capacity of memoryless channels. Then we derive closed form expressions for the capacity achieving channel input distribution and feedback capacity. We use the closed form expressions to evaluate an error exponent for ML decoding.

Research paper thumbnail of Optimal Nonanticipative Estimation Schemes for Time-Varying Gauss-Markov Processes

arXiv (Cornell University), Mar 14, 2016

In this paper, we derive recursive filters for time-varying multidimensional Gauss-Markov process... more In this paper, we derive recursive filters for time-varying multidimensional Gauss-Markov processes, which satisfy a mean square error fidelity, using the concept of Finite Time Horizon (FTH) Nonanticipative Rate Distortion Function (NRDF) and its connection to real-time realizable filtering theory. Moreover, we derive a universal lower bound on the mean square error of any estimator of time-varying multidimensional Gauss-Markov processes in terms of conditional mutual information. Unlike classical Kalman filters, the proposed filter is constructed from the solution of a reverse-waterfilling problem, which ensures that the mean square error fidelity is met. Our theoretical results are demonstrated via illustrative examples.

Research paper thumbnail of Capacity of the Binary State Symmetric Channel with Cost Constraint

arXiv (Cornell University), Jan 31, 2014

We consider a unit memory channel, called Binary State Symmetric Channel (BSSC), in which the cha... more We consider a unit memory channel, called Binary State Symmetric Channel (BSSC), in which the channel state is the modulo2 addition of the current channel input and the previous channel output. We derive closed form expressions for the capacity and corresponding channel input distribution, of this BSSC with and without feedback and transmission cost. We also show that the capacity of the BSSC is not increased by feedback, and it is achieved by a first order symmetric Markov process.

Research paper thumbnail of Nonanticipative Rate Distortion Function and Filtering Theory: A weak Convergence Approach

arXiv (Cornell University), Dec 29, 2012

In this paper the relation between nonanticipative rate distortion function (RDF) and Bayesian fi... more In this paper the relation between nonanticipative rate distortion function (RDF) and Bayesian filtering theory is further investigated on general Polish spaces. The relation is established via an optimization on the space of conditional distributions of the so-called directed information subject to fidelity constraints. Existence of the optimal reproduction distribution of the nonanticipative RDF is shown using the topology of weak convergence of probability measures. Subsequently, we use the solution of the nonanticipative RDF to present the realization of a multidimensional partially observable source over a scalar Gaussian channel. We show that linear encoders are optimal, establishing joint source-channel coding in real-time.

Research paper thumbnail of Capacity of Binary State Symmetric Channel with and without feedback and transmission cost

We consider a unit memory channel, called Binary State Symmetric Channel (BSSC), in which the cha... more We consider a unit memory channel, called Binary State Symmetric Channel (BSSC), in which the channel state is the modulo2 addition of the current channel input and the previous channel output. We derive closed form expressions for the capacity and corresponding channel input distribution, of this BSSC with and without feedback and transmission cost. We also show that the capacity of the BSSC is not increased by feedback, and it is achieved by a first order symmetric Markov process.

Research paper thumbnail of State-Dependent Generalizations of Nonanticipatory Epsilon Entropy of Partially Observable Processes

2022 IEEE 61st Conference on Decision and Control (CDC), Dec 6, 2022

Two fundamental generalizations of Gorbunov's and Pinsker's nonanticipatory ϵ−entropy are formula... more Two fundamental generalizations of Gorbunov's and Pinsker's nonanticipatory ϵ−entropy are formulated and analyzed for a tuple of partially observable, finite-dimensional, random processes (X n , Y n), where X n △ = {X1,. .. , Xn} is the unobserved state process and Y n △ = {Y1,. .. , Yn} is the observable process-a noisy version of X n , subject to a fidelity between X n , and its reproduction X n △ = { X1,. .. , Xn}. The encoder observes causally Y n and past reproductions X n may or may not be available to both the encoder and the decoder. Theorem 1 gives a tight lower bound on the operational rate of zero-delay codes, when X n is causally available to the decoder only, in terms of a state-dependent nonanticipatory ϵ−entropy of a state process Z n , which is fundamentally different from a corresponding nonanticipatory ϵ−entropy, when X n is causally available to both the encoder and the decoder. Theorem 2 identifies sufficient conditions for the two nonanticipatory ϵ−entropies to coincide. Theorem 3 identifies the information structure of the optimal test-channel distributions. The paper also discusses applications to jointly Gaussian partially observable processes (X n , Y n) with a square-error fidelity criterion, and derives characterizations of the two nonanticipatory ϵ−entropies.

Research paper thumbnail of Dynamic Programming Subject to Total Variation Distance Ambiguity

arXiv (Cornell University), Feb 5, 2014

The aim of this paper is to address optimality of stochastic control strategies via dynamic progr... more The aim of this paper is to address optimality of stochastic control strategies via dynamic programming subject to total variation distance ambiguity on the conditional distribution of the controlled process. We formulate the stochastic control problem using minimax theory, in which the control minimizes the pay-off while the conditional distribution, from the total variation distance set, maximizes it. First, we investigate the maximization of a linear functional on the space of probability measures on abstract spaces, among those probability measures which are within a total variation distance from a nominal probability measure, and then we give the maximizing probability measure in closed form. Second, we utilize the solution of the maximization to solve minimax stochastic control with deterministic control strategies, under a Markovian and a non-Markovian assumption, on the conditional distributions of the controlled process. The results of this part include: 1) Minimax optimization subject to total variation distance ambiguity constraint; 2) new dynamic programming recursions, which involve the oscillator seminorm of the value function, in addition to the standard terms; 3) new infinite horizon discounted dynamic programming equation, the associated contractive property, and a new policy iteration algorithm. Finally, we provide illustrative examples for both the finite and infinite horizon cases. For the infinite horizon case we invoke the new policy iteration algorithm to compute the optimal strategies.

Research paper thumbnail of Rate Distortion Function for a Class of Relative Entropy Sources

arXiv (Cornell University), May 6, 2013

This paper deals with rate distortion or source coding with fidelity criterion, in measure spaces... more This paper deals with rate distortion or source coding with fidelity criterion, in measure spaces, for a class of source distributions. The class of source distributions is described by a relative entropy constraint set between the true and a nominal distribution. The rate distortion problem for the class is thus formulated and solved using minimax strategies, which result in robust source coding with fidelity criterion. It is shown that minimax and maxmin strategies can be computed explicitly, and they are generalizations of the classical solution. Finally, for discrete memoryless uncertain sources, the rate distortion theorem is stated for the class omitting the derivations while the converse is derived. I. INTRODUCTION This paper is concerned with lossy data compression for a class of sources defined on the space of probability distributions on general alphabet spaces. In the classical rate distortion formulation with the fidelity decoding criterion, Shannon has shown that minimization of mutual information between finite alphabet source and reproduction sequences subject to fidelity criterion over the reproduction kernel has an operational meaning. Hence, it gives the minimum amount of information of representing a source symbol by a reproduction symbol with a pre-specified fidelity or distortion criterion. The classical rate distortion function for finite-alphabet and continuous sources has been studied thoroughly in the literature [1], [2], [3], [4] and [5]. A survey of the theory of rate distortion is given in [4]. The formulation of rate distortion function for abstract alphabets is investigated by Csiszár in [5]. Specifically, in [5] the question of existence of solution in Polish spaces under some continuity assumptions on the distortion function and compactness of the reproduction space, is established under the topology of weak convergence. The formulation in [5] is based on two important assumptions, namely, 1) compactness of the reproduction space, 2) absolute continuity of all marginal distributions with respect to the optimal marginal distribution. The compactness assumption is crucial in order to formulate the problem using countably additive measures, and to show existence of the minimizing measure using The research leading to these results has received funding from the European Community's Seventh Framework Programme (FP7/2007-2013) under grant agreement no. INFSO-ICT-223844.

Research paper thumbnail of Variational Equalities of Directed Information and Applications

arXiv (Cornell University), Jan 28, 2013

In this paper we introduce two variational equalities of directed information, which are analogou... more In this paper we introduce two variational equalities of directed information, which are analogous to those of mutual information employed in the Blahut-Arimoto Algorithm (BAA). Subsequently, we introduce nonanticipative Rate Distortion Function (RDF) R na 0,n (D) defined via directed information introduced in [1], and we establish its equivalence to Gorbunov-Pinsker's nonanticipatory ǫ-entropy R ε 0,n (D). By invoking certain results we first establish existence of the infimizing reproduction distribution for R na 0,n (D), and then we give its implicit form for the stationary case. Finally, we utilize one of the variational equalities and the closed form expression of the optimal reproduction distribution to provide an algorithm for the computation of R na 0,n (D).

Research paper thumbnail of On the Capacity of Gaussian MIMO Channels With Memory

IEEE Communications Letters, Aug 1, 2022

The operational capacity of Gaussian MIMO channels with memory was obtained by Brandenburg and Wy... more The operational capacity of Gaussian MIMO channels with memory was obtained by Brandenburg and Wyner in [9] under certain mild assumptions on the channel impulse response and its noise covariance matrix, which essentuially require channel memory to be not too strong. This channel was also considered by Tsybakov in [10] and its information capacity was obtained in some cases. It was further conjectured, based on numerical evidence, that these capacities are the same in all cases. This conjecture is proved here. An explicit closed-form expression for the optimal input power spectral density matrix is also given. The obtained result is further extended to the case of joint constraints, including per-antenna and interference power constraints as well as energy harvesting constraints. These results imply the information-theoretic optimality of OFDM-type transmission systems for such channels with memory.

Research paper thumbnail of Sequential Necessary and Sufficient Conditions for Capacity Achieving Distributions of Channels With Memory and Feedback

IEEE Transactions on Information Theory, Nov 1, 2017

We derive sequential necessary and sufficient conditions for any channel input conditional distri... more We derive sequential necessary and sufficient conditions for any channel input conditional distribution

Research paper thumbnail of Generalizations of Nonanticipative Rate Distortion Function to Multivariate Nonstationary Gaussian Autoregressive Processes

2019 IEEE 58th Conference on Decision and Control (CDC)

The characterizations of nonanticipative rate distortion function (NRDF) on a finite horizon are ... more The characterizations of nonanticipative rate distortion function (NRDF) on a finite horizon are generalized to nonstationary multivariate Gaussian order L autoregressive, AR(L), source processes, with respect to mean square error (MSE) distortion functions. It is shown that the optimal reproduction distributions are induced by a reproduction process, which is a linear function of the state of the source, its best mean-square error estimate, and a Gaussian random process.

Research paper thumbnail of Directed Information on Abstract Spaces: Properties and Variational Equalities

arXiv (Cornell University), Feb 16, 2013

Directed information or its variants are utilized extensively in the characterization of the capa... more Directed information or its variants are utilized extensively in the characterization of the capacity of channels with memory and feedback, nonanticipative lossy data compression, and their generalizations to networks. In this paper, we derive several functional and topological properties of directed information for general abstract alphabets (complete separable metric spaces) using the topology of weak convergence of probability measures. These include convexity of the set of consistent distributions, which uniquely define causally conditioned distributions, convexity and concavity of directed information with respect to the sets of consistent distributions, weak compactness of these sets of distributions, their joint distributions and their marginals. Furthermore, we show lower semicontinuity of directed information, and under certain conditions we also establish continuity of directed information. Finally, we derive variational equalities for directed information, including sequential versions. These may be viewed as the analogue of the variational equalities of mutual information (utilized in Blahut-Arimoto algorithm). In summary, we extend the basic functional and topological properties of mutual information to directed information. These properties are discussed in the context of extremum problems of directed information.

Research paper thumbnail of An upper bound to zero-delay rate distortion via Kalman filtering for vector Gaussian sources

We deal with zero-delay source coding of a vector Gaussian autoregressive (AR) source subject to ... more We deal with zero-delay source coding of a vector Gaussian autoregressive (AR) source subject to an average mean squared error (MSE) fidelity criterion. Toward this end, we consider the nonanticipative rate distortion function (NRDF) which is a lower bound to the causal and zero-delay rate distortion function (RDF). We use the realization scheme with feedback proposed in [1] to model the corresponding optimal "testchannel" of the NRDF, when considering vector Gaussian AR(1) sources subject to an average MSE distortion. We give conditions on the vector Gaussian AR(1) source to ensure asymptotic stationarity of the realization scheme (bounded performance). Then, we encode the vector innovations due to Kalman filtering via lattice quantization with subtractive dither and memoryless entropy coding. This coding scheme provides a tight upper bound to the zero-delay Gaussian RDF. We extend this result to vector Gaussian AR sources of any finite order. Further, we show that for infinite dimensional vector Gaussian AR sources of any finite order, the NRDF coincides with the zero-delay RDF. Our theoretical framework is corroborated with a simulation example.

Research paper thumbnail of Infinite Horizon Average Cost Dynamic Programming Subject to Total Variation Distance Ambiguity

arXiv (Cornell University), Dec 21, 2015

We analyze the infinite horizon minimax average cost Markov Control Model (MCM), for a class of c... more We analyze the infinite horizon minimax average cost Markov Control Model (MCM), for a class of controlled process conditional distributions, which belong to a ball, with respect to total variation distance metric, centered at a known nominal controlled conditional distribution with radius R ∈ [0, 2], in which the minimization is over the control strategies and the maximization is over conditional distributions. Upon performing the maximization, a dynamic programming equation is obtained which includes, in addition to the standard terms, the oscillator semi-norm of the cost-to-go. First, the dynamic programming equation is analyzed for finite state and control spaces. We show that if the nominal controlled process distribution is irreducible, then for every stationary Markov control policy the maximizing conditional distribution of the controlled process is also irreducible for R ∈ [0, Rmax]. Second, the generalized dynamic programming is analyzed for Borel spaces. We derive necessary and sufficient conditions for any control strategy to be optimal. Through our analysis, new dynamic programming equations and new policy iteration algorithms are derived. The main feature of the new policy iteration algorithms (which are applied for finite alphabet spaces) is that the policy evaluation and policy improvement steps are performed by using the maximizing conditional distribution, which is obtained via a water filling solution. Finally, the application of the new dynamic programming equations and the corresponding policy iteration algorithms are shown via illustrative examples.

Research paper thumbnail of Capacity Achieving Distributions & Information Lossless Randomized Strategies for Feedback Channels with Memory: The LQG Theory of Directed Information-Part II

arXiv (Cornell University), Apr 4, 2016

A methodology is developed to realized optimal channel input conditional distributions, which max... more A methodology is developed to realized optimal channel input conditional distributions, which maximize the finite-time horizon directed information, for channels with memory and feedback, by information lossless randomized strategies. The methodology is applied to general Time-Varying Multiple Input Multiple Output (MIMO) Gaussian Linear Channel Models (G-LCMs) with memory, subject to average transmission cost constraints of quadratic form. The realizations of optimal distributions by randomized strategies are shown to exhibit a decomposion into a deterministic part and a random part. The decomposition reveals the dual role of randomized strategies, to control the channel output process and to transmit new information over the channels. Moreover, a separation principle is shown between the computation of the optimal deterministic part and the random part of the randomized strategies. The dual role of randomized strategies generalizes the Linear-Quadratic-Gaussian (LQG) stochastic optimal control theory to directed information pay-offs. The characterizations of feedback capacity are obtained from the per unit time limits of finite-time horizon directed information, without imposingá priori assumptions, such as, stability of channel models or ergodicity of channel input and output processes. For time-invariant MIMO G-LCMs with memory, it is shown that whether feedback increases capacity, is directly related to the channel parameters and the transmission cost function, through the solutions of Riccati matrix equations, and moreover for unstable channels, feedback capacity is non-zero, provided the power exceeds a critical level.

Research paper thumbnail of A General Formula for Compound Channel Capacity

IEEE Transactions on Information Theory, Jul 1, 2016

A general formula for the capacity of arbitrary compound channels with the receiver channel state... more A general formula for the capacity of arbitrary compound channels with the receiver channel state information is obtained using the information density approach. No assumptions of ergodicity, stationarity or information stability are made and the channel state set is arbitrary. A direct (constructive) proof is given. To prove achievability, we generalize Feinstein Lemma to the compound channel setting, and to prove converse, we generalize Verdu-Han Lemma to the same compound setting. A notion of a uniform compound channel is introduced and the general formula is shown to reduce to the familiar sup − inf expression for such channels. As a by-product, the arbitrary varying channel capacity is established under maximum error probability and deterministic coding. Conditions are established under which the worst-case and compound channel capacities are equal so that the full channel state information at the transmitter brings in no advantage. The compound inf-information rate plays a prominent role in the general formula. Its properties are studied and a link between information-unstable and information-stable regimes of a compound channel is established. The results are extended to include ε-capacity of compound channels. Sufficient and necessary conditions for the strong converse to hold are given.

Research paper thumbnail of Information Structures of Extremum Problems of Feedback Capacity for General Channels with Memory

arXiv (Cornell University), Apr 4, 2016

For any class of channel conditional distributions, with finite memory dependence on channel inpu... more For any class of channel conditional distributions, with finite memory dependence on channel input RVs A n = {A i : i = 0,. .. , n} or channel output RVs B n = {B i : i = 0,. .. , n} or both, we characterize the subsets of channel input distributions P CI [0,n] ⊆ P [0,n] = P A i |A i−1 ,B i−1 : i = 1,. .. , n , which satisfy conditional independence on past information, and maximize directed information defined by I(A n → B n) = n ∑ i=0 I(A i ; B i |B i−1) and we derive the corresponding expressions, called "characterizations of Finite Transmission Feedback Information (FTFI) capacity". We derive similar characterizations, when general transmission cost constraints are imposed. Moreover, we also show that the structural properties apply to general nonlinear and linear autoregressive channel models defined by discrete-time recursions on general alphabet spaces, and driven by arbitrary distributed noise processes. We derive these structural properties by invoking stochastic optimal control theory and variational equalities of directed information, to identify tight upper bounds on I(A n → B n), which are achievable over subsets of conditional independence distributions P CI [0,n] ⊆ P [0,n] and specified by the dependence of channel distributions and transmission cost functions on inputs and output symbols. We apply the characterizations to recursive Multiple Input Multiple Output Gaussian Linear Channel Models with limited memory on channel input and output sequences. The structural properties of optimal channel input distributions, generalize the structural properties of Memoryless Channels with feedback, to any channel distribution with memory, and settle various long standing problems in information theory.

Research paper thumbnail of Variable Length Lossless Coding for Variational Distance Class: An Optimal Merging Algorithm

arXiv (Cornell University), Feb 1, 2012

In this paper we consider lossless source coding for a class of sources specified by the total va... more In this paper we consider lossless source coding for a class of sources specified by the total variational distance ball centred at a fixed nominal probability distribution. The objective is to find a minimax average length source code, where the minimizers are the codeword lengths-real numbers for arithmetic or Shannon codes-while the maximizers are the source distributions from the total variational distance ball. Firstly, we examine the maximization of the average codeword length by converting it into an equivalent optimization problem, and we give the optimal codeword lenghts via a waterfilling solution. Secondly, we show that the equivalent optimization problem can be solved via an optimal partition of the source alphabet, and re-normalization and merging of the fixed nominal probabilities. For the computation of the optimal codeword lengths we also develop a fast algorithm with a computational complexity of order O(n). I. INTRODUCTION Lossless fixed to variable length source codes are often categorized into problems of known source probability distribution and unknown source probability distribution. For known source T. Charalambous was with the

Research paper thumbnail of Causal Rate Distortion Function and Relations to Filtering Theory

arXiv (Cornell University), Apr 5, 2011

A causal rate distortion function (RDF) is defined, existence of extremum solution is described v... more A causal rate distortion function (RDF) is defined, existence of extremum solution is described via weak *-convergence, and its relation to filtering theory is discussed. The relation to filtering is obtained via a causal constraint imposed on the reconstruction kernel to be realizable while the extremum solution is given for the stationary case.

Research paper thumbnail of Single Letter Expression of Capacity for a Class of Channels with Memory

arXiv (Cornell University), Jan 4, 2017

We study finite alphabet channels with Unit Memory on the previous Channel Outputs called UMCO ch... more We study finite alphabet channels with Unit Memory on the previous Channel Outputs called UMCO channels. We identify necessary and sufficient conditions, to test whether the capacity achieving channel input distributions with feedback are time-invariant, and whether feedback capacity is characterized by single letter, expressions, similar to that of memoryless channels. The method is based on showing that a certain dynamic programming equation, which in general, is a nested optimization problem over the sequence of channel input distributions, reduces to a non-nested optimization problem. Moreover, for UMCO channels, we give a simple expression for the ML error exponent, and we identify sufficient conditions to test whether feedback does not increase capacity. We derive similar results, when transmission cost constraints are imposed. We apply the results to a special class of the UMCO channels, the Binary State Symmetric Channel (BSSC) with and without transmission cost constraints, to show that the optimization problem of feedback capacity is non-nested, the capacity achieving channel input distribution and the corresponding channel output transition probability distribution are time-invariant, and feedback capacity is characterized by a single letter formulae, precisely as Shannon's single letter characterization of capacity of memoryless channels. Then we derive closed form expressions for the capacity achieving channel input distribution and feedback capacity. We use the closed form expressions to evaluate an error exponent for ML decoding.

Research paper thumbnail of Optimal Nonanticipative Estimation Schemes for Time-Varying Gauss-Markov Processes

arXiv (Cornell University), Mar 14, 2016

In this paper, we derive recursive filters for time-varying multidimensional Gauss-Markov process... more In this paper, we derive recursive filters for time-varying multidimensional Gauss-Markov processes, which satisfy a mean square error fidelity, using the concept of Finite Time Horizon (FTH) Nonanticipative Rate Distortion Function (NRDF) and its connection to real-time realizable filtering theory. Moreover, we derive a universal lower bound on the mean square error of any estimator of time-varying multidimensional Gauss-Markov processes in terms of conditional mutual information. Unlike classical Kalman filters, the proposed filter is constructed from the solution of a reverse-waterfilling problem, which ensures that the mean square error fidelity is met. Our theoretical results are demonstrated via illustrative examples.

Research paper thumbnail of Capacity of the Binary State Symmetric Channel with Cost Constraint

arXiv (Cornell University), Jan 31, 2014

We consider a unit memory channel, called Binary State Symmetric Channel (BSSC), in which the cha... more We consider a unit memory channel, called Binary State Symmetric Channel (BSSC), in which the channel state is the modulo2 addition of the current channel input and the previous channel output. We derive closed form expressions for the capacity and corresponding channel input distribution, of this BSSC with and without feedback and transmission cost. We also show that the capacity of the BSSC is not increased by feedback, and it is achieved by a first order symmetric Markov process.

Research paper thumbnail of Nonanticipative Rate Distortion Function and Filtering Theory: A weak Convergence Approach

arXiv (Cornell University), Dec 29, 2012

In this paper the relation between nonanticipative rate distortion function (RDF) and Bayesian fi... more In this paper the relation between nonanticipative rate distortion function (RDF) and Bayesian filtering theory is further investigated on general Polish spaces. The relation is established via an optimization on the space of conditional distributions of the so-called directed information subject to fidelity constraints. Existence of the optimal reproduction distribution of the nonanticipative RDF is shown using the topology of weak convergence of probability measures. Subsequently, we use the solution of the nonanticipative RDF to present the realization of a multidimensional partially observable source over a scalar Gaussian channel. We show that linear encoders are optimal, establishing joint source-channel coding in real-time.

Research paper thumbnail of Capacity of Binary State Symmetric Channel with and without feedback and transmission cost

We consider a unit memory channel, called Binary State Symmetric Channel (BSSC), in which the cha... more We consider a unit memory channel, called Binary State Symmetric Channel (BSSC), in which the channel state is the modulo2 addition of the current channel input and the previous channel output. We derive closed form expressions for the capacity and corresponding channel input distribution, of this BSSC with and without feedback and transmission cost. We also show that the capacity of the BSSC is not increased by feedback, and it is achieved by a first order symmetric Markov process.

Research paper thumbnail of State-Dependent Generalizations of Nonanticipatory Epsilon Entropy of Partially Observable Processes

2022 IEEE 61st Conference on Decision and Control (CDC), Dec 6, 2022

Two fundamental generalizations of Gorbunov's and Pinsker's nonanticipatory ϵ−entropy are formula... more Two fundamental generalizations of Gorbunov's and Pinsker's nonanticipatory ϵ−entropy are formulated and analyzed for a tuple of partially observable, finite-dimensional, random processes (X n , Y n), where X n △ = {X1,. .. , Xn} is the unobserved state process and Y n △ = {Y1,. .. , Yn} is the observable process-a noisy version of X n , subject to a fidelity between X n , and its reproduction X n △ = { X1,. .. , Xn}. The encoder observes causally Y n and past reproductions X n may or may not be available to both the encoder and the decoder. Theorem 1 gives a tight lower bound on the operational rate of zero-delay codes, when X n is causally available to the decoder only, in terms of a state-dependent nonanticipatory ϵ−entropy of a state process Z n , which is fundamentally different from a corresponding nonanticipatory ϵ−entropy, when X n is causally available to both the encoder and the decoder. Theorem 2 identifies sufficient conditions for the two nonanticipatory ϵ−entropies to coincide. Theorem 3 identifies the information structure of the optimal test-channel distributions. The paper also discusses applications to jointly Gaussian partially observable processes (X n , Y n) with a square-error fidelity criterion, and derives characterizations of the two nonanticipatory ϵ−entropies.

Research paper thumbnail of Dynamic Programming Subject to Total Variation Distance Ambiguity

arXiv (Cornell University), Feb 5, 2014

The aim of this paper is to address optimality of stochastic control strategies via dynamic progr... more The aim of this paper is to address optimality of stochastic control strategies via dynamic programming subject to total variation distance ambiguity on the conditional distribution of the controlled process. We formulate the stochastic control problem using minimax theory, in which the control minimizes the pay-off while the conditional distribution, from the total variation distance set, maximizes it. First, we investigate the maximization of a linear functional on the space of probability measures on abstract spaces, among those probability measures which are within a total variation distance from a nominal probability measure, and then we give the maximizing probability measure in closed form. Second, we utilize the solution of the maximization to solve minimax stochastic control with deterministic control strategies, under a Markovian and a non-Markovian assumption, on the conditional distributions of the controlled process. The results of this part include: 1) Minimax optimization subject to total variation distance ambiguity constraint; 2) new dynamic programming recursions, which involve the oscillator seminorm of the value function, in addition to the standard terms; 3) new infinite horizon discounted dynamic programming equation, the associated contractive property, and a new policy iteration algorithm. Finally, we provide illustrative examples for both the finite and infinite horizon cases. For the infinite horizon case we invoke the new policy iteration algorithm to compute the optimal strategies.

Research paper thumbnail of Rate Distortion Function for a Class of Relative Entropy Sources

arXiv (Cornell University), May 6, 2013

This paper deals with rate distortion or source coding with fidelity criterion, in measure spaces... more This paper deals with rate distortion or source coding with fidelity criterion, in measure spaces, for a class of source distributions. The class of source distributions is described by a relative entropy constraint set between the true and a nominal distribution. The rate distortion problem for the class is thus formulated and solved using minimax strategies, which result in robust source coding with fidelity criterion. It is shown that minimax and maxmin strategies can be computed explicitly, and they are generalizations of the classical solution. Finally, for discrete memoryless uncertain sources, the rate distortion theorem is stated for the class omitting the derivations while the converse is derived. I. INTRODUCTION This paper is concerned with lossy data compression for a class of sources defined on the space of probability distributions on general alphabet spaces. In the classical rate distortion formulation with the fidelity decoding criterion, Shannon has shown that minimization of mutual information between finite alphabet source and reproduction sequences subject to fidelity criterion over the reproduction kernel has an operational meaning. Hence, it gives the minimum amount of information of representing a source symbol by a reproduction symbol with a pre-specified fidelity or distortion criterion. The classical rate distortion function for finite-alphabet and continuous sources has been studied thoroughly in the literature [1], [2], [3], [4] and [5]. A survey of the theory of rate distortion is given in [4]. The formulation of rate distortion function for abstract alphabets is investigated by Csiszár in [5]. Specifically, in [5] the question of existence of solution in Polish spaces under some continuity assumptions on the distortion function and compactness of the reproduction space, is established under the topology of weak convergence. The formulation in [5] is based on two important assumptions, namely, 1) compactness of the reproduction space, 2) absolute continuity of all marginal distributions with respect to the optimal marginal distribution. The compactness assumption is crucial in order to formulate the problem using countably additive measures, and to show existence of the minimizing measure using The research leading to these results has received funding from the European Community's Seventh Framework Programme (FP7/2007-2013) under grant agreement no. INFSO-ICT-223844.

Research paper thumbnail of Variational Equalities of Directed Information and Applications

arXiv (Cornell University), Jan 28, 2013

In this paper we introduce two variational equalities of directed information, which are analogou... more In this paper we introduce two variational equalities of directed information, which are analogous to those of mutual information employed in the Blahut-Arimoto Algorithm (BAA). Subsequently, we introduce nonanticipative Rate Distortion Function (RDF) R na 0,n (D) defined via directed information introduced in [1], and we establish its equivalence to Gorbunov-Pinsker's nonanticipatory ǫ-entropy R ε 0,n (D). By invoking certain results we first establish existence of the infimizing reproduction distribution for R na 0,n (D), and then we give its implicit form for the stationary case. Finally, we utilize one of the variational equalities and the closed form expression of the optimal reproduction distribution to provide an algorithm for the computation of R na 0,n (D).

Research paper thumbnail of On the Capacity of Gaussian MIMO Channels With Memory

IEEE Communications Letters, Aug 1, 2022

The operational capacity of Gaussian MIMO channels with memory was obtained by Brandenburg and Wy... more The operational capacity of Gaussian MIMO channels with memory was obtained by Brandenburg and Wyner in [9] under certain mild assumptions on the channel impulse response and its noise covariance matrix, which essentuially require channel memory to be not too strong. This channel was also considered by Tsybakov in [10] and its information capacity was obtained in some cases. It was further conjectured, based on numerical evidence, that these capacities are the same in all cases. This conjecture is proved here. An explicit closed-form expression for the optimal input power spectral density matrix is also given. The obtained result is further extended to the case of joint constraints, including per-antenna and interference power constraints as well as energy harvesting constraints. These results imply the information-theoretic optimality of OFDM-type transmission systems for such channels with memory.

Research paper thumbnail of Sequential Necessary and Sufficient Conditions for Capacity Achieving Distributions of Channels With Memory and Feedback

IEEE Transactions on Information Theory, Nov 1, 2017

We derive sequential necessary and sufficient conditions for any channel input conditional distri... more We derive sequential necessary and sufficient conditions for any channel input conditional distribution

Research paper thumbnail of Generalizations of Nonanticipative Rate Distortion Function to Multivariate Nonstationary Gaussian Autoregressive Processes

2019 IEEE 58th Conference on Decision and Control (CDC)

The characterizations of nonanticipative rate distortion function (NRDF) on a finite horizon are ... more The characterizations of nonanticipative rate distortion function (NRDF) on a finite horizon are generalized to nonstationary multivariate Gaussian order L autoregressive, AR(L), source processes, with respect to mean square error (MSE) distortion functions. It is shown that the optimal reproduction distributions are induced by a reproduction process, which is a linear function of the state of the source, its best mean-square error estimate, and a Gaussian random process.