Tutorial on information theory in visualization (original) (raw)
Related papers
Information, Entropy and Their Geometric Structures
MDPI eBooks, 2015
In the information theory community, the following "historical" statements are generally well accepted: (1) Hartley did put forth his rule twenty years before Shannon; (2) Shannon's formula as a fundamental tradeoff between transmission rate, bandwidth, and signal-to-noise ratio came out unexpected in 1948; (3) Hartley's rule is inexact while Shannon's formula is characteristic of the additive white Gaussian noise channel; (4) Hartley's rule is an imprecise relation that is not an appropriate formula for the capacity of a communication channel. We show that all these four statements are somewhat wrong. In fact, a careful calculation shows that "Hartley's rule" in fact coincides with Shannon's formula. We explain this mathematical coincidence by deriving the necessary and sufficient conditions on an additive noise channel such that its capacity is given by Shannon's formula and construct a sequence of such channels that makes the link between the uniform (Hartley) and Gaussian (Shannon) channels.
Data Mining Algorithms in C++
Much of the material in this chapter is extracted from my prior book, Assessing and Improving Prediction and Classification. My apologies to those readers who may feel cheated by this. However, this material is critical to the current text, and I felt that it would be unfair to force readers to buy my prior book in order to procure required background. The essence of data mining is the discovery of relationships among variables that we have measured. Throughout this book we will explore many ways to find, present, and capitalize on such relationships. In this chapter, we focus primarily on a specific aspect of this task: evaluating and perhaps improving the information content of a measured variable. What is information? This term has a rigorously defined meaning, which we will now pursue. Entropy Suppose you have to send a message to someone, giving this person the answer to a multiple-choice question. The catch is, you are only allowed to send the message by means of a string of ones and zeros, called bits. What is the minimum number of bits that you need to communicate the answer? Well, if it is a true/false question, one bit will obviously do. If four answers are possible, you will need two bits, which provide four possible patterns: 00, 01, 10, and 11. Eight answers will require three bits, and so forth. In general, to identify one of K possibilities, you will need log 2 (K) bits, where log 2 (.) is the logarithm base two. Working with base-two logarithms is unconventional. Mathematicians and computer programs almost always use natural logarithms, in which the base is e≈2.718. The material in this chapter does not require base two; any base will do. By tradition, when natural logarithms are used in information theory, the unit of information is called
Although the use of the word 'information', with different meanings, can be traced back to antique and medieval texts (see Adriaans 2013), it is only in the 20 th century that the term begins to acquire the present-day sense. Nevertheless, the pervasiveness of the notion of information both in our everyday life and in our scientific practice does not imply the agreement about the content of the concept. As Luciano stresses, it is a polysemantic concept associated with different phenomena, such as communication, computation, knowledge, reference, meaning, truth, etc. In the second half of the 20 th century, philosophy begins to direct its attention to this omnipresent but intricate concept in an effort of unravel the tangle of significances surrounding it.
Information Theory, Relative Entropy and Statistics
Lecture Notes in Computer Science, 2009
It is commonly assumed that computers process information. But what is information? In a technical, important, but nevertheless rather narrow sense, Shannon's information theory gives a first answer to this question. This theory focuses on measuring the information content of a message. Essentially this measure is the reduction of the uncertainty obtained by receiving a message. The uncertainty of a situation of ignorance in turn is measured by entropy. This theory has had an immense impact on the technology of information storage, data compression, information transmission and coding and still is a very active domain of research.
Shannon Entropy, Renyi Entropy, and Information
2000
This memo,contains proofs that the Shannon entropy is the limiting case of both the Renyi entropy and the information. These results are also confirmed experimentally. We conclude with some general observations on the utility of entropy measures.
A Possible Extension of Shannon's Information Theory
Entropy, 2001
As a possible generalization of Shannon's information theory, we review the formalism based on the non-logarithmic information content parametrized by a real number q, which exhibits nonadditivity of the associated uncertainty. Moreover it is shown that the establishment of the concept of the mutual information is of importance upon the generalization.