Search   Memberlist   Usergroups
 Page 1 of 1 [5 Posts]
Author Message
yangyu05@gmail.com
science forum beginner

Joined: 28 Oct 2005
Posts: 7

Posted: Tue Jun 06, 2006 7:52 pm    Post subject: Question on Kullback-Leibler Divergence

hi all,

I have a question regarding Kullback-Leibler Divergence (KL) in
information theory. Given a true distribution p and an estimated
distribution q for a random variable X, KL(p, q) measures the
"distance" from p to q.

KL(p, q) = \sum_i p(i) * log (p(i) / q(i)).

We know that KL(p, q) is low-bounded by 0. My question is: Does KL(p,
q) has an upper-bound? i.e., for a given distribution p, is there a
distribution q that maximizes KL(p, q).

Thanks for the help.

Yang
Janusz Kawczak
science forum beginner

Joined: 18 Feb 2006
Posts: 10

Posted: Wed Jun 07, 2006 6:16 am    Post subject: Re: Question on Kullback-Leibler Divergence

Yes, positive infinity as an upper bound!
BTW, usually q is the theoretical measure. But of course it does not

Janusz.

yang wrote:
 Quote: hi all, I have a question regarding Kullback-Leibler Divergence (KL) in information theory. Given a true distribution p and an estimated distribution q for a random variable X, KL(p, q) measures the "distance" from p to q. KL(p, q) = \sum_i p(i) * log (p(i) / q(i)). We know that KL(p, q) is low-bounded by 0. My question is: Does KL(p, q) has an upper-bound? i.e., for a given distribution p, is there a distribution q that maximizes KL(p, q). Thanks for the help. Yang
yangyu05@gmail.com
science forum beginner

Joined: 28 Oct 2005
Posts: 7

Posted: Thu Jun 08, 2006 4:18 pm    Post subject: Re: Question on Kullback-Leibler Divergence

thanks. my next question: does there exist q, such that for any p, q
maximize KL(p, q) on average? I am not sure if my question make sense
here. :)

Janusz Kawczak wrote:
 Quote: Yes, positive infinity as an upper bound! BTW, usually q is the theoretical measure. But of course it does not matter in your question. Janusz. yang wrote: hi all, I have a question regarding Kullback-Leibler Divergence (KL) in information theory. Given a true distribution p and an estimated distribution q for a random variable X, KL(p, q) measures the "distance" from p to q. KL(p, q) = \sum_i p(i) * log (p(i) / q(i)). We know that KL(p, q) is low-bounded by 0. My question is: Does KL(p, q) has an upper-bound? i.e., for a given distribution p, is there a distribution q that maximizes KL(p, q). Thanks for the help. Yang
Janusz Kawczak
science forum beginner

Joined: 18 Feb 2006
Posts: 10

Posted: Thu Jun 08, 2006 5:58 pm    Post subject: Re: Question on Kullback-Leibler Divergence

You are right. Your question makes no sense as you posed it. You need to
explain what you intend to achieve in more "understandable" way. Maybe
an example would help?

Janusz.

yang wrote:
 Quote: thanks. my next question: does there exist q, such that for any p, q maximize KL(p, q) on average? I am not sure if my question make sense here. :) Janusz Kawczak wrote: Yes, positive infinity as an upper bound! BTW, usually q is the theoretical measure. But of course it does not matter in your question. Janusz. yang wrote: hi all, I have a question regarding Kullback-Leibler Divergence (KL) in information theory. Given a true distribution p and an estimated distribution q for a random variable X, KL(p, q) measures the "distance" from p to q. KL(p, q) = \sum_i p(i) * log (p(i) / q(i)). We know that KL(p, q) is low-bounded by 0. My question is: Does KL(p, q) has an upper-bound? i.e., for a given distribution p, is there a distribution q that maximizes KL(p, q). Thanks for the help. Yang
yangyu05@gmail.com
science forum beginner

Joined: 28 Oct 2005
Posts: 7

Posted: Thu Jun 08, 2006 6:50 pm    Post subject: Re: Question on Kullback-Leibler Divergence

Ok, the basic situation is as follows.

Consider a sequence of numbers {a_1, a_2, ..., a_m}. Each a_i is in the
set [n]={1, ..., n}. Numbers in the sequence do not need to be unique.
We want to hide the pattern of the sequence by mapping each a_i to a
subset of [n]. For example, mapping sequence {1, 1, 2, 3} to {{1, 2,
3}, {1, 2, 3}, {1, 2, 3}, {1, 2, 3}} hides the pattern of the origianl
sequence.

We also want to minimize the length of the outcome of the mapping. The
above trivial mapping in the example is not always preferred. Thus, we
need a way to measure the goodness of a mapping. That's why we come up
with KL divergence. Then the question is what's the best mapping we can
achieve under this metric. Also, is this metric good enough?

Hope i am making more sense here. Thanks.

yang

Janusz Kawczak wrote:
 Quote: You are right. Your question makes no sense as you posed it. You need to explain what you intend to achieve in more "understandable" way. Maybe an example would help? Janusz. yang wrote: thanks. my next question: does there exist q, such that for any p, q maximize KL(p, q) on average? I am not sure if my question make sense here. :) Janusz Kawczak wrote: Yes, positive infinity as an upper bound! BTW, usually q is the theoretical measure. But of course it does not matter in your question. Janusz. yang wrote: hi all, I have a question regarding Kullback-Leibler Divergence (KL) in information theory. Given a true distribution p and an estimated distribution q for a random variable X, KL(p, q) measures the "distance" from p to q. KL(p, q) = \sum_i p(i) * log (p(i) / q(i)). We know that KL(p, q) is low-bounded by 0. My question is: Does KL(p, q) has an upper-bound? i.e., for a given distribution p, is there a distribution q that maximizes KL(p, q). Thanks for the help. Yang

 Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First
 Page 1 of 1 [5 Posts]
 The time now is Sat Mar 23, 2019 10:48 am | All times are GMT
 Jump to: Select a forum-------------------Forum index|___Science and Technology    |___Math    |   |___Research    |   |___num-analysis    |   |___Symbolic    |   |___Combinatorics    |   |___Probability    |   |   |___Prediction    |   |       |   |___Undergraduate    |   |___Recreational    |       |___Physics    |   |___Research    |   |___New Theories    |   |___Acoustics    |   |___Electromagnetics    |   |___Strings    |   |___Particle    |   |___Fusion    |   |___Relativity    |       |___Chem    |   |___Analytical    |   |___Electrochem    |   |   |___Battery    |   |       |   |___Coatings    |       |___Engineering        |___Control        |___Mechanics        |___Chemical

 Topic Author Forum Replies Last Post Similar Topics Question about Life. socratus Probability 0 Sun Jan 06, 2008 10:01 pm Probability Question dumont Probability 0 Mon Oct 23, 2006 3:38 pm Question about exponention WingDragon@gmail.com Math 2 Fri Jul 21, 2006 8:13 am question on solartron 1260 carrie_yao@hotmail.com Electrochem 0 Fri Jul 21, 2006 7:11 am A Combinatorics/Graph Theory Question mathlover Undergraduate 1 Wed Jul 19, 2006 11:30 pm