|
|
| Author |
Message |
gino
science forum beginner
Joined: 01 Feb 2006
Posts: 29
|
 Posted: Tue Jun 27, 2006 4:45 am Post subject:
Maximum Likelihood Estimation not reliable???
|
|
|
Hi all,
I am doing some fun experiments in studying Maximum Likelihood estimation.
Let's say I generate a Geometric Brownian motion process using random number
generator and via some transformation. I've made certain this step is
correct. And I know the true values of drift and sigma that I've used. Let
me call them drift_star and sigma_star.
Now I want to estimate the drift and volatility of my generated data, as an
exercise of MLE:
I've derived analytically the log-likelihood function. And I plug in the
simulated data, and then feed this into a well-known local optimizer, SNOPT.
Funny things occured. Let me now describe the situation:
----------------------
I know MLE needs relative large data set. So I make it 10^6 sample points in
my descritization of the Geometric Brownian motion, for time horizon fixed
at T=1. That's to say, divide T by 10^6 and that's the step size.
I did my local optimization search with a starting point which is exact the
true values, since I simulated this data from the true values.
I was hoping the local optimizer -- SNOPT can converge back to true value
again. In this way, I can verify if MLE does really work, and my analytical
derivation of the log-likelihood function is correct.
The result was surprising. The SNOPT does converge to a local optimum, which
is different the starting point -- the true value. The loglikelihood
function at that point is always larger than that of the true value. I did
many runs of this simulation. The local maximum around the true values are
always larger than that of the true value.
My question is: does this mean MLE is not usable, since the true value point
is not actually the maximum likelihood point of the log-likelihood function.
Even in a neighborhood of true values, we can still find point that has even
larger log-likelihood value? In this case, suppose when we don't know the
true values, then even we take effort to find the global maximum of the
log-likelihood function, it still might not be the true values... not to say
for a hundred data points(not 10^6) in reality.
I hope the more experienced people can help me. If MLE is reliable(given the
sample size is 10^6 which is so big, MLE should converge to true values
asympototically), then my analytical expression of the log-likelihood
function might be wrong... I believe the local search part SNOPT should be
no problem, because it is just a search.
-------------------------
Thanks a lot! |
|
| Back to top |
|
 |
gino
science forum beginner
Joined: 01 Feb 2006
Posts: 29
|
| Posted: Tue Jun 27, 2006 4:51 am Post subject:
Re: Maximum Likelihood Estimation not reliable???
|
|
|
One more report:
This is actually very interesting.
I did a lot more experiments. And found out the following:
I still start my local optimizer search from the true values.
If the local optimizer finishes very quickly with only a few iterations,
then it almost always return the true values, or very near to the true
values;
If the local optimizer took a lot iterations to finish, then it always find
a point that has a larger log-likelihood function value than the true
values.
"gino" <loseminds@hotmail.com> wrote in message
news:e7qd4d$6dn$1@news.Stanford.EDU...
| Quote: | Hi all,
I am doing some fun experiments in studying Maximum Likelihood estimation.
Let's say I generate a Geometric Brownian motion process using random
number generator and via some transformation. I've made certain this step
is correct. And I know the true values of drift and sigma that I've used.
Let me call them drift_star and sigma_star.
Now I want to estimate the drift and volatility of my generated data, as
an exercise of MLE:
I've derived analytically the log-likelihood function. And I plug in the
simulated data, and then feed this into a well-known local optimizer,
SNOPT. Funny things occured. Let me now describe the situation:
----------------------
I know MLE needs relative large data set. So I make it 10^6 sample points
in my descritization of the Geometric Brownian motion, for time horizon
fixed at T=1. That's to say, divide T by 10^6 and that's the step size.
I did my local optimization search with a starting point which is exact
the true values, since I simulated this data from the true values.
I was hoping the local optimizer -- SNOPT can converge back to true value
again. In this way, I can verify if MLE does really work, and my
analytical derivation of the log-likelihood function is correct.
The result was surprising. The SNOPT does converge to a local optimum,
which is different the starting point -- the true value. The loglikelihood
function at that point is always larger than that of the true value. I did
many runs of this simulation. The local maximum around the true values are
always larger than that of the true value.
My question is: does this mean MLE is not usable, since the true value
point is not actually the maximum likelihood point of the log-likelihood
function. Even in a neighborhood of true values, we can still find point
that has even larger log-likelihood value? In this case, suppose when we
don't know the true values, then even we take effort to find the global
maximum of the log-likelihood function, it still might not be the true
values... not to say for a hundred data points(not 10^6) in reality.
I hope the more experienced people can help me. If MLE is reliable(given
the sample size is 10^6 which is so big, MLE should converge to true
values asympototically), then my analytical expression of the
log-likelihood function might be wrong... I believe the local search part
SNOPT should be no problem, because it is just a search.
-------------------------
Thanks a lot!
|
|
|
| Back to top |
|
|
gino
science forum beginner
Joined: 01 Feb 2006
Posts: 29
|
| Posted: Tue Jun 27, 2006 4:54 am Post subject:
Re: Maximum Likelihood Estimation not reliable???
|
|
|
| Quote: | One more report:
This is actually very interesting.
I did a lot more experiments. And found out the following:
I still start my local optimizer search from the true values.
If the local optimizer finishes very quickly with only a few iterations,
then it almost always return the true values, or very near to the true
values;
If the local optimizer took a lot iterations to finish, then it always
find a point that has a larger log-likelihood function value than the true
values.
|
Based on those cases that if I start from the true values and the local
optimizer will finishes with that local optimum point, can I claim my
analytical derivation of the log-likelihood function is correct, or not?
I am imagining that if the log-likelihood function expression was wrong,
then even if I start from the true values, I may not never get back to the
true values? |
|
| Back to top |
|
|
glenbarnett@geocities.com
science forum beginner
Joined: 13 Jun 2006
Posts: 3
|
| Posted: Tue Jun 27, 2006 6:40 am Post subject:
Re: Maximum Likelihood Estimation not reliable???
|
|
|
gino wrote:
| Quote: | The result was surprising. The SNOPT does converge to a local optimum, which
is different the starting point -- the true value. The loglikelihood
function at that point is always larger than that of the true value. I did
many runs of this simulation. The local maximum around the true values are
always larger than that of the true value.
|
This is always the case; a little thought makes it obvious why.
You're maximising the likelihood, so at the place where that maximum is
the likelihood will always be higher than somewhere else.
Due to random variation, the sample is always a better fit to something
a little way away from the true value.
[Think of the simplest case, fitting an normal distribution to a sample
that is assumed iid.
The sum of squares from the sample mean is always smaller than the sum
of squares from the true mean, since the sample mean is the least
squares estimator. Indeed you can write the sum of squares from the
true mean as a sum of squares from the sample mean plus a multiple of
another square.]
Consequently the true values are always going to have a lower
likelihood than the ML estimates!
This is not a problem, but an inherent characteristic of what you're
trying to do.
Glen |
|
| Back to top |
|
|
gino
science forum beginner
Joined: 01 Feb 2006
Posts: 29
|
| Posted: Tue Jun 27, 2006 6:51 am Post subject:
Re: Maximum Likelihood Estimation not reliable???
|
|
|
<glenbarnett@geocities.com> wrote in message
news:1151390443.192914.183270@75g2000cwc.googlegroups.com...
| Quote: |
gino wrote:
The result was surprising. The SNOPT does converge to a local optimum,
which
is different the starting point -- the true value. The loglikelihood
function at that point is always larger than that of the true value. I
did
many runs of this simulation. The local maximum around the true values
are
always larger than that of the true value.
This is always the case; a little thought makes it obvious why.
You're maximising the likelihood, so at the place where that maximum is
the likelihood will always be higher than somewhere else.
Due to random variation, the sample is always a better fit to something
a little way away from the true value.
[Think of the simplest case, fitting an normal distribution to a sample
that is assumed iid.
The sum of squares from the sample mean is always smaller than the sum
of squares from the true mean, since the sample mean is the least
squares estimator. Indeed you can write the sum of squares from the
true mean as a sum of squares from the sample mean plus a multiple of
another square.]
Consequently the true values are always going to have a lower
likelihood than the ML estimates!
This is not a problem, but an inherent characteristic of what you're
trying to do.
Glen
|
Glen,
Thanks for your input. I agree with you that this is an inherent
characteristic of MLE of random variated data.
The problem is that in my understanding, if I increase the sample size, the
consistency guarantees that it will converge to the true value with no bias.
As you can see, I am already doing 10^6 samples, but the deviation is
actually quite large, both in terms of objective function values and the
returned estimated parameters.
Also, I am looking for a way to make sure the formula for log-likelihood
function which I've derived is correct... |
|
| Back to top |
|
|
David Jones
science forum beginner
Joined: 19 May 2005
Posts: 15
|
| Posted: Tue Jun 27, 2006 9:28 am Post subject:
Re: Maximum Likelihood Estimation not reliable???
|
|
|
gino wrote:
| Quote: | glenbarnett@geocities.com> wrote in message
news:1151390443.192914.183270@75g2000cwc.googlegroups.com...
gino wrote:
The result was surprising. The SNOPT does converge to a local
optimum, which
is different the starting point -- the true value. The
loglikelihood
function at that point is always larger than that of the true
value. I did
many runs of this simulation. The local maximum around the true
values are
always larger than that of the true value.
This is always the case; a little thought makes it obvious why.
You're maximising the likelihood, so at the place where that
maximum
is the likelihood will always be higher than somewhere else.
Due to random variation, the sample is always a better fit to
something a little way away from the true value.
[Think of the simplest case, fitting an normal distribution to a
sample that is assumed iid.
The sum of squares from the sample mean is always smaller than the
sum of squares from the true mean, since the sample mean is the
least
squares estimator. Indeed you can write the sum of squares from the
true mean as a sum of squares from the sample mean plus a multiple
of
another square.]
Consequently the true values are always going to have a lower
likelihood than the ML estimates!
This is not a problem, but an inherent characteristic of what
you're
trying to do.
Glen
Glen,
Thanks for your input. I agree with you that this is an inherent
characteristic of MLE of random variated data.
The problem is that in my understanding, if I increase the sample
size, the consistency guarantees that it will converge to the true
value with no bias. As you can see, I am already doing 10^6 samples,
but the deviation is actually quite large, both in terms of
objective
function values and the returned estimated parameters.
Also, I am looking for a way to make sure the formula for
log-likelihood function which I've derived is correct...
|
(i) if applied correctly the theory for maximum likelihood will tell
you both:
(a) how far away the estimated parameters are likely to be from the
true values;
(b) how different the likelihood functions at the estimated and
true parameter values are likely to be.
(ii) You may not actually have a large sample depending on what you
are really doing. You might only have single sample. Just because you
are splitting the interval into a large number of sub-steps doesn't
change this. The literature should contain some stuff where the
likelihood function is treated for the case of a continuously-observed
process, which is obviously the limit as your number of time-steps
within a fixed interval increases. It is not the case that this
automatically produces the "true" values. The flaw in that argument is
that the amount of information about the parameters provided by the
observations is bounded as the size of the time-steps reduces.
David Jones |
|
| Back to top |
|
|
Herman Rubin
science forum Guru
Joined: 25 Mar 2005
Posts: 730
|
| Posted: Tue Jun 27, 2006 4:03 pm Post subject:
Re: Maximum Likelihood Estimation not reliable???
|
|
|
In article <e7qd4d$6dn$1@news.Stanford.EDU>,
gino <loseminds@hotmail.com> wrote:
| Quote: | Hi all,
I am doing some fun experiments in studying Maximum Likelihood estimation.
Let's say I generate a Geometric Brownian motion process using random number
generator and via some transformation. I've made certain this step is
correct. And I know the true values of drift and sigma that I've used. Let
me call them drift_star and sigma_star.
Now I want to estimate the drift and volatility of my generated data, as an
exercise of MLE:
I've derived analytically the log-likelihood function. And I plug in the
simulated data, and then feed this into a well-known local optimizer, SNOPT.
Funny things occured. Let me now describe the situation:
|
----------------------
| Quote: | I know MLE needs relative large data set. So I make it 10^6 sample points in
my descritization of the Geometric Brownian motion, for time horizon fixed
at T=1. That's to say, divide T by 10^6 and that's the step size.
I did my local optimization search with a starting point which is exact the
true values, since I simulated this data from the true values.
I was hoping the local optimizer -- SNOPT can converge back to true value
again. In this way, I can verify if MLE does really work, and my analytical
derivation of the log-likelihood function is correct.
The result was surprising. The SNOPT does converge to a local optimum, which
is different the starting point -- the true value. The loglikelihood
function at that point is always larger than that of the true value. I did
many runs of this simulation. The local maximum around the true values are
always larger than that of the true value.
My question is: does this mean MLE is not usable, since the true value point
is not actually the maximum likelihood point of the log-likelihood function.
Even in a neighborhood of true values, we can still find point that has even
larger log-likelihood value? In this case, suppose when we don't know the
true values, then even we take effort to find the global maximum of the
log-likelihood function, it still might not be the true values... not to say
for a hundred data points(not 10^6) in reality.
I hope the more experienced people can help me. If MLE is reliable(given the
sample size is 10^6 which is so big, MLE should converge to true values
asympototically), then my analytical expression of the log-likelihood
function might be wrong... I believe the local search part SNOPT should be
no problem, because it is just a search.
|
-------------------------
There is sample size, and there is sample size.
If you look at the estimator of the variance of the
Brownian motion, assuming you do not know it, you
will find it estimated quite well. But as for the
drift, you have no more information than the value
at T=1. The sufficient statistics are the value at
T=1 and the sum of squares of the increments.
Even if you could analyze the entire continuous process,
you would be able to get the variance exactly, but still
only the value at T=1 would be helpful about the drift.
--
This address is for information only. I do not claim that these views
are those of the Statistics Department or of Purdue University.
Herman Rubin, Department of Statistics, Purdue University
hrubin@stat.purdue.edu Phone: (765)494-6054 FAX: (765)494-0558 |
|
| Back to top |
|
|
bdmccullough@drexel.edu
science forum beginner
Joined: 27 Jun 2006
Posts: 1
|
| Posted: Tue Jun 27, 2006 4:23 pm Post subject:
Re: Maximum Likelihood Estimation not reliable???
|
|
|
It is not a good idea to start at the true values. See, e.g., Gill,
Murray and Wright.
| Quote: | I did my local optimization search with a starting point which is exact the
true values, since I simulated this data from the true values.
|
|
|
| Back to top |
|
|
C6L1V@shaw.ca
science forum Guru
Joined: 23 May 2005
Posts: 628
|
| Posted: Tue Jun 27, 2006 6:36 pm Post subject:
Re: Maximum Likelihood Estimation not reliable???
|
|
|
gino wrote:
| Quote: | Hi all,
I am doing some fun experiments in studying Maximum Likelihood estimation.
Let's say I generate a Geometric Brownian motion process using random number
generator and via some transformation. I've made certain this step is
correct. And I know the true values of drift and sigma that I've used. Let
me call them drift_star and sigma_star.
Now I want to estimate the drift and volatility of my generated data, as an
exercise of MLE:
I've derived analytically the log-likelihood function. And I plug in the
simulated data, and then feed this into a well-known local optimizer, SNOPT.
Funny things occured. Let me now describe the situation:
----------------------
I know MLE needs relative large data set. So I make it 10^6 sample points in
my descritization of the Geometric Brownian motion, for time horizon fixed
at T=1. That's to say, divide T by 10^6 and that's the step size.
|
If I understand you, you find an approximate solution to a stochastic
differential equation by using a step-size of 10^(-6) and so compute
10^6 intermediate values. But, isn't that just ONE sample? To have a
sample size of 10^6, would you not need to repeat your computations
10^6 times? Also: numerical solution of stoch. DEs can be tricky, and
can need methods very different from the deterministic case. If you
have not done so already, I suggest you check out the book "Numerical
Solution of Stochastic Differential Equations", by P.E. Kloeden and E.
Platen, Springer-Verlag (1991).
R.G. Vickson
| Quote: |
I did my local optimization search with a starting point which is exact the
true values, since I simulated this data from the true values.
I was hoping the local optimizer -- SNOPT can converge back to true value
again. In this way, I can verify if MLE does really work, and my analytical
derivation of the log-likelihood function is correct.
The result was surprising. The SNOPT does converge to a local optimum, which
is different the starting point -- the true value. The loglikelihood
function at that point is always larger than that of the true value. I did
many runs of this simulation. The local maximum around the true values are
always larger than that of the true value.
My question is: does this mean MLE is not usable, since the true value point
is not actually the maximum likelihood point of the log-likelihood function.
Even in a neighborhood of true values, we can still find point that has even
larger log-likelihood value? In this case, suppose when we don't know the
true values, then even we take effort to find the global maximum of the
log-likelihood function, it still might not be the true values... not to say
for a hundred data points(not 10^6) in reality.
I hope the more experienced people can help me. If MLE is reliable(given the
sample size is 10^6 which is so big, MLE should converge to true values
asympototically), then my analytical expression of the log-likelihood
function might be wrong... I believe the local search part SNOPT should be
no problem, because it is just a search.
-------------------------
Thanks a lot! |
|
|
| Back to top |
|
|
gino
science forum beginner
Joined: 01 Feb 2006
Posts: 29
|
| Posted: Tue Jun 27, 2006 9:23 pm Post subject:
Re: Maximum Likelihood Estimation not reliable???
|
|
|
<C6L1V@shaw.ca> wrote in message
news:1151433413.609745.314680@75g2000cwc.googlegroups.com...
| Quote: | gino wrote:
Hi all,
I am doing some fun experiments in studying Maximum Likelihood
estimation.
Let's say I generate a Geometric Brownian motion process using random
number
generator and via some transformation. I've made certain this step is
correct. And I know the true values of drift and sigma that I've used.
Let
me call them drift_star and sigma_star.
Now I want to estimate the drift and volatility of my generated data, as
an
exercise of MLE:
I've derived analytically the log-likelihood function. And I plug in the
simulated data, and then feed this into a well-known local optimizer,
SNOPT.
Funny things occured. Let me now describe the situation:
----------------------
I know MLE needs relative large data set. So I make it 10^6 sample points
in
my descritization of the Geometric Brownian motion, for time horizon
fixed
at T=1. That's to say, divide T by 10^6 and that's the step size.
If I understand you, you find an approximate solution to a stochastic
differential equation by using a step-size of 10^(-6) and so compute
10^6 intermediate values. But, isn't that just ONE sample? To have a
sample size of 10^6, would you not need to repeat your computations
10^6 times? Also: numerical solution of stoch. DEs can be tricky, and
can need methods very different from the deterministic case. If you
have not done so already, I suggest you check out the book "Numerical
Solution of Stochastic Differential Equations", by P.E. Kloeden and E.
Platen, Springer-Verlag (1991).
R.G. Vickson
|
Thanks a lot Mr. Vickson for your pointers. Now I got lost:
I am trying to generate some time series data which has 10^6 data points,
and then I try to fit this data to Geometrical Brownian motion model. You
call this data set just one sample? I was calling it 10^6 samples...
I don't use the solution of the Geometrical Brownian motion eqaution, I just
discretize the dynamics, generating an independent Gaussian random variable
at each time step and put that as a Brownian increment... is that okay? |
|
| Back to top |
|
|
C6L1V@shaw.ca
science forum Guru
Joined: 23 May 2005
Posts: 628
|
| Posted: Tue Jun 27, 2006 9:41 pm Post subject:
Re: Maximum Likelihood Estimation not reliable???
|
|
|
gino wrote:
| Quote: | C6L1V@shaw.ca> wrote in message
news:1151433413.609745.314680@75g2000cwc.googlegroups.com...
gino wrote:
Hi all,
I am doing some fun experiments in studying Maximum Likelihood
estimation.
Let's say I generate a Geometric Brownian motion process using random
number
generator and via some transformation. I've made certain this step is
correct. And I know the true values of drift and sigma that I've used.
Let
me call them drift_star and sigma_star.
Now I want to estimate the drift and volatility of my generated data, as
an
exercise of MLE:
I've derived analytically the log-likelihood function. And I plug in the
simulated data, and then feed this into a well-known local optimizer,
SNOPT.
Funny things occured. Let me now describe the situation:
----------------------
I know MLE needs relative large data set. So I make it 10^6 sample points
in
my descritization of the Geometric Brownian motion, for time horizon
fixed
at T=1. That's to say, divide T by 10^6 and that's the step size.
If I understand you, you find an approximate solution to a stochastic
differential equation by using a step-size of 10^(-6) and so compute
10^6 intermediate values. But, isn't that just ONE sample? To have a
sample size of 10^6, would you not need to repeat your computations
10^6 times? Also: numerical solution of stoch. DEs can be tricky, and
can need methods very different from the deterministic case. If you
have not done so already, I suggest you check out the book "Numerical
Solution of Stochastic Differential Equations", by P.E. Kloeden and E.
Platen, Springer-Verlag (1991).
R.G. Vickson
Thanks a lot Mr. Vickson for your pointers. Now I got lost:
I am trying to generate some time series data which has 10^6 data points,
and then I try to fit this data to Geometrical Brownian motion model. You
call this data set just one sample? I was calling it 10^6 samples...
|
Well, it depends on how the thing is set up. All I had to go on was
your verbal description, which did not have much detail.
The next thing to watch out for is the quality of your (pseudo) random
number generator. Some of them are pretty bad.
| Quote: |
I don't use the solution of the Geometrical Brownian motion eqaution, I just
discretize the dynamics, generating an independent Gaussian random variable
at each time step and put that as a Brownian increment... is that okay?
|
Yes, it ought to be. Of course, if the motion is geometric, the
increment should be a ratio rather than a difference.
RGV |
|
| Back to top |
|
|
Herman Rubin
science forum Guru
Joined: 25 Mar 2005
Posts: 730
|
| Posted: Wed Jun 28, 2006 12:53 am Post subject:
Re: Maximum Likelihood Estimation not reliable???
|
|
|
In article <e7s7kv$1j8$1@news.Stanford.EDU>,
gino <loseminds@hotmail.com> wrote:
| Quote: | C6L1V@shaw.ca> wrote in message
news:1151433413.609745.314680@75g2000cwc.googlegroups.com...
gino wrote:
Hi all,
I am doing some fun experiments in studying Maximum Likelihood
estimation.
Let's say I generate a Geometric Brownian motion process using random
number
generator and via some transformation. I've made certain this step is
correct. And I know the true values of drift and sigma that I've used.
Let
me call them drift_star and sigma_star.
Now I want to estimate the drift and volatility of my generated data, as
an
exercise of MLE:
I've derived analytically the log-likelihood function. And I plug in the
simulated data, and then feed this into a well-known local optimizer,
SNOPT.
Funny things occured. Let me now describe the situation:
----------------------
I know MLE needs relative large data set. So I make it 10^6 sample points
in
my descritization of the Geometric Brownian motion, for time horizon
fixed
at T=1. That's to say, divide T by 10^6 and that's the step size.
If I understand you, you find an approximate solution to a stochastic
differential equation by using a step-size of 10^(-6) and so compute
10^6 intermediate values. But, isn't that just ONE sample? To have a
sample size of 10^6, would you not need to repeat your computations
10^6 times? Also: numerical solution of stoch. DEs can be tricky, and
can need methods very different from the deterministic case. If you
have not done so already, I suggest you check out the book "Numerical
Solution of Stochastic Differential Equations", by P.E. Kloeden and E.
Platen, Springer-Verlag (1991).
R.G. Vickson
|
| Quote: | Thanks a lot Mr. Vickson for your pointers. Now I got lost:
I am trying to generate some time series data which has 10^6 data points,
and then I try to fit this data to Geometrical Brownian motion model. You
call this data set just one sample? I was calling it 10^6 samples...
I don't use the solution of the Geometrical Brownian motion eqaution, I just
discretize the dynamics, generating an independent Gaussian random variable
at each time step and put that as a Brownian increment... is that okay?
|
If you generate each random variable with mean mu/10^6 and
variance v/10^6, you will get an accurate estimate of the
variance (10^6 - 1 degrees of freedom), but as for the mean,
only the total will be relevant. So for the mean, you have
effectively one observation with mean mu and variance v.
--
This address is for information only. I do not claim that these views
are those of the Statistics Department or of Purdue University.
Herman Rubin, Department of Statistics, Purdue University
hrubin@stat.purdue.edu Phone: (765)494-6054 FAX: (765)494-0558 |
|
| Back to top |
|
|
gino
science forum beginner
Joined: 01 Feb 2006
Posts: 29
|
| Posted: Wed Jun 28, 2006 3:23 am Post subject:
Re: Maximum Likelihood Estimation not reliable???
|
|
|
"Herman Rubin" <hrubin@odds.stat.purdue.edu> wrote in message
news:e7sjup$2uue@odds.stat.purdue.edu...
| Quote: | In article <e7s7kv$1j8$1@news.Stanford.EDU>,
gino <loseminds@hotmail.com> wrote:
C6L1V@shaw.ca> wrote in message
news:1151433413.609745.314680@75g2000cwc.googlegroups.com...
gino wrote:
Hi all,
I am doing some fun experiments in studying Maximum Likelihood
estimation.
Let's say I generate a Geometric Brownian motion process using random
number
generator and via some transformation. I've made certain this step is
correct. And I know the true values of drift and sigma that I've used.
Let
me call them drift_star and sigma_star.
Now I want to estimate the drift and volatility of my generated data,
as
an
exercise of MLE:
I've derived analytically the log-likelihood function. And I plug in
the
simulated data, and then feed this into a well-known local optimizer,
SNOPT.
Funny things occured. Let me now describe the situation:
----------------------
I know MLE needs relative large data set. So I make it 10^6 sample
points
in
my descritization of the Geometric Brownian motion, for time horizon
fixed
at T=1. That's to say, divide T by 10^6 and that's the step size.
If I understand you, you find an approximate solution to a stochastic
differential equation by using a step-size of 10^(-6) and so compute
10^6 intermediate values. But, isn't that just ONE sample? To have a
sample size of 10^6, would you not need to repeat your computations
10^6 times? Also: numerical solution of stoch. DEs can be tricky, and
can need methods very different from the deterministic case. If you
have not done so already, I suggest you check out the book "Numerical
Solution of Stochastic Differential Equations", by P.E. Kloeden and E.
Platen, Springer-Verlag (1991).
R.G. Vickson
Thanks a lot Mr. Vickson for your pointers. Now I got lost:
I am trying to generate some time series data which has 10^6 data points,
and then I try to fit this data to Geometrical Brownian motion model. You
call this data set just one sample? I was calling it 10^6 samples...
I don't use the solution of the Geometrical Brownian motion eqaution, I
just
discretize the dynamics, generating an independent Gaussian random
variable
at each time step and put that as a Brownian increment... is that okay?
If you generate each random variable with mean mu/10^6 and
variance v/10^6, you will get an accurate estimate of the
variance (10^6 - 1 degrees of freedom), but as for the mean,
only the total will be relevant. So for the mean, you have
effectively one observation with mean mu and variance v.
|
Thanks a lot Herman,
I've also learned about the sufficient statistics about the mu from you in
your earlier post this morning.
Could you please give me some pointers on books/notes that talk about these
stuff. I am wondering where to find statistics inference about these special
processes such as GBM and BM and diffusion type processes, etc.
Also is there any other estimation methods that I can use aside from MLE? I
keep getting stuck at local optimum when using MLE so I would like to try
other estimation methods...
Thank you so much! |
|
| Back to top |
|
|
Herman Rubin
science forum Guru
Joined: 25 Mar 2005
Posts: 730
|
| Posted: Wed Jun 28, 2006 6:46 pm Post subject:
Re: Maximum Likelihood Estimation not reliable???
|
|
|
In article <e7ssn7$idg$1@news.Stanford.EDU>,
gino <loseminds@hotmail.com> wrote:
| Quote: | "Herman Rubin" <hrubin@odds.stat.purdue.edu> wrote in message
news:e7sjup$2uue@odds.stat.purdue.edu...
In article <e7s7kv$1j8$1@news.Stanford.EDU>,
gino <loseminds@hotmail.com> wrote:
C6L1V@shaw.ca> wrote in message
news:1151433413.609745.314680@75g2000cwc.googlegroups.com...
gino wrote:
Hi all,
I am doing some fun experiments in studying Maximum Likelihood
estimation.
Let's say I generate a Geometric Brownian motion process using random
number
generator and via some transformation. I've made certain this step is
correct. And I know the true values of drift and sigma that I've used.
Let
me call them drift_star and sigma_star.
Now I want to estimate the drift and volatility of my generated data,
as
an
exercise of MLE:
I've derived analytically the log-likelihood function. And I plug in
the
simulated data, and then feed this into a well-known local optimizer,
SNOPT.
Funny things occured. Let me now describe the situation:
----------------------
I know MLE needs relative large data set. So I make it 10^6 sample
points
in
my descritization of the Geometric Brownian motion, for time horizon
fixed
at T=1. That's to say, divide T by 10^6 and that's the step size.
If I understand you, you find an approximate solution to a stochastic
differential equation by using a step-size of 10^(-6) and so compute
10^6 intermediate values. But, isn't that just ONE sample? To have a
sample size of 10^6, would you not need to repeat your computations
10^6 times? Also: numerical solution of stoch. DEs can be tricky, and
can need methods very different from the deterministic case. If you
have not done so already, I suggest you check out the book "Numerical
Solution of Stochastic Differential Equations", by P.E. Kloeden and E.
Platen, Springer-Verlag (1991).
R.G. Vickson
|
| Quote: | Thanks a lot Mr. Vickson for your pointers. Now I got lost:
I am trying to generate some time series data which has 10^6 data points,
and then I try to fit this data to Geometrical Brownian motion model. You
call this data set just one sample? I was calling it 10^6 samples...
I don't use the solution of the Geometrical Brownian motion eqaution, I
just
discretize the dynamics, generating an independent Gaussian random
variable
at each time step and put that as a Brownian increment... is that okay?
|
| Quote: | If you generate each random variable with mean mu/10^6 and
variance v/10^6, you will get an accurate estimate of the
variance (10^6 - 1 degrees of freedom), but as for the mean,
only the total will be relevant. So for the mean, you have
effectively one observation with mean mu and variance v.
|
| Quote: | Thanks a lot Herman,
I've also learned about the sufficient statistics about the mu from you in
your earlier post this morning.
Could you please give me some pointers on books/notes that talk about these
stuff. I am wondering where to find statistics inference about these special
processes such as GBM and BM and diffusion type processes, etc.
Also is there any other estimation methods that I can use aside from MLE? I
keep getting stuck at local optimum when using MLE so I would like to try
other estimation methods...
Thank you so much!
|
You are trying to get what isn't there. Any reasonable
mathematical statistics book will explain why a sufficient
statistic is sufficient; there is no more information
available.
One can define the continuous time likelihood function.
SOME quantities, like the variance of the normal term
in any process with independent increments, and generalizations
of that, can be estimated with probability one from continuous
observations. Most others of interest cannot; drift terms, in
particular, cannot be if there is any variance component.
You will have to be satisfied with MLE and related estimators.
The additional information you seek is not there.
--
This address is for information only. I do not claim that these views
are those of the Statistics Department or of Purdue University.
Herman Rubin, Department of Statistics, Purdue University
hrubin@stat.purdue.edu Phone: (765)494-6054 FAX: (765)494-0558 |
|
| Back to top |
|
|
gino
science forum beginner
Joined: 01 Feb 2006
Posts: 29
|
| Posted: Thu Jun 29, 2006 7:03 pm Post subject:
Re: Maximum Likelihood Estimation not reliable???
|
|
|
| Quote: | You are trying to get what isn't there. Any reasonable
mathematical statistics book will explain why a sufficient
statistic is sufficient; there is no more information
available.
One can define the continuous time likelihood function.
SOME quantities, like the variance of the normal term
in any process with independent increments, and generalizations
of that, can be estimated with probability one from continuous
observations. Most others of interest cannot; drift terms, in
particular, cannot be if there is any variance component.
You will have to be satisfied with MLE and related estimators.
The additional information you seek is not there.
|
Thanks a lot Herman.
Aside from the GBM model, I also like to play with the model such as
dV=a*(V+c)*dt+b*(V+c)*dW(t)
Here c is another parameter to estimate. Only V is observed.
You have warned about the difficulty of estimating "a", I am wondering if
there is a similar difficulty of estimating "c"?
Which books/notes talk about sufficient statistics in estimating diffusion
type stochastic processes?
Thanks a lot! |
|
| Back to top |
|
|
Google
|
|
| Back to top |
|
|
|
|
The time now is Sat Jan 10, 2009 4:03 am | All times are GMT
|
|
Buy Anything On eBay | Credit Report | Loans | Myspace Background Generator | Credit Card Debt Consolidation
|
|
Copyright © 2004-2005 DeniX Solutions SRL
|
|
Other DeniX Solutions sites:
Electronics forum |
Medicine forum |
Unix/Linux blog |
Unix/Linux documentation |
Unix/Linux forums
|
Powered by phpBB © 2001, 2005 phpBB Group
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
Forum index
»
Science and Technology
»
Math
