Re: How to estimate missing data points in a time series (financial time series e.g. stock quote, asset price)

IRISHSTAT wrote:
On Dec 19, 6:46 am, "David Jones" <dajx...@xxxxxxxxx> wrote:
IRISHSTAT wrote:
On Dec 17, 1:08 pm, "arun.kumar.s...@xxxxxxxxx"
<arun.kumar.s...@xxxxxxxxx> wrote:
hi all Statisticians,

In your opinion, what is the best way exists today to estimate the
missing data points in a time series? Here my concern is estimated
data-points (as proxy for missing points) should be as accurate as
possible as well as statistically valid.

Can you please suggest me some good text books as well as online
note on that?

King regards,

One more comment . Given that you have atime series it is possible
to identify a robust ARIMA model which is then useful in making
predictions. A missing value can be predicted thus obtaining an
estimate.

Dave R

The above "making predictions" may imply only using the observations
before any missing data. I guess time-series packages may do the
following in a better way, and it is just a version of the
suggestion by Mark Fisher, but the following would work for general
patterns of missing values ... (1) construct a covariance matrix
from a fitted model for all the observations around and including
the missinf values. Make the extent of the time-points included
reasonable with respect to the model, and it doesn't matter if there
are valid observations between the missing ones. (2) construct the
conditional covariance matrix of the missing values given the
observed ones according to the usual matrix rules for the Normal
distribution. (3) used the expectation of the conditional
distribution as the "infilled value" (4) use the conditional
variance to indicated the estimation error.

David Jones- Hide quoted text -

- Show quoted text -

David,

As you very politely pointed out time series approaches using the
general ARIMA form use the observations before any missing data.
However one can always reverse the process by using more recent data
to predict an older one i.e. backcasting . This is easily accomplished
by reordering the time series from latest to oldest. Iterating back
and forth like this was suggested to me by Ted Anderson of the T.W.
ANDERSON fame as a way of using "information" surrounding the missing
value.

Regards

Dave

There is an "optimal" way of combining forward and backward passes through the data that can be found in some expositions of the theory of Kalman Filtering ... ordinary Kalman Filtering gives an optimal estimate based in information up to "time-now" in the interative scheme but there is version where the basic filtering steps are run up to the end of the observed data, followed by a reverse-time-stepping interation (with different details).

One advantage of the covariance approach is that one can decide to limit the time-range over which information is used, which may be helpful for convenience of computation or if there is doubt that the process really is stable over the requisite time-period.

David Jones
.

Relevant Pages

  • Re: Automatically substituting missing data
    ... Is it not the case that if there insufficient entries pre and post the ... hence you still having missing data. ... > Hi Nigel ...
    (microsoft.public.excel.programming)
  • Re: Accomodating non-valid dates in a date/time field
    ... And possibly add a yes/no field that stores -1 if the date is missing. ... Microsoft Access MVP ... provided on the paper form, when entering this data into the database ... indicate missing data and we'd like this coding to be consistent. ...
    (microsoft.public.access.tablesdbdesign)
  • Re: How to estimate missing data points in a time series (financial time series e.g. stock quote, as
    ... data-points (as proxy for missing points) should be as accurate as ... As you very politely pointed out time series approaches using the ... general ARIMA form use the observations before any missing data. ... and forth like this was suggested to me by Ted Anderson of the T.W. ...
    (sci.stat.math)
  • Re: Vantage Pro and BST -> GMT
    ... I have the data for March, and (in the Weatherlink software), there ... is indeed a 'missing' hour. ... No missing data, of course, just a gap in ... In the Vantage Pro setup there's an option to use GMT all the time. ...
    (uk.sci.weather)
  • Re: paired observations
    ... by imputation of missing data). ... as inference of missing values) is the potential for selection bias - ...
    (sci.stat.consult)