DJ Patil

Succeeding with data isn’t just a matter of putting Hadoop in your machine room, or hiring some physicists with crazy math skills. It requires you to develop a data culture that involves people throughout the organization.

One of the most cited articles in business on how to think about the emerging area of data science

The tips and key lessons of how to turn data into products. In this publication we talk through the idea of data jujitsu. This is a frame of how to approach building data products.

As data science evolves to become a business necessity, the importance of assembling a strong and innovative data teams grows. In this in-depth report, data scientist DJ Patil explains the skills, perspectives, tools and processes that position data science teams for success.

Topics include: What it means to be "data driven." The unique roles of data scientists. The four essential qualities of data scientists. Patil's first-hand experience building the LinkedIn data science team.

In this paper, we introduce a new, local formulation of the ensemble Kalman filter approach for atmospheric data assimilation. Our scheme is based on the hypothesis that, when the Earth's surface is divided up into local regions of moderate size, vectors of the forecast uncertainties in such regions tend to lie in a subspace of much lower dimension than that of the full atmospheric state vector of such a region. Ensemble Kalman filters, in general, take the analysis resulting from the data… Show more

In this paper, we introduce a new, local formulation of the ensemble Kalman filter approach for atmospheric data assimilation. Our scheme is based on the hypothesis that, when the Earth's surface is divided up into local regions of moderate size, vectors of the forecast uncertainties in such regions tend to lie in a subspace of much lower dimension than that of the full atmospheric state vector of such a region. Ensemble Kalman filters, in general, take the analysis resulting from the data assimilation to lie in the same subspace as the expected forecast error. Under our hypothesis the dimension of the subspace corresponding to local regions is low. This is used in our scheme to allow operations only on relatively low-dimensional matrices. The data assimilation analysis is performed locally in a manner allowing massively parallel computation to be exploited. The local analyses are then used to construct global states for advancement to the next forecast time. One advantage, which may take on more importance as ever-increasing amounts of remotely-sensed satellite data become available, is the favorable scaling of the computational cost of our method with increasing data size, as compared to other methods that assimilate data sequentially. The method, its potential advantages, properties, and implementation requirements are illustrated by numerical experiments on the Lorenz-96 model. It is found that accurate analysis can be achieved at a cost which is very modest compared to that of a full global ensemble Kalman filter. Show less

We consider the estimation of the state of a large spatio-temporally chaotic system from noisy observations and knowledge of a system model. Standard state estimation techniques using the Kalman filter approach are not computationally feasible or systems with very many effective degrees of freedom. We present and test a new technique (called a Local Ensemble Kalman Filter), generally applicable to large spatio-temporally chaotic systems for which correlations between system variables evaluated… Show more

We consider the estimation of the state of a large spatio-temporally chaotic system from noisy observations and knowledge of a system model. Standard state estimation techniques using the Kalman filter approach are not computationally feasible or systems with very many effective degrees of freedom. We present and test a new technique (called a Local Ensemble Kalman Filter), generally applicable to large spatio-temporally chaotic systems for which correlations between system variables evaluated at different points become small at large separation between the points. Show less

Ensemble Kalman filtering was developed as a way to assimilate observed data to track the current state in a computational model. In this paper we show that the ensemble approach makes possible an additional benefit: the timing of observations, whether they occur at the assimilation time or at some earlier or later time, can be effectively accounted for at low computational expense. In the case of linear dynamics, the technique is equivalent to instantaneously assimilating data as they are… Show more

Ensemble Kalman filtering was developed as a way to assimilate observed data to track the current state in a computational model. In this paper we show that the ensemble approach makes possible an additional benefit: the timing of observations, whether they occur at the assimilation time or at some earlier or later time, can be effectively accounted for at low computational expense. In the case of linear dynamics, the technique is equivalent to instantaneously assimilating data as they are measured. The results of numerical tests of the technique on a simple model problem are shown. Show less

In this paper simple graphical representations of the time series and the errors made by a simple predictive model of the time series (known as residual delay maps) are used to extract information about the nature of the time evolution of the system (in this paper referred to as the dynamics). Two different uses for these graphical representations are presented in this paper. First, a test for the comparison of two competing models or of a model and observational data is proposed. The utility… Show more

In this paper simple graphical representations of the time series and the errors made by a simple predictive model of the time series (known as residual delay maps) are used to extract information about the nature of the time evolution of the system (in this paper referred to as the dynamics). Two different uses for these graphical representations are presented in this paper. First, a test for the comparison of two competing models or of a model and observational data is proposed. The utility of this test is that it is based on comparing the underlying dynamical processes rather than looking directly at differences between two datasets. An example of this test is provided by comparing station data and NCEP–NCAR reanalysis data on the Australian continent. Second, the technique is applied to the global NCEP–NCAR reanalysis data. From this a composite image is created that effectively identifies regions of the atmosphere where the dynamics are strongly dependent on lowdimensional nonlinear processes. It is also shown how the transition between such regions can be depicted using residual delay maps. This allows for the investigation of the conjecture of Sugihara et al.: sites in the midlatitudes are significantly more nonlinear than sites in the Tropics. Show less

A statistic, the BV (bred vector) dimension, is introduced to measure the effective local finite-time
dimensionality of a spatiotemporally chaotic system. It is shown that the Earth’s atmosphere often has
low BV dimension, and the implications for improving weather forecasting are discussed.

n this paper, the spatiotemporally changing nature of predictability is studied in a reduced-resolution version of the National Centers for Environmental Prediction (NCEP) Global Forecast System (GFS), a state-of-the-art numerical weather prediction model. Atmospheric predictability is assessed in the perfect model scenario for which forecast uncertainties are entirely due to uncertainties in the estimates of the initial states. Uncertain initial conditions (analyses) are obtained by… Show more

n this paper, the spatiotemporally changing nature of predictability is studied in a reduced-resolution version of the National Centers for Environmental Prediction (NCEP) Global Forecast System (GFS), a state-of-the-art numerical weather prediction model. Atmospheric predictability is assessed in the perfect model scenario for which forecast uncertainties are entirely due to uncertainties in the estimates of the initial states. Uncertain initial conditions (analyses) are obtained by assimilating simulated noisy vertical soundings of the “true” atmospheric states with the local ensemble Kalman filter (LEKF) data assimilation scheme. This data assimilation scheme provides an ensemble of initial conditions. The ensemble mean defines the initial condition of 5-day deterministic model forecasts, while the time-evolved members of the ensemble provide an estimate of the evolving forecast uncertainties. The observations are randomly distributed in space to ensure that the geographical distribution of the analysis and forecast errors reflect predictability limits due to the model dynamics and are not affected by inhomogeneities of the observational coverage. Show less