How to Avoid Data Rupture and Data Lapse

Data sprawl refers to an increase in data sources and silos within your organization. It has the unfortunate side effect of diverting talented staff away from impactful projects and tasks and into administration roles that cannot easily be analyzed, automated or managed – ultimately leading to reduced productivity, or in some extreme cases even killing your business altogether. This article details tools and strategies you can use to reduce data sprawl.

SGP is an R package that provides classes, functions and data to calculate student growth percentiles and percentile growth projections/trajectories from large scale longitudinal education assessment data. Using quantile regression it estimates the conditional density associated with each student’s achievement history before using these estimated percentiles to calculate a matrix of student growth percentiles based on these estimated percentiles.

The SGP package offers two data formats, WIDE and LONG. Lower level functions (studentGrowthPercentiles and studentGrowthProjections) use WIDE format data while wrapper functions like summarySGP and studentAggregates are optimized to work with LONG format. If you plan to run SGP analyses year after year, LONG format is highly recommended as this will facilitate operational analysis processes more smoothly while saving significant preparation and storage costs when compared with working with WIDE data formats.

To perform SGP analyses, a computer with R installed is required. R is an open-source, free program available on Windows, OSX, and Linux operating systems – getting started is straightforward thanks to plenty of resources online that provide help getting things rolling fast.

As well as collecting observations, the SGP site boasts sophisticated modeling and simulation capabilities. These range from simple models used to understand basic processes in the atmosphere to more sophisticated large-eddy simulation (LES) models used for climate modeling, aerosol research and air quality monitoring applications.

All these capabilities depend on high-quality, consistent and comprehensive datasets collected and made publicly available by ARM. Scientists around the globe can use this wide variety of data sets for various research questions that they pose. These datasets range from single observation analyses to multi-observation process studies and assimilation into Earth system models. All the SGP observatory’s collected data are transmitted directly to the ARM Data Center for free availability via Data Discovery. ARM also strives to offer a model-observation framework in which its instruments can be seamlessly integrated into atmospheric models, enabling researchers to examine physical processes without needing to visit SGP Observatory directly. Through its LES sgp Symbiotic Simulation and Observation (LASSO) activity, researchers are now able to study physical processes of interest without visiting. It has accomplished this through connecting instrument data directly to large eddy simulation models which run alongside instrument data at SGP observatory.