Skip to content
Technical & algorithm documentationSource reviewed

Data-quality algorithms and transformations

Technical inventory of missingness, distribution, response-pattern, multicollinearity and controlled transformation algorithms.

Overview

The quality engine profiles evidence without mutating the source and records every approved transformation as a new version.

Theoretical foundation

Diagnostics describe distributions, covariance and response patterns. Transformations are modelling decisions whose appropriateness depends on the study and instrument.

Algorithm and computation

Pandas/NumPy/SciPy compute missing patterns, sample skewness/kurtosis, Shapiro–Wilk for 3–5,000 nonconstant observations, z/MAD/IQR flags, guarded Mahalanobis distance, exact duplicates, selected-battery long strings and VIF. Available transformations include explicit row exclusion, recoding, reverse scoring, winsorisation, standardisation, mean centring and composite scoring; every operation is previewed.

Required data

A private immutable dataset version and researcher-selected variables for multivariate or battery checks.

Assumptions

Codes and valid ranges are correct, selected batteries are substantively related, and treatments are justified before approval.

Available settings

Selected diagnostics; recode mapping; scale endpoints; winsor percentiles; composite name; variables; row IDs; justification.

Worked example

A synthetic reverse-worded 1–5 item uses new = 6 − old after confirming endpoints. The preview shows changed cells before a new version is approved.

Limitations and non-claims

Little MCAR, multiple imputation, robust covariance distance, response-time analysis and automated case deletion are unavailable.

Validation notes

Quality calculations, safeguards, preview immutability and version creation are covered by repository tests.

How Surv.lytics computes Data-quality diagnostics
EstimatorDescriptive screening and guarded covariance diagnostics
Librariespandas 3.0.0; NumPy 2.4.2; SciPy 1.17.0
Missing dataDiagnostic-specific complete observations
ValidationImplemented with disclosed limitations
Defaults{}
Open full technical documentation

References and verification status

  1. Little, R. J. A. (1988). A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association, 83(404), 1198–1202. https://doi.org/10.1080/01621459.1988.10478722

    Unverified draftOpen DOI

Was this article helpful?

Your response helps the methodology team improve this documentation.