PyDataQuality: Enterprise Data Quality Engine

Why PyDataQuality?

You just received a new dataset. You need to know what's wrong with it and how to fix it - fast. Existing tools either:

Take too long (pandas-profiling: 10+ minutes, 200MB reports)
Require too much setup (Great Expectations: YAML configs, checkpoints)
Don't give actionable insights (pandas .describe(): just basic stats)

PyDataQuality fills the gap between "too simple" and "too complex". It just works, without making you read 50 pages of documentation.

Feature	pandas-profiling	Great Expectations	PyDataQuality
One-liner usage	NO	NO	YES
Extract bad rows	NO	NO	YES
CLI support	NO	NO	YES
Fast on large data	NO	YES	YES
No config needed	YES	NO	YES
AI integration	NO	NO	YES
Beginner-friendly	PARTIAL	NO	YES

Real-World Example (1M+ rows)

With pandas-profiling, generating a report takes 10+ minutes and creates a 200MB HTML file. With PyDataQuality, it takes 30 seconds and allows immediate extraction of problematic rows.

# Takes 30 seconds
analyzer = pdq.analyze_dataframe(df)
bad_ages = analyzer.get_problematic_rows('age', 'outliers')
bad_ages.to_csv('fix_these.csv')  # Send to data team

Core Capabilities

Data Quality Analysis: Detect missing values, outliers, inconsistencies, and data type issues.
Data Extraction: Pinpoint and extract problematic rows for remediation via get_problematic_rows().
Multi-Format Support: Native support for CSV, Excel (.xlsx, .xls), JSON, and Parquet files.
CLI Interface: Auto-detects file formats from the terminal - no coding required.
Interactive Notebook Display: Direct rendering in Jupyter/Colab with show_report().
Batch Sampling: Process large datasets (GBs) efficiently with chunk-based sampling.
Custom YAML Rules: Enterprise-grade validation with configuration files.
AI Remediation Prompts: Auto-generated Python scripts for fixing detected issues.

Quickstart Guide

Installation

# Install from local development
pip install -e .

# Or install requirements directly
pip install pandas numpy matplotlib seaborn jinja2

# For full feature support (including notebooks)
pip install ".[notebook]"

CLI Usage (No Code Required)

# Basic check (generates report.html)
python cli.py data.csv

# Specify output folder
python cli.py data.xlsx --output my_reports/

# Generate JSON report instead of HTML
python cli.py data.parquet --report json

Basic Usage (Python)

import pandas as pd
import pydataquality as pdq

# Load your data
df = pd.read_csv('your_data.csv')

# 1. Comprehensive analysis
analyzer = pdq.analyze_dataframe(df, name="My Dataset", verbose=True)

# 2. Interactive Report (Jupyter/Colab)
pdq.show_report(analyzer, theme='creative')

# 3. Generate HTML file
pdq.generate_report(analyzer, output_path="quality_report.html", format='html')

# 4. Get AI assistance (optional)
prompt = pdq.generate_ai_prompt(analyzer)
print(prompt)  # Copy to ChatGPT/Claude/Gemini for automated fixes

Advanced Features & Workflows

Extracting Bad Data for Remediation

You found 500 outliers in 'Age'. Now what? Use get_problematic_rows to extract them into a new dataframe for cleaning.

# 1. Identify issues
analyzer = pdq.analyze_dataframe(df)

# 2. Extract rows with invalid ages
bad_age_rows = analyzer.get_problematic_rows('Age', issue_type='outliers')

# 3. Inspect or Fix
print(f"Found {len(bad_age_rows)} bad rows")
bad_age_rows.to_csv("ages_to_fix.csv")

Data Drift Detection

# Compare two analyzers
drift_stats = pdq.compare_reports(analyzer_last_week, analyzer_today)
print(drift_stats)

Custom Configurations

config = {
    'missing_critical': 0.4,       # 40% instead of 30%
    'outlier_threshold': 3.0,      # Use 3*IQR for outliers
    'zero_threshold': 0.9,         # Strict zero check
    
    # Whitelist specific values (e.g. known sentinel values)
    'exclude_values': {
        'customer_age': [999, -1],
        'region': ['Unknown', 'N/A']
    }
}

analyzer = pdq.analyze_dataframe(df, config=config)

Research Paper & Academic Backing

PyDataQuality is backed by an academic paper that formally describes its architecture, mathematical foundations, and empirical benchmarks:

PyDataQuality: An Actionable, Lightweight Data Profiling and Distribution Drift Detection Framework for Production Machine Learning Pipelines — Dominion Akinrotimi, 2026

The paper covers:

The five-layer modular architecture
Formal derivations of Shannon Entropy, IQR (Tukey Fences), PSI, and the KS Test
Benchmark results showing ~80ms constant-time execution in sampled mode
An MLOps case study demonstrating drift detection as a model quality gate