Great Expectations

GX Core is an open-source Python framework designed for testing, validating, and documenting data quality across data pipelines. Built to integrate directly into existing workflows and interactive environments like Jupyter Notebooks, it allows data engineers, data scientists, and analysts to define expected data standards and continuously verify datasets against those requirements.

The framework automatically generates Data Docs, which render test results and data quality metrics into accessible visual interfaces and plain-language summaries for technical and non-technical stakeholders. Through integrations with popular data warehouses, storage systems, and workflow orchestrators, GX Core helps prevent corrupted or invalid data from propagating downstream.

Key features

• Automated data validation: Evaluates datasets against defined expectations and rules to ensure data consistency and accuracy.
• Data Docs generation: Automatically creates plain-language, visual documentation of test results and data quality profiles for team visibility.
• Pipeline orchestration integration: Connects with workflow tools like Apache Airflow, Dagster, and Prefect to trigger validation steps in real time.
• Broad data store compatibility: Works with engines and storage formats including Databricks, Snowflake, BigQuery, PostgreSQL, Pandas, and Spark.
• Alerting and custom actions: Triggers notifications via Slack, Email, PagerDuty, or Microsoft Teams when critical data quality checks fail.
• Custom expectations: Allows teams to write programmatic rules in Python to validate unique schema, statistical, or business logic requirements.