Data Quality – AWS Glue Data Quality– AWS (original) (raw)

Serverless, cost-effective, petabyte-scale data quality without lock-in

AWS Glue is serverless, so you can scale without having to manage infrastructure. It scales for any data size, and it features pay-as-you-go billing to increase agility and improve costs. AWS Glue Data Quality uses Deequ, an open-source framework built by Amazon used to manage petabyte-scale datasets. Because it’s built using open source, AWS Glue Data Quality provides flexibility and portability without lock-in.

Get started quickly with automatic rule recommendations

AWS Glue Data Quality automatically computes statistics for your datasets. It uses these statistics to recommend a set of quality rules that checks for freshness, accuracy, integrity and even hard-to-find issues. You can adjust recommended rules, discard rules, or add new rules as needed. If it detects quality issues, AWS Glue Data Quality also alerts you so that you can act on them.

Uncover hidden data quality issues and anomalies using ML

AWS Glue Data Quality is intelligent. It learns patterns on data statistics gathered over time using ML algorithms. It detects anomalies, unusual data patterns and alerts users. It also auto-creates rules to monitor these specific patterns so that you can progressively build data quality rules.

Achieve data quality at rest and in pipelines

Your data rests in different repositories, and it moves from one repository to another. Monitoring data quality both once it lands and while it is in transit is important. AWS Glue Data Quality rules can be applied to data at rest in your datasets and data lakes and to entire data pipelines where data is in motion. You can apply rules across multiple datasets. For data pipelines built on AWS Glue Studio, you can apply a transform to evaluate the quality for the entire pipeline at a fraction of the cost as data is already in memory. You can also define rules to stop the pipeline if quality deteriorates, preventing bad data from landing in your data lakes.

Understand and correct data quality issues

Use over 25 out-of-the box AWS Glue Data Quality rules to validate your data and identify specific data that causes issues. Implement data quality checks that compare different data sets in disparate data sources in minutes with out-of-the-box rules. Using Glue ETL you can easily remediate these issues and ingest high quality data into your data repositories.