质量数据管理

gaowenhua

数据管理

由于数据清洗（Data Cleaning）工具通常简单地被称为数据质量（Data Quality）工具，因此很多人认为数据质量管理，就是修改数据中的错误、是对错误数据和垃圾数据进行清理。这个理解是片面的，其实数据清洗只是数据质量管理中的一步。数据质量管理（DQM），不仅包含了对数据质量的改善，同时还包含了对组织的改善。针对数据的改善和管理，主要包括数据分析、数据评估、数据清洗、数据监控、错误预警等内容；针对组织的改善和管理，主要包括确立组织数据质量改进目标、评估组织流程、制定组织流程改善计划、制定组织监督审核机制、实施改进、评估改善效果等多个环节。
任何改善都是建立在评估的基础上，知道问题在哪才能实施改进。通常数据质量评估和管理评估需通过以下几个维度衡量。
评估
Data managementSince data cleaning tools are often simply referred to as data quality tools, many people think of data quality management as correcting errors in data and cleaning up erroneous and junk data. This understanding is one-sided. In fact, data cleaning is only a step in data quality management. Data Quality Management (DQM) not only includes the improvement of data quality, but also the improvement of the organization. For data improvement and management, it mainly includes data analysis, data evaluation, data cleaning, data monitoring, error warning, etc. For organizational improvement and management, it mainly includes establishing organizational data quality improvement goals, evaluating organizational processes, and formulating organizational process improvement Plan, formulate organizational supervision and review mechanism, implement improvement, evaluate improvement effect and other links.Any improvement is based on evaluation, knowing where the problem is before implementing the improvement. Usually data quality assessment and management assessment need to be measured through the following dimensions.Evaluate

gaowenhua

质量数据管理特点：

完整性 Completeness：完整性用于度量哪些数据丢失了或者哪些数据不可用。
规范性 Conformity：规范性用于度量哪些数据未按统一格式存储。
一致性 Consistency：一致性用于度量哪些数据的值在信息含义上是冲突的。
准确性 Accuracy：准确性用于度量哪些数据和信息是不正确的，或者数据是超期的。
唯一性 Uniqueness：唯一性用于度量哪些数据是重复数据或者数据的哪些属性是重复的。
关联性 Integration：关联性用于度量哪些关联的数据缺失或者未建立索引。

Quality Data Management Features:

Integrity Completeness: Integrity is a measure of what data is missing or not available.Normative Conformity: Conformity is used to measure which data is not stored in a uniform format
.Consistency: Consistency is used to measure which data values ​​are conflicting in information meaning.
Accuracy Accuracy: Accuracy is a measure of which data and information is incorrect, or data is overdue.
Uniqueness Uniqueness: Uniqueness is used to measure which data is repeated data or which attributes of data are repeated.

Relevance Integration: Relevance is a measure of which associated data is missing or not indexed

gaowenhua

数据管理不当的因素：
信息因素：产生这部分数据质量问题的原因主要有：元数据描述及理解错误、数据度量的各种性质（如：数据源规格不统一）得不到保证和变化频度不恰当等。
技术因素：主要是指由于具体数据处理的各技术环节的异常造成的数据质量问题。数据质量问题的产生环节主要包括数据创建、数据获取、数据传输、数据装载、数据使用、数据维护等方面的内容。
流程因素：是指由于系统作业流程和人工操作流程设置不当造成的数据质量问题，主要来源于系统数据的创建流程、传递流程、装载流程、使用流程、维护流程和稽核流程等各环节。
管理因素：是指由于人员素质及管理机制方面的原因造成的数据质量问题。如人员培训、人员管理、培训或者奖惩措施不当导致的管理缺失或者管理缺陷。

Factors of improper data management:Information factor: The main reasons for this part of data quality problems are: metadata description and misunderstanding, various properties of data measurement (such as inconsistent data source specifications) cannot be guaranteed, and the frequency of changes is inappropriate.
Technical factors: mainly refer to the data quality problems caused by the abnormality of each technical link of specific data processing. The production links of data quality problems mainly include data creation, data acquisition, data transmission, data loading, data use, data maintenance and so on
.Process factors: refer to the data quality problems caused by improper system operation process and manual operation process settings, mainly from the system data creation process, transfer process, loading process, use process, maintenance process and audit process.
Management factors: refers to the data quality problems caused by the quality of personnel and management mechanism. Such as lack of management or management defects caused by improper personnel training, personnel management, training or reward and punishment measures.

gaowenhua

质量数据管理和改善：1. 定义和商定问题、时机和目标，以指导整个数据质量管理的工作。
2. 收集、汇总、分析有关形式和信息环境。设计捕获和评估的方案。
3. 按照数据质量维度对数据质量进行评估。
4. 使用各种技术评估劣质数据对业务产生的影响。
5. 确定影响数据质量的真实原因，并区分这些原因的影响的数据质量的级别。
6. 最终确定行动的建议，为数据质量改善制定方案，包括数据级和组织级的。
7. 建立数据错误预防方案，并改正当前数据问题。
8. 通过改进组织管理流程，最大限度控制由管理上的缺陷造成的数据质量问题。
9. 对数据和管理实施监控，维护已改善的效果。
10.沟通贯穿管理始终，循环的评估组织管理流程，以确保数据质量改善的成果得到有效保持

Quality Data Management and Improvement:

1. Define and agree on issues, timing, and goals to guide the overall data quality management effort.2. Collect, aggregate, and analyze relevant forms and information environments. Design capture and evaluation scenarios.3. Evaluate data quality according to data quality dimensions.4. Use a variety of techniques to assess the impact of poor quality data on the business.5. Identify the real causes that affect data quality and differentiate the level of data quality that affects these causes.6. Finalize recommendations for actions and develop plans for data quality improvement, both at the data level and at the organizational level.7. Establish data error prevention programs and correct current data problems.8. Maximize control of data quality issues caused by management deficiencies by improving organizational management processes.9. Monitor data and management to maintain improved performance.

10. Communicate throughout the management, cyclically evaluate the organizational management process to ensure that the results of data quality improvement are effectively maintained