Testing & Metrics

Before you implement an extraction or classification model, you want to be confident that it effectively serves its purpose. You can test these models directly within DocCenter and view metrics on performance. If you see an opportunity to improve, you can create a new version of the model and make adjustments.

• Extraction models: Test a model and create a new version.
• Classification models: Test a model and create a new version.

After determining the models that perform best for your use case, use DocCenter to view performance metrics.

Classification accuracy

Classification accuracy is measured for each model version.

Accuracy is the ratio of correct predictions to total predictions, expressed as a percentage. "Correct predictions" includes true positives (correctly predicting that the document is part of a certain category) and true negatives (correctly predicting that the document isn't part of a certain category).

To view model version accuracy:

1. Navigate to the Classification page group > Models page.
2. In the list, locate the model you're interested in.
3. Refer to the Accuracy column to view latest model version's accuracy.
4. To view accuracy for all versions of a model, click on the Model Name.

How is accuracy calculated for classification?

Classification accuracy is calculated using historical reconciliation information. If a classification category is changed during reconciliation, it is considered inaccurate. Overall accuracy is calculated using a ratio of accurate and inaccurate classifications.

Example

You've created a model to recognize and classify two document types: invoices and purchase orders. You uploaded 35 test documents (all invoices), and the model positively identifies 27 of them as invoices and the remaining 8 as purchase orders. In reality, all 35 are invoices.

The model's accuracy is (27 / 35) * 100, or 77%.

Extraction accuracy

Extraction accuracy is measured and displayed at multiple levels:

• Individual fields
• Model version
• Extraction instance

To view model version accuracy:

1. Navigate to the Extraction page group > Models page.
2. In the list, locate the model you're interested in.
3. Refer to the Accuracy column to view the latest model version's accuracy.
4. To view accuracy for all fields in the latest version of a model, click on a Model Name.

Scroll or drag to pan · ⌘+/⌘− to zoom · Esc to close

To view instance accuracy:

1. Navigate to the Extraction page group > Instances page.
2. In the list, locate the instance you're interested in.
3. Refer to the Accuracy column to view the instance's accuracy.
4. To view accuracy for all fields in an instance, click on the Instance Name.

Scroll or drag to pan · ⌘+/⌘− to zoom · Esc to close

How is accuracy calculated for extraction?

Extraction accuracy is calculated using reconciliation information. If a field value is changed during reconciliation, it is considered inaccurate.

Level	Accuracy is calculated by…
Fields	dividing the number of times the field value wasn't changed during reconciliation by the number of total times the field was extracted for this model version.
Extraction instance	dividing the number of fields not changed during reconciliation by the number of total fields for this instance.
Model version	average extraction accuracy across all extraction instances associated with the model version.

The field data type also determines what's counted as "accurate":

• Normal field: A field is considered accurate if the extracted result is not changed, overridden, or cleared.
• List field: A list field is considered accurate if all elements were extracted correctly. If any were wrong, the field is considered inaccurate.
• Table field: A table field is calculated at the cell level. The accuracy of a table field is the number of correctly extracted cells divided by the total number of extracted cells.

Example

You've created a model to extract five fields from resumes:

• Name (Text)
• Email address (Text)
• Current title (Text)
• Current employer (Text)
• Skills (List)

During testing, you upload 20 resumes. All test documents have the five fields you're extracting:

The model is able to extract:

Field	Total extracted	Total correct extraction
Name	20	20
Email address	19	17
Current title	20	15
Current employer	15	15
Skills	20	15 (3 are missing a value, and 2 include incorrectly extracted values)

Here's how accuracy would be calculated for the document, fields, version, and instance:

Model version fields:

Field	Formula	Accuracy	Explanation
Name	20 / 20	100%	The field was extracted 20 times in 20 instances, and those 20 were correct.
Email address	17 / 20	85%	The field was extracted 19 times in 20 instances, but only 17 were correct.
Current title	15 / 20	75%	The field was extracted 20 times in 20 instances, but only 15 were correct.
Current employer	15 / 20	75%	The field was extracted 15 times in 20 instances, and those 15 were correct.
Skills	15 / 20	75%	The lists were extracted 20 times in 20 instances, but only 15 were entirely correct

Confidence score

The confidence score measures how sure the model is of its prediction. It is calculated for each prediction the model makes. For extraction models, confidence score is a reliability metric assigned to each word or line extracted from a document.

Most decisions (made by humans or computers) can vary in confidence. If someone asked you, "what's 2 + 2?" and you answer "4," your confidence in that answer is probably very high. But if someone asked you, "what did you eat for lunch last Tuesday?" you might not be so sure of your reply. "I think I had pizza" indicates you aren't very confident in your answer. Machine learning models apply the same confidence to their predictions. The confidence score indicates the system's certainty in the accuracy of the extracted value for a specific field.

How to use confidence scores

Enable Advanced Layout during model creation to utilize confidence scores and the threshold validation features.

Configure the threshold

To enable and configure the confidence threshold from the model summary:

1. Choose a model and navigate to the Straight Through Processing tab.
2. Under Settings, click Confidence Threshold.
3. Enable Use Confidence to Flag Fields and set the threshold value.
4. Click SAVE.

View low-confidence fields

After running extraction, you can review the count of low-confidence fields and their specific confidence scores.

View the count of fields below the confidence threshold:

• After the extraction completes, go to the Instances tab.
• On the Summary page, click Errors. This view shows model and field-level errors and states exactly how many fields fall below the confidence threshold

View confidence scores and resolve flags

To see individual confidence scores for fields that were flagged and correct the values, click Reconcile Fields. The reconcile fields screen applies these visual cues:

• Field-level warnings: These flags display directly on the data fields that fell below the configured threshold. They also show the specific confidence score, allowing you to quickly correct questionable values.
• Document-level flags: These prominently highlight the entire record, signaling that the document requires attention.

Feedback