Extraction Model Development Strategy

This page describes a strategy for building production-ready extraction models in DocCenter. It walks through the journey from initial setup through prompt iteration to production readiness, starting with the simplest configuration and gradually adding complexity.

Before you begin, make sure you understand the design tradeoffs that affect accuracy, speed, and AI action consumption. Align on these priorities with stakeholders early so that your configuration choices are intentional.

Build an extraction model in phases

Building a production-ready extraction model is not a one-shot task. The recommended approach starts simple, validates results at each stage, and layers in advanced features only when simpler configurations fall short. Each iteration brings you closer to your accuracy target while keeping AI action consumption predictable.

The workflow has five phases:

1. Start with a baseline text-only extraction.
2. Reconcile results and establish test cases.
3. Iterate on prompts and configuration.
4. Add advanced features where needed.
5. Validate for production readiness.

The process of building an extraction model in DocCenter is summarized in the following flow chart:

Scroll or drag to pan · ⌘+/⌘− to zoom · Esc to close

Phase 1: Start with a baseline

Begin with the simplest possible extraction model configuration. Starting simple gives you a clear baseline to measure against. Every feature you add later should demonstrably improve accuracy or meet a specific business requirement. If you enable everything at once, you can't isolate what configurations are helping add value versus adding unnecessary cost.

To build a simple extraction model:

1. Create an extraction model with only the fields your business process requires.
2. Use the Generative AI method for all fields. Leave Vision, AI Review, and AI-Suggested Improvements disabled.
3. Write minimal field instructions—just the field label and basic description.
4. Select the best model available for initial testing.
5. Run your first extraction against representative sample documents. Use at least 15 to 30 documents that represent the variations you expect in production.

Phase 2: Reconcile and establish test cases

After your first extraction runs, reconcile the results to create a ground truth for future comparisons:

1. Reconcile the extracted fields for each sample document.
2. Set instances as test cases so you can quickly re-validate when you change the model.
3. Group related test cases into test suites organized by document type or specific fields you want to target.
4. Review accuracy metrics to identify which fields need improvement.

Phase 3: Iterate on prompts and configuration

With test cases in place, iterate on the model to improve accuracy. Work through configurations in the following order, adding complexity only when the previous configuration does not meet your accuracy target.

Text-only extraction

After making each of the following improvements, run your test suite to assess the improvements:

1. Refine field instructions by adding descriptions, location cues, and format guidance.
2. Add options rules for fields with a fixed set of valid values.
3. Test different LLMs—try both fast and reasoning models to compare accuracy.
4. Add examples to the prompt to guide the model with sample inputs and outputs.

Enable Vision

If text-only extraction does not meet your target for certain fields, enable Visual on fields that contain visual elements (checkboxes, signatures, stamps) or where spatial layout matters (complex tables). Remember that if you enable Vision for one field, enable it for all fields to avoid doubling AI action consumption.

Re-run your test suite and compare accuracy against the text-only baseline you established previously.

Use sections for long documents

If your documents exceed 40 pages or contain distinct logical parts, define sections to guide the LLM to specific page ranges.

Sections provide advantages such as:

• Assigning fields to their relevant sections reduces noise and improves accuracy.
• Sections also simplify the reconciliation task by grouping fields logically.
• Sections also address the issue of tables spanning multiple pages.
• For documents exceeding 100 pages, sections are essential to manage context and memory.

Phase 4: Add advanced features

Once your extraction model meets accuracy targets, consider additional features based on your business requirements:

Bounding boxes

Enable Bounding Boxes when human reviewers need to see the source location of extracted values in the document. Bounding boxes work best with structured documents that do not change format between versions. Accuracy of bounding box placement is not guaranteed for unstructured content or text-embedded fields.

Auto document rotation

Enable Automatic Rotation when processing scanned documents that may arrive in mixed orientations. Rotation corrects document orientation before extraction, which improves both extraction accuracy and the reconciliation experience.

AI-Suggested Improvements

Enable AI-Suggested Improvements after you have run several iterations and want AI-generated recommendations for improving model accuracy. This feature significantly increases AI action consumption, so use it selectively in non-production environments.

In production, enable this feature only if model accuracy falls below your target.

Enable Advanced Layout

If all previous configurations do not achieve the desired accuracy, enable Advanced Layout at the model level. For details on the cost profile and when this tradeoff is appropriate, see Use Advanced Layout only as a last resort.

Re-run tests and verify the accuracy improvement justifies the additional cost.

Phase 5: Validate for production readiness

Before deploying to production, confirm the model meets your requirements:

1. Run your full test suite against the final model version.
2. Verify accuracy meets the target agreed upon with stakeholders.
3. Confirm the chosen LLM, field configuration, and advanced features balance accuracy, speed, and consumption appropriately.
4. If your workflow targets straight-through processing, configure confidence thresholds, AI Review, sampling, and validations before going live.
5. Create a new model version to capture the final configuration.
6. Deploy the model to higher environments.

When to create a new version versus update

Create a new model version for meaningful changes that benefit from traceability, A/B comparison, and easy rollback. Update the existing version only for minor corrections such as fixing a typo or adjusting a single instruction.

We also recommend creating a new version when the selected LLM is scheduled for deprecation. Run your test suites after switching models to validate performance before deploying.

Feedback