Free cookie consent management tool by TermsFeed

Extraction Models

On This Page
The capabilities described on this page are included in Appian's advanced and premium capability tiers. Usage limits may apply.

DocCenter extraction capabilities

Appian DocCenter uses artificial intelligence (AI) to extract structured data from structured and unstructured documents and forms. DocCenter extraction models minimize repetitive manual data extraction and eliminate the need for optical character recognition (OCR) software.

An extraction model takes a document as input, identifies field values based on labels you define, and provides that data as structured output. You can also add a user reconciliation task to manually review that data was extracted correctly.

Extraction tab in DocCenter

The Extraction tab in DocCenter has two pages: Models and Instances.

  • Models: Lists the extraction models you've previously created. When you click on a model, you see version information and performance metrics.
  • Instances: Lists the processes where extraction models ran.

Scroll or drag to pan · ⌘+/⌘− to zoom · Esc to closeimages/extraction-models-page.png

About natural language prompting

You use natural language prompting to interact with AI models that power Appian's classification and extraction capabilities. Natural language means you describe a task in your own words, with no need to write code or AI-specific terminology. The models are designed to understand and act on natural language, but for the best results, you'll want to provide a lot of context.

Imagine you are onboarding a new employee. On their first day, you are explaining to them how to identify a document type or the various fields within a document type. Your new employee is eager to help, but they might not have the experience to know what traits or fields to look for in an invoice or purchase order. So you provide them with examples or guidelines to help them see what patterns or traits are present in invoices or purchase orders.

With these examples and details, the employee can quickly identify these documents and fields accurately. Or maybe you notice they need some additional examples. With some practice, the employee gets better at completing their task.

The technology that powers DocCenter works much the same way. The more detail you can provide, the more likely the AI model will be to consistently provide correct results.

Tip:  Take a look at our tips for writing effective prompts.

Extraction model limits

DocCenter extraction models have the following limits:

Configuration Limit
File size Up to 100 MB.
Pages (with reconciliation) Up to 500 pages.
Pages (without reconciliation) Up to 1,000 pages.
Pages (Visual extraction) Up to 500 pages (with reconciliation) or 1,000 pages (without reconciliation).
Fields per model Up to 100 fields.
Large Table file size Up to 10 MB.
Large Table cells Up to 2 million cells across all sheets.
Large Table rows Up to 40,000 rows per sheet.
Large Table columns Up to 200 columns per sheet.
Large Table sheets Up to 100 sheets per file.

If your document requires more than 100 fields, create multiple extraction models for different parts of the document. You can also use a table field to consolidate related data points into fewer fields.

Supported file types

DocCenter extracts data from the following file types:

  • CSV
  • DOCX (DOC files are not supported)
  • EML
  • GIF
  • HTML
  • JPG, JPEG
  • MSG
  • PDF
  • PNG
  • TIFF
  • XLSX, XLS, XLM

Supported languages

DocCenter supports the following languages for all configurations of extraction models:

  • English
  • French
  • German
  • Italian
  • Portuguese
  • Spanish

Most other languages are supported with certain configurations enabled. To extract data in languages other than English, select Generative AI as the method and enable Visual field extraction as you define fields. Bounding boxes are not supported in this scenario.

Document encryption

PDFs can contain sensitive information that encryption protects from unintended recipients. The type of encryption on a PDF affects whether DocCenter can extract data from the file.

DocCenter extracts data from unencrypted PDFs. DocCenter cannot extract data from RC4 encrypted PDFs in GovCloud environments.

Create and configure an extraction model

To create an extraction model, navigate to the Extraction page group, click Models, and click + Extraction Model. The creation wizard guides you through five steps: model details, sections, field creation, field configuration, and prompt setup.

Step 1: Enter model details

In step 1 of the creation wizard, enter the model details:

Field Description
Name Name to uniquely identify model.
Description Description of model (optional).
Key A read-only string, automatically generated to uniquely identify model. You'll use this value as the modelKey parameter when you use your extraction model as a subprocess.

Configure Additional Options for the extraction model:

Field Description
Bounding Boxes (Recommended) Displays bounding boxes in the reconciliation task. These boxes show where the model found the data to extract from the document.
Automatic Rotation If needed, automatically rotate the document to its proper orientation to help improve extraction accuracy.
Log AI Usage View detailed information on how this model uses AI-powered features. Must be enabled to generate suggested improvements, but may slow down overall extraction times.

When enabled, AI usage details appear in the AI Usage tab when viewing an instance.
AI-Suggested Improvements Get AI-generated suggestions on how to improve model accuracy. Enabling this option significantly increases AI action consumption because it runs an AI agent that makes multiple calls per instance.

Step 2: Configure sections

In step 2 of the creation wizard, configure sections that appear in your document:

  • Section Name: Name of section.
  • Section Type: Choose how to configure sections in the document, if any.

Sections break larger documents into smaller chunks to help focus the LLM for improved accuracy, conserve AI actions, and optimize processing time. Sections also organize data on the reconciliation screen. For more information about available section types, see Extraction model sections.

Step 3: Create fields

In step 3, choose how to create fields for the extraction model:

  • Create & Configure Manually: Select this option if you want to add all fields manually in step 4.
  • Describe Fields: Using natural language, describe the type of document you're extracting data from and AI Copilot suggests fields that appear in those documents.
  • Extract from Document: Upload a digital PDF, Text file, or Excel workbook and describe the document. Scanned PDFs are not supported for this option.

Click NEXT to proceed to field configuration.

Step 4: Configure fields

In step 4, configure the fields on each tab:

Note:  You can create up to 100 fields per extraction model.

  • Field Definition:

    Field Definition
    Label Field name as it most often appears in documents.
    Key Field name as you'd like it to appear in a structured response such as JSON.
    Data Type Type of data present in this field. Options include Text, Paragraph, Integer, Decimal, Date, DateTime, Boolean, Table, Large Table.
    List Select this option if the field contains multiple values. If you selected Table or Large Table as the Data Type, click Edit table here.
    Method Choose how you'd like to extract the field data.
    Instructions Provide input that corresponds to the method you select. See Methods for more information.
    Visual When enabled, the AI model uses additional capabilities to locate the field. We recommend selecting this option for fields containing visual elements, such as a checkbox or a wet signature, or for elements where spatial layout is important such as a complex table. Visual supports up to 500 pages (with reconciliation) or 1,000 pages (without reconciliation) with the latest models (Sonnet 4+, Nova 2 Lite). If you are using Visual for one field, use it for all fields in order to reduce AI Action consumption. Click VISUAL SELECT ALL to enable Vision on all fields at once, or VISUAL CLEAR ALL to disable it. For guidance on when to use Visual versus Advanced Layout, see Design Considerations.
  • Extraction Details:

    Field Definition
    Label Read-only label of the field.
    Section If using section, choose the section where the field is most likely to appear.
    Options Rule Generates the options included in the prompt and as a dropdown on the reconcile screen.
    Post Process Rule Processes fields before displaying them on the reconcile screen. This can be used for doing things like reformatting dates or phone numbers.
  • Reconciliation:

    Field Definition
    Label Read-only label of the field.
    Required If selected, this field is required to be populated during reconciliation and throws a validation error if blank.
    Field Validation Used to run additional validations on the extracted value in the reconciliation form. For example, flagging a date field value that's in the future.
    Reconciliation Instructions Instructions that appear for this field in the reconciliation task.
    Hidden Choose to hide a field from the reconciliation task.
    Exclude from Metrics Exclude this field from the accuracy calculation. This is useful for calculated fields or other information that won't appear in the reconciliation task.

Click NEXT to proceed to prompt configuration.

Step 5: Configure the extraction prompt

In step 5 of the creation wizard, refine and preview the prompt that the extraction model uses to extract data:

Field Definition
Large Language Model Choose the provider and model you'd like to use to extract data. The dropdown displays all preconfigured provider and model combinations available in your environment (for example, "Appian - Claude Sonnet 4.6 (Reasoning)" or "AWS - Claude Sonnet 4.6 (Reasoning)"). Some models prioritize speed, while others provide more advanced reasoning. Choosing the right model depends on your extraction model's performance and accuracy needs. To add more providers, see AI Services.
Extended Thinking Enable extended thinking to enhance reasoning for complex tasks. When enabled, the model works through its approach more thoroughly, breaking down intricate tasks into logical, sequential steps before generating a final response. This increases accuracy for complex documents that require multi-step logic or nuanced interpretation. Extended thinking appears as a toggle directly below the LLM dropdown. When enabled, the model's reasoning is available in the AI Usage log. Note that not all LLM models support extended thinking and enabling this option increases AI action consumption.
Role Tell the model what it is responsible for in this task. The role is pre-populated with text, but you can edit it as you see fit.
Additional Instructions Tell the model anything else it should keep in mind as it performs analysis and extracts data. For example, you could tell the model, "Not all documents have all information and that's ok."
Examples Add inputs and outputs the AI model can use as an example of what you expect to receive as a response when presented with text.

Click CREATE to proceed with creating the extraction model.

Extract from split classification documents

If you process large PDF packets containing multiple documents, you can use a classification model to automatically identify and split documents, then send those documents to the appropriate extraction model. This workflow allows you to:

  • Classify and split a large packet into individual documents.
  • Extract data from each split document using extraction models.
  • Maintain a clear audit trail of the original packet and its component documents.

Packet classification to extraction workflow

  1. Classify the packet: Run a classification model configured with the Split Packets Into Individual Documents option on your large PDF packet. The model identifies document boundaries and assigns each section to a category.
  2. Reconcile splits (optional): If needed, reconcile the classification results to correct any inaccurate page ranges or document types.
  3. Extract from split documents: The split documents are automatically generated and named using the convention: [Original Filename]-[Category]-[PageRange].pdf
  4. Run extraction models: Use your extraction models on the split documents. Since each split contains a single document type, your extraction models can focus on extracting data from that specific document type.
  5. Process results: Review and reconcile extraction results as needed using the standard extraction reconciliation workflow.

Test an extraction model

Test an extraction model in the DocCenter application to verify accuracy before deploying to production. You can review accuracy metrics and make adjustments before the model extracts real data.

Note:  Test extraction models against at least 30 unique documents to ensure statistically significant results.

Select documents that include variations and edge cases, such as low-quality scans, unusual formatting, or documents with missing data fields, to ensure robustness against real-world data.

To test a model:

  1. In the Extraction page group, click Models.
  2. In the list, find and click on the model you want to test.
  3. Click RUN EXTRACTION.
  4. Choose what type of document you want to classify:
    • Existing Document: Choose from a file in Appian.
    • New Document: Upload a file.
    • Add Text: Enter plain text.
  5. Optionally, turn on the Set as Test Case toggle to use this document to quickly and accurately validate new model versions.
    • If you have existing test suites, an Add to Test Suite dropdown appears when you turn on the Set as Test Case toggle. Select a test suite to add the test case.
  6. Click EXTRACT.
  7. Click the Instance tab, where your test run appears.
  8. Click the instance that represents your test to learn more, or glance at the Accuracy column to quickly see how well the model extracted data.
  9. If the instance status is Extracted, click Reconcile Fields in the related action menu.
  10. After running the full test set, refine the model as needed:
    • If you notice that the model isn't performing well enough for your purposes, create a new version.
    • If testing reveals errors, edit the prompt with additional instructions to cover the identified failures.

Tip:  To save time during frequent testing, set a particular instance as a test case. Test cases use the documents you previously uploaded and the values you reconciled, allowing you to quickly and accurately validate new model versions.

For even faster iteration, group related test cases into test suites. Test suites let you focus on specific document types or fields, enabling targeted refinement before applying changes to your full model.

View AI-Suggested Improvements

AI-Suggested Improvements uses AI to analyze your extraction model's performance and generate recommendations for improving accuracy. If you enabled AI-Suggested Improvements in the model's additional options, the suggested improvements appear on the Testing tab and when you update a model. Developers can view detailed suggestions about how to update a model's configurations to improve extraction accuracy.

When you've implemented the suggested improvements, click the checkmark to mark the item as done. Click MARK AS DONE at the top of the list to mark all items as done. Marking an item as done removes it from the list, so you and other developers won't see it anymore.

Create test cases

You can set an extraction instance as a test case in two ways: during initial extraction or from a previously run instance.

During initial extraction

  1. Go to the Models tab and choose a model.
  2. Click the options icon and select Run Extraction.
  3. In the extraction run dialog, turn on the Set as Test Case toggle.

Using this method, the reconciled values from your test input become the baseline for future instance retests.

From a previously run instance

  1. Navigate to the Instance tab.
  2. Choose a previously run instance that represents your complete, reconciled test set.
  3. Click the options icon and select Set as Test Case.

Run test cases

Run test cases to evaluate subsequent model versions against your saved baseline. You can run all test cases or select a specific test suite to focus on targeted improvements.

Run all test cases

  1. Update the prompt with your refinements to create a new model version.
  2. Click RUN ALL TESTS.
  3. Select the Model Version you want to test against.
  4. Click RUN to start the tests.

Create a test suite

Test suites are collections of test cases that allow you to focus on specific improvements. Instead of running all test cases every time, define a test suite to quickly iterate on certain document types or fields.

To create a test suite:

  1. Navigate to the extraction Models tab and choose a model where you want to create a test suite.
  2. Go to the Testing tab.
  3. Click +NEW TEST SUITE (available on both the Test Cases tab and Test Suites tab).
  4. Use the checkboxes to select the test cases you want to include in this suite.
  5. Enter a Suite Name (for example, "High-value invoices" or "Complex forms").
  6. (Optional) Enter a Description to document the purpose of this test suite.
  7. Click SAVE.

Your test suite is now ready to use. You can run it at any time to validate model improvements against this specific subset of documents.

Add test cases to a test suite

You can add test cases to a test suite during extraction or from the test suite management interface.

During extraction:

While running an extraction, turn on the Set as Test Case toggle. The Add to Test Suite dropdown appears.

  1. Select an existing test suite from the Add to Test Suite dropdown to add this test case to.
  2. Click ADD.
  3. After extraction completes, reconcile the fields to validate the extracted data.

From the test suite interface:

  1. On the Test Suites tab, select the test suite you want to update.
  2. Click on the options icon and select Update Test Suite.
  3. Select or search the available test cases you want to add.
  4. Click SAVE.

View and delete test suites

  1. From the Testing tab and select Test Suites to see all test suites for this model.
  2. Select a test suite from the list.
  3. Click the options icon next to Run Test Suite.
  4. Select Remove test suite and click DELETE.

Note:  Deleting a test suite does not delete the individual test cases. The test cases remain available and can be added to other test suites or run independently.

Run a test suite

Test suites allow you to group specific test cases and run them against different prompt versions. This enables faster, more targeted testing and refinement.

  1. In the Test Suites tab, locate the test suite you want to run and click RUN TEST SUITE.
  2. Select the Model Version you want to test against.
  3. Click RUN.
  4. Review the accuracy metrics for the test suite run in the Test Runs tab.

View test suite runs

After running a test suite, you can view the results and accuracy metrics to understand how your model performed against that specific subset of documents.

Access test suite runs

Choose an extraction model and go to the Testing tab. Next, select Test Runs to see all test suite executions.

Review test suite run details

  1. Click on a test run to view:
    • Test Suite Name: The name of the test suite that was run.
    • Prompt Version: The model version used for this run.
    • Run Date: When the test suite was executed.
    • Instances: Individual extraction results for each document in the test suite. Each instance shows:
      • Instance: The name or identifier of the extracted document.
      • Status: The current status of the extraction.
      • Actions: Number of AI actions consumed for this instance.
      • Accuracy: Percentage of field and table accuracy.
      • Last Reconciled: When the instance was last reconciled.
  2. Click on an individual instance to view detailed extraction results and reconciliation data.

Processing versions

Each model version carries a processing version that determines how DocCenter processes documents internally. Unlike a model version, which captures your own configuration, a processing version reflects changes DocCenter makes to how it processes documents (for example, how it routes DOCX files or the prompt schema). Your fields, prompts, and LLM selection are not affected by processing version changes.

Upgrade a processing version

When a newer processing version is available, an upgrade banner appears on the model version Summary tab.

To upgrade a processing version, click UPGRADE MODEL. DocCenter creates a new model version with your existing fields, prompts, and settings preserved.

After upgrading, run your test suites to validate the new version before publishing. If the results are not acceptable, restore the previous model version.

Create a new model version

  1. Click on Extraction page group and select Models.
  2. Find the model you want to use as the basis for your new version. Then:
    • Click the overflow menu in the model's row, and click New Version.

    OR

    • Click the model name to open its summary. In the menu of options at the top of the page, click New Version.
  3. On the Create Extraction Model Version page, edit the following details:
    • Version Description: Provide information about the updates you're making in this version.
    • Fields
    • Sections
    • Prompt
    • Examples

    Tip:  See the previous section for a refresher on how to configure the model details.

  4. (Optional) Validate your changes by running test cases or a test suite:
    • Review the accuracy metrics to ensure your changes improved performance. Update the prompt with your refinements to create a new model version.
    • If satisfied, proceed to save. If not, return to step 3 to make additional refinements.
  5. Click SAVE to save this version for future updates, or SAVE AND PUBLISH to save your changes and use this version now.

All new extraction processes are now run and recorded against the latest published model version.

Restore a previous model version

If you want to use a previous version of an extraction model, you can restore it. This process creates a new version of the model, using a previous version's configurations.

  1. Click on Extraction page group and select Models.
  2. Click the name of a model in the list.
  3. Go to the Versions tab.
  4. Find the version you want to restore, and open its overflow menu.
  5. Click Restore Version. The Create Extraction Model Version page displays, with the configurations from that model version prepopulated.
  6. Update the model configurations if desired, and click SAVE or SAVE AND PUBLISH.

In the list of versions, a new model version appears. The new model version's description refers to the previous model version you used to create it.

Update model details

To update an extraction model's name, description, layout orientation, or additional options after creation:

  1. In the Extraction page group, click Models.
  2. Click the name of the model you want to update.
  3. Click the overflow menu and select Update Details.
  4. Edit the model name, description, or reconciliation layout orientation as needed.
  5. Expand Additional Options to enable or disable the following settings:

    Field Description
    Bounding Boxes Displays bounding boxes in the reconciliation task.
    Auto Document Rotation Automatically rotates the document to its proper orientation to help improve extraction accuracy.
    Log AI Usage View detailed information on how this model uses AI-powered features. Must be enabled to generate suggested improvements, but may slow down overall processing times.
    AI-Suggested Improvements Get AI-generated suggestions on how to improve model accuracy. This will increase your AI action consumption.
    Advanced Layout Improves extraction accuracy for longer documents, tables, checkboxes, or signatures. Significantly increases AI action consumption. Before enabling, try using Visual extraction and other prompt refinement techniques. Usage limits apply.
  6. Click UPDATE.

Delete an extraction model

Deleting an extraction model permanently removes all model data, including versions, instances, and test cases.

To delete an extraction model:

  1. In the Extraction page group, click Models.
  2. Click the name of the model you want to delete.
  3. Click the overflow menu and select Delete Model.
  4. Click DELETE to confirm.

Caution:  This action cannot be undone. Model data is permanently deleted.

Extraction rules and methods

Rules

Rules help provide the extraction model with additional guardrails that ensure field data is extracted in the way you expect. When you create an extraction model, you can add three types of rules:

  • Options: This rule provides the model with a list of valid options, which also appear as a dropdown list in the reconciliation interface. For example, if you're extracting data from a field for a US state, you can create an options rule with the 50 states to ensure the model chooses from one of those.
  • Field Validations: This rule validates that the extracted data fits certain logical criteria. If the data fails validation, the message appears on the reconciliation interface. For example, a validation rule can flag a date field value that's in the future.
  • Post-Process: This rule transforms or changes extracted data before displaying that information on the reconcile screen. For example, a post-process rule can be used to reformat a phone number.

Options rule

Options rules provide the extraction model with valid responses it should be extracting.

To create an options rule:

  1. Create a new expression rule.
  2. In the expression editor, create an array and list the valid options. For example, values can be hard-coded in a list or calculated by querying a record type.

To see an example options rule, view AIA_Example_OptionsRule which returns three options:

1
2
3
4
5
{
  "Option 1",
  "Option 2",
  "Option 3"
}

Field Validation

Field validation rules provide additional validation on the reconciliation form. For example, if a field is supposed to contain a date in the past, you can create a validation rule to catch any dates that don't adhere. The validation message appears on the reconciliation interface, prompting the user to correct the value.

To create a field validation rule:

  1. Create a new expression rule.
  2. Create a rule input named value and type Any Type. The field value is passed into the validation rule using this rule input.
  3. In the expression editor, create your validation rule. It should use the input value and return a string error if invalid.

To see an example validation rule, view AIA_Example_ValidationRule which returns a validation error if the input is not in the correct format:

1
2
3
4
5
6
7
8
9
if(
  a!isNullOrEmpty(ri!value),
  null,
  if(
    regexmatch("^\(\d{3}\) \d{3}-\d{4}$", ri!value),
    null,
    "Phone number must be in the format of (XXX) XXX-XXXX"
  )
)

Post-process rule

Post-process rules reformat the extracted data before proceeding to validation and reconciliation. These rules can remove invalid characters or reformat dates.

To create a post-process rule:

  1. Create a new expression rule.
  2. Create a rule input named input and type Map.
  3. In the expression editor, create your post-processing rule. It should index values out of the map, use them to post-process the extracted field, and return the newly formatted string. The available parameters are:
    • Map: map containing the key-value pairs of the extracted data.
    • ExtractionInstance: The Extraction Instance record for this extraction, used to get document or model version details.

To see an example post-process rule, view AIA_Example_PostProcessRule which returns the value stripped of dashes:

1
2
3
4
5
6
7
8
if(
  or(
    a!isNullOrEmpty(ri!input),
    a!isNullOrEmpty(ri!input.value)
  ),
  null,
  substitute(ri!input.value, "-", "")
)

Methods

Methods instruct the model on how to extract data from the content. You'll choose an extraction method for each field. Methods can also help extract information that doesn't appear in the content, but is calculated using extracted information.

The Instructions field takes different inputs based on the method you choose:

Field Definition Input
Generative AI (Default) Extract field from document using generative AI. Text - A natural language description of the field and the contents you expect it to contain. For example, you could ask for ZIP codes to only extract 5 digits instead of 9.

The more detail you provide, the better outcomes you'll see. This prompt is the primary way to improve the accuracy of the model.
Rule Calculate the field using an expression rule. Expression rule. The inputs to this expression rule are:
  • ExtractionInstance
  • Other extracted fields (excluding calculated fields)
Calculated Calculate the field using another extraction field + generative AI. Text - A natural language description of a field that can be calculated using the other extracted fields.

Examples:

  • Extract the postal code from the address block
  • Count the number of line items from the invoice
Analyze Expense Extract the field using data often found in expense-related documents. Usage limits apply. Field selection
Regex Extract the field using a regular expression. Text - A regular expression with a single capturing group.

Examples:

  • Phone number: ((\d{3}) \d{3}-\d{4})
  • Email: ([\w.-]+@[\w-]+.[a-zA-Z]{2,})
  • Social Security Number: (\d{3}-\d{2}-\d{4})
Search Query Extract additional data by asking questions or looking for specific terms or phrases in the document. Text - Natural language questions about the document, or terms or phrases to find.

Generative AI method

The Generative AI method is the default way to extract data. DocCenter extracts data from input documents using the field's data type, extraction instructions, and options as hints.

Use the Generative AI method for most extraction fields. The natural language instructions you provide in the Instructions field are the primary way to improve extraction accuracy.

Rule-based method

The Rule-based method programmatically extracts fields without using AI. Use rule-based extraction to return metadata about the document, such as page count, or to calculate values from document properties.

To create an extraction rule:

  1. Create a new expression rule.
  2. Create a rule input named input and type Map.
  3. In the expression editor, create your rule. It should index values out of the map and use them to calculate the extraction field. The available parameters are:
    • ExtractionInstance: The Extraction Instance record for this extraction.

To see an example extraction rule, view AIA_Example_ExtractionRule which returns whether the input document is over 20 pages:

1
2
3
4
5
6
7
8
9
10
11
12
/* This rule returns true if the document has more than 20 pages */
if(
  a!isNullOrEmpty(ri!input),
  null,
  if(
    getpdfmetadata(
      ri!input.ExtractionInstance['recordType!AIA Extraction Instance.documentId']
    ).pageCount > 20,
    true,
    false
  )
)

Calculated method

Note:  Calculated fields only use other extracted data as inputs. Ensure that you set up fields to extract the data you are using in your calculation.

The Calculated method uses the results from all other extraction methods as input and applies Generative AI to calculate a derived field. Use the Calculated method to combine extracted fields or further parse extracted values.

For example, if extracting dates from a resume, a calculated field can add all time periods together to determine the total amount of work experience.

Analyze Expense method

Usage limits apply.

The Analyze Expense method extracts financial information, such as costs and line items, from documents such as invoices. Use the Analyze Expense method for standard expense-related documents.

When using the Analyze Expense method, select a single field type. The model extracts the field value with the highest confidence value for that field type.

Regex method

The Regex method extracts text programmatically from the input document text layer using a regular expression. The regex pattern must have a single capturing group. Use the Regex method when the field value follows a predictable pattern, such as phone numbers, email addresses, or ID numbers.

For example, to extract an email address, use ([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}).

Type your regular expression as plain text in the Extraction Prompt field when you choose Regex as the extraction method.

Search Query method

The Search Query method allows you to ask questions about the document contents, or look for specific terms or phrases. Use the Search Query method when you need to find specific information that may not have a consistent label or location in the document. The Search Query method leverages the Advanced IDP Tools AI skill design object.

Extraction model sections

As you create an extraction model, you can configure sections to divide documents into targeted regions. Sections are helpful for two reasons:

  • Sections break larger documents into smaller chunks. You assign fields to those sections, which helps the AI model more easily locate and extract data. For larger documents (> 20 pages), it can be helpful to break the document into smaller chunks using various sections with rules.
  • Sections organize data on the reconciliation screen. This can help users find and confirm that data was extracted correctly, or help them update fields if needed. To use sections solely to organize data on the reconciliation screen, simply enter the section name into the picker in the extraction model version update form.

Available section types:

  • General: Creates no section for the document.
  • Page Range: Split a document into a section using a fixed page range.
    • Start Page: Page number to start the section.
    • End Page: Page number to end the section.
  • End of Document: Create a section starting at the end of the document and going back a certain number of pages. For example, if you want to create a section for an index that appears at the end of a document, count the number of pages in the index and enter it here. If the index is 5 pages, the AI model considers the last 5 pages of each document to be the index.
    • Number of Pages: Number of pages before the end of the document to start the section.
  • Table of Contents: Create a section based on the table of contents in the document. The table of contents is assumed to be within the first 10 pages of the document. Generative AI determines the start and end page for a particular section title you input. Additional processing occurs to determine if there is any offset between the page numbers in the table of contents and the actual page numbers of the document.
    • Section Name: Input the name of the section as it appears in the table of contents.
  • Find Text on Page: Create a section using the pages where particular text is first found. Ensure that the section start page is before the section end page when you enter the start and end text.
    • Start Text: The text that first appears on a page where the section starts.
    • End Text: The text that first appears on a page where the section ends.

Deploy the extraction model

Once your extraction model meets your accuracy benchmarks, add the extraction model to your business process.

Straight through processing

Straight through processing (STP) automates the validation of extracted data so that only flagged documents require manual review. The Straight Through Processing tab provides multiple automated checks that you can configure in combination:

  • Confidence threshold: Call attention to extracted fields that the model isn't confident it extracted correctly.
  • Sampling: Review a small portion of extracted documents to verify data is extracted correctly.
  • Validations: Use conditions to verify anything about the extracted content.
  • AI Review: use AI to further validate that your model's extracted values are accurate.

Tip:  If any model instance is flagged by the options that you have set up on this tab, that instance will appear as Extracted with Errors, encouraging manual review.

Looking for a way to improve accuracy over time? Enable AI-Suggested Improvements for tips on how to adjust your model's configuration for higher extraction accuracy.

Confidence threshold

Note:  Confidence scores aren't available for large tables.

Models provide confidence scores to indicate how sure they are of their predictions, including extracted values. Confidence thresholds define the lower limit of what you are comfortable automating: if a model's confidence score for a field falls below the threshold you define, that field is flagged for manual review.

Human reviewers can focus their time on fields and documents where AI is less sure, while automatically processed documents meet your defined standards. The confidence score can also be used in downstream workflows to further automate your document processes.

Note:  To use confidence scores on an extraction model, the model must have Bounding Boxes or Advanced Layout enabled in the Model Details.

To enable field flagging based on confidence thresholds:

  1. Go to the Straight Through Processing tab.
  2. In the Settings section, click next to Confidence Threshold.
  3. Enable Use Confidence to Flag Fields.
  4. Set the Threshold value to your desired level.

    Note:  The confidence threshold can only be set in increments of 10, from 10 to 90.

  5. Click SAVE.

When confidence thresholds are enabled, DocCenter flags extracted fields that fall below the threshold. You'll see these flags next to the extracted fields in the reconciliation task.

See Confidence score for details on how confidence scores are measured and how to view them.

Sampling rate

Sampling randomly selects a portion of extracted documents for manual reconciliation. Use sampling to verify extraction accuracy on high-volume document processing without reviewing every document. DocCenter randomly chooses documents for manual review based on the sampling rate you configure.

Tip:  Sampled documents won't include other documents flagged due to confidence thresholds, validations, or AI review.

To enable sampling:

  1. Go to the Straight Through Processing tab.
  2. In the Settings section, click next to Sampling Rate.
  3. Turn on Enable Sampling to Verify Accuracy.
  4. Adjust the Sampling Rate value based on your preference. You can update this value as needed. We recommend starting at a higher sampling rate for new models or if you're extracting data from important documents.
  5. Click SAVE.

Validations

Validation rules verify conditions in the extracted document. Validations range from verifying that fields are not blank to calling an integration to verify an address exists.

You can validate data for individual fields or entire document:

  • Field validations only validate the data within a single field and have no context about the rest of the extracted data. You configure field validations when you create the model.
  • Model validations can validate across fields in a document. For example, a validation can ensure that the total of each of my line items adds up to the invoice total.

Validations are configured at a model version level and can be accessed through the Validation Rules tab on a Model Version dashboard.

Create a model validation

To create a new validation do the following:

  1. In DocCenter, navigate to the model where you want to add a validation.
  2. Click the Straight Through Processing tab.
  3. Click the CREATE MODEL VALIDATION RULE button.
  4. Populate the following fields:
    • Name: This name appears on extraction instances when a validation error is triggered.
    • Priority: Relative importance of the validation rule, which displays if the validation error is triggered.
    • Build Your Rule:
      • Describe Validation: Describe the validation in your own words, and AI will generate conditions for you.
      • Expression Rule: Select the Appian expression rule containing your validation. This rule should:
        • take a single parameter called parameters of type Map, containing the instance, instanceMap, and modelVersionFields.
        • return a Map with the following fields:
          • result: Whether the rule passed or failed.
          • message: Error message to display if failed.
  5. Test the rule by providing sample data, or use AI Copilot to generate sample test data for you.
  6. Click RUN TEST.
  7. Confirm that the output in the Result field makes sense.
  8. Click CREATE.

DocCenter includes the following example expression rules:

  • AIA_RULE_ExampleValidationFirstLastName - validates that a first and last name are not both blank.
  • AIA_RULE_ExampleValidationRuleSalesOrderLines - validates that the number of sales order line items is > 0.

AI Review

Note:  AI Review isn't available for large tables.

Note:  AI Review does not support looping. When a document is processed with looping, AI Review evaluates only the first 20 or 100 pages depending on the selected LLM model, not the full document.

AI Review uses a second AI model to verify that your extraction model's extracted values are accurate. After the extraction model extracts data from a document, the AI Review model checks those extractions. If AI Review detects that an extracted field may be incorrect, AI Review provides suggestions on what the correct value may be.

Tip:  AI Review is enabled per model. When enabled, AI Review applies to all instances for that model, increasing your AI Action usage.

Set up AI review

To enable AI review for your model:

  1. Go to the Straight Through Processing tab.
  2. In the AI Review section, click SET UP AI REVIEW.
  3. Select the fields and tables you want AI to review.
    • All fields are selected by default, except those you've hidden from reconciliation or excluded from metrics.
    • All tables are selected by default.
  4. Review the AI Review Prompt:
    • Select the Large Language Model you'd like to use. The dropdown displays all preconfigured provider and model combinations available in your environment, allowing you to choose a different provider for AI review than the one used for extraction. To add more providers, see AI Services.
    • Review the Role field to add or change details about the role the AI model should assume when reviewing.
    • Go to the Additional Instructions field to add specific instructions for the AI model to use when reviewing extracted values.
  5. Click SAVE.

Scroll or drag to pan · ⌘+/⌘− to zoom · Esc to closesetup AI review

When AI review is enabled, the review automatically occurs on all instances that use this extraction model. Users see AI's suggestions as they complete the reconciliation task.

To disable AI review for a model:

  1. Go to the Straight Through Processing tab.
  2. Click .
  3. Click Clear All to deselect all fields and tables.
  4. Click SAVE.

Add triggers to an extraction model

Triggers initiate process models when extraction activities complete. For example, after the model extracts data from invoices, a trigger can start the process that saves the data to your record types and compiles a report for the finance department.

Note:  You can only create one trigger per activity type in a given model. Reconciliation is the only available activity type at this time.

Keep in mind that triggered processes need to meet certain requirements.

To create a trigger:

  1. Go to the extraction model where you want to add a trigger.
  2. Open the Triggers tab.
  3. Click CREATE TRIGGER.
  4. Enter the following configurations:
    1. When this activity completes: Select the activity you want to base the trigger on. Reconciliation is the only available activity type.
    2. Run this process model: Choose the process model to start when the activity is complete.
    3. Trigger this process to run: Synchronously or Asynchronously.
  5. Click ADD TRIGGER.

Every time the extraction model runs, the chosen process model automatically starts when the chosen activity completes.

Scroll or drag to pan · ⌘+/⌘− to zoom · Esc to closeimages/create-trigger-extraction.png

To edit a trigger:

  1. Go to the extraction model where you want to edit a trigger.
  2. Open the Triggers tab.
  3. Click Update Trigger .
  4. Change the configurations as desired.
  5. Click UPDATE TRIGGER.

To delete a trigger, open the Triggers tab and click Delete Trigger in the list.

Requirements for triggered processes

For triggers to work effectively and execute processes reliably, the process model must meet the following requirements:

  • The process model must have a single parameterized process variable named extractionInstance with AIA Extraction Instance as its type.
  • The process model object must have View permissions for the AIA Operations group.
  • The process model must have flows that account for re-runs or updated runs.

You can only create one trigger per AI model. If you need to trigger more than one process model, create a wrapper process model containing all needed processes, and call that wrapper process model here.

Feedback