Create an Advanced IDP Tools AI Skill

Overview

This topic covers how to create an AI skill to process complex documents.

An Advanced IDP Tools AI skill takes documents as inputs via the Advanced IDP Tools smart service, analyzes the content, and returns a map of comprehensive data according to the tools selected.

Here's a high-level breakdown of how to create and use an Advanced IDP Tools AI skill:

1. Create an AI skill
2. Choose tool types
3. Test selected tools
4. Use the skill!

See below to learn more about when to use generative AI or Advanced IDP Tools for document extraction.

Create an AI skill

1. In the Build view, click NEW > AI Skill.

   

2. In the Extraction section, select Documents > Advanced IDP Tools. The configuration screen displays.

   

3. Configure the following properties:

   Property	Description
   Name	Enter a name that follows the recommended naming standard.
   Description	(Optional) Enter a brief description of the AI skill.

4. Click CREATE.
5. On the Review AI Skill Security window, configure security.
6. Click SAVE. The AI skill opens in a new dialog or window.

Choose tool types

Select as many tools as you need to extract data.

To see what data each tool returns, see output examples.

Create search queries for testing

When you select Search Queries as a tool, you can create search queries within the AI skill for testing purposes. A search query can be phrased as a question, or you can include a specific term or phrase you'd like the AI skill to find within the document.

To add a search query for testing:

1. Select the Search Queries tool within the AI skill object.
2. In the Test Selected Tools pane, under Search Queries, type a word, phrase, or question.
3. Click ADD SEARCH QUERY. When the query is added, a key is automatically created. The key is used in the map of data returned as a response.
4. Repeat the process for each search query you want to add. You can add up to 15 queries for each document.

You can also edit or delete existing search queries.

Scroll or drag to pan · ⌘+/⌘− to zoom · Esc to close

Test selected tools

Before you integrate the AI skill into a process, you want to be confident the model extracts data properly and returns the results you expect. You can test the AI skill using sample documents to verify it works as you expect directly within the AI skill object.

To test the skill:

1. Within the AI skill, locate the Test Selected Tools area.
2. Upload a document into the Document field.
3. Click GENERATE OUTPUT. The AI model returns a response based on the tools you selected and test input.

Output will vary based on the type of tool you selected. As you test, you can select or deselect tools to get results that you find most useful.

Test results display an AI Actions Consumed indicator showing the number of AI Actions used by that test execution. Use this indicator to estimate costs during design before deploying the skill in a process model.

Output examples

Text output

Returns a map of text values and types (such as line, word, or both), the position of where the text was found in the document, confidence scores, and page locations.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
a!map(
  data:
    {   
      text: "People",
      type: "line",
        a!map(
          width: 0.1607328,
          height: 0.097553568,
          left: 0.2871447,
          top: 0.387261
        ),
      confidenceScore: 25,
      pageNumber: 1
    }
)

Expense Data output

Returns a map of data containing expense information such as line item types and values, as well as document reference numbers such as invoice numbers. Contains confidence scores and page locations.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
a!map(
  flatTextFields: {
    a!map(
      normalizedKey: "PO_NUMBER",
      documentKey: "PO #",
      value: "12345",
      keyConfidenceScore: 99,
      valueConfidenceScore: 98,
      pageNumber: 1
    ),
    a!map(
      /*<Other fields here>*/

    )/*<Maps of all identified fields>*/

  },
  lineItemGroups: {
    a!map(
      lineItems: {
        a!map(
          lineItemExpenseFields: {
            a!map(
              type: a!map(text: "ITEM", confidence: 70, ),
              labelDetection: a!map(text: "Usage Charges", confidence: 79),
              valueDetection: a!map(text: "AZ Prem Files", confidence: 98)
            ),
            a!map(
              type: a!map(text: "UNIT_PRICE", confidence: 70),
              labelDetection: a!map(text: "Unit Price", confidence: 80),
              valueDetection: a!map(text: "13.12", confidence: 99)
            )
          },
          pageNumber: 1
        )
      },

    )},
    a!map(
      lineItems: {
        a!map(lineItemExpenseFields: {}, pageNumber: 2)
      }
    )
)

Key-Value Pairs output

Returns a map of data containing matched labels and values, confidence scores, and page location.

1
2
3
4
5
6
7
8
a!map(
  key: "First Name",
  value: "John",
  keyConfidenceScore: 97,
  valueConfidenceScore: 99,
  pageNumber: 1
  ),
  /*<All other KVPs found in document>*/

Checkboxes output

Returns a map of data containing the checkbox label, whether the box was checked, confidence scores, and page location.

1
2
3
4
5
6
7
8
a!map(
  key: "Other (Specify)",
  value: true,
  keyConfidenceScore: 97,
  valueConfidenceScore: 99,
  pageNumber: 1
  ),
  /*<All other checkboxes found in document>*/

Tables output

Returns a map of data containing table data and information on table structure.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
a!map(
  data: {
    a!map(
      c3: "DATA Requested by client"),
      a!map(c1: "2015", c3: "2016"),
      a!map(
        c1: a!map(value: "the value", confidence: 95),
        c2: a!map(value: "the value", confidence: 95),
        c3: a!map(value: "the value", confidence: 95),
        c4: a!map(value: "the value", confidence: 95)
        ),
      a!map(
        c1: a!map(value: "the value", confidence: 95),
        c2: a!map(value: "the value", confidence: 95),
        c3: a!map(value: "the value", confidence: 95),
        c4: a!map(value: "the value", confidence: 95)
        )
    },
  pageNumber: 1,
  tableTitleRows: { 1 },
  tableHeaderRows: { 2, 3 },
  tableSectionTitleRows: { 15 },
  tableSummaryRows: { 7 },
  tableFooterRows: {}
  tableType: "SEMI_STRUCTURED_TABLE" // or 'STRUCTURED_TABLE'``
),
/*<list of Maps for all other tables found in document>*/

Signatures output

Returns a map of data on the position of signatures detected in the document.

1
2
3
4
5
6
7
8
9
10
11
12
13
a!map(
  data:
    {
      a!map(
        width: 0.1607328,
        height: 0.097553568,
        left: 0.2871447,
        top: 0.387261
        ),
      confidenceScore: 25,
      pageNumber: 1
    }
)

Search Queries output

Returns a map of data containing the original query and maps of answers. Each answer contains the value, the position of relevant content detected in the document, confidence scores, and page location.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
a!map(
  searchQuery: "Who is the manufacturer of the 2nd dose?",
  key: "whoIsTheManufacturerOfThe2ndDose",
  values: {
    a!map(
      value: "ACME Pharma",
     	valueBoundingBox: a!map(
       	top: 0.10982168,
        left: 0.075827464,
        width: 0.6908191,
        height: 0.340923
        ),
     	valueConfidenceScore: 99,
     	pageNumber: 1
    ),
  /*<All responses for the same query>*/
  }
),
a!map(
  searchQuery: "What is the lot number of the 1st dose?",
  key: "whatIsTheLotNumberOfThe1stDose",
  values: {
    a!map(
      value: "AA1234",
     	valueBoundingBox: a!map(
       	top: 0.90,
        left: 0.19,
        width: 0.29,
        height: 0.64
      ),
     	valueConfidenceScore: 99,
     	pageNumber: 1
    ),
  /*<All responses for all the queries found in the document>*/
  })
a!map(
  searchQuery: "What is patient's first name?",
  key: "whatIsThePatientsFirstName",
  values: {
    a!map(
      value: "John",
     	valueBoundingBox: a!map(
       	top: 0.90,
        left: 0.19,
        width: 0.29,
        height: 0.64
      ),
     	valueConfidenceScore: 99,
     	pageNumber: 1
    ),
  /*<All responses for all the queries found in the document>*/
  })  

Use the skill

Once the test results return output you're looking for, you're ready to use advanced IDP tools in a process.

Add the Advanced IDP Tools smart service and configure it to call your new skill.

Review the feature's compliance to ensure it aligns with your organization's security requirements.

Usage considerations

Keep in mind the following requirements when using the Advanced IDP Tools AI skill:

When using the Search Queries tool, you can add up to 15 search queries for each document. Each search query can be up to 200 characters.

Refer to the table below to see the requirements for each input type.

Document extraction: Generative AI or Advanced IDP Tools?

When you extract data from documents, you can choose from different AI skills based on your document structure and extraction goals. Each AI skill is designed to handle specific scenarios and can be used individually or together for more accurate results.

This section explains the difference between Appian's two main document processing options and provides patterns to help you decide which is best for your use case.

• Advanced IDP Skill is a high-precision document recognition engine. It excels at identifying the text, layout, and visual elements in a document.
• Generative AI Skill is a reasoning engine. It excels at understanding the context of the text to answer questions and extract information.

When you create a document extraction model in DocCenter you can combine these AI skills by selecting the Advanced Layout option.

How Advanced IDP and Generative AI skills differ

Think of the Advanced IDP Tools skill as the tool that reads the document and the Generative AI skill as the tool that understands it.

When you send a document directly to a generative AI skill as a Text only document type, Appian performs a standard optical character recognition (OCR) step to read the text. For more complex documents, you can use the Advanced IDP skill first to perform a more detailed reading of the document's structure and content, then send that detailed map to the Generative AI skill for a more accurate analysis.

The table below summarizes the key differences.

Choosing the right pattern for document extraction

Your document's complexity and your business needs determine the best way to configure document extraction. This section describes three common patterns.

• Use only Generative AI skill
• Use only Advanced IDP skill
• Use Advanced IDP skill with Generative AI skill

Use only Generative AI skill

This approach is best for simple, text-heavy documents where the visual layout is not important. By passing the document directly to the Execute Generative AI Skill smart service, you can choose between two document processing modes:

• Text only: Use this mode when your document contains only text content. This mode uses built-in OCR to retrieve the text and supports up to 100 pages.
• Visual elements & text: Use this mode when your document contains visual elements like checkboxes, charts, images, or handwritten annotations that you need to extract. For Claude 4+ and Nova 2 Lite models, this mode also supports up to 100 pages.

Choose the mode based on your document's content type, not page count. If your document has visual elements you need to extract, use "Visual elements & text" mode. If it's text-only, use "Text only" mode for faster processing.

Use this pattern for documents like:

• Emails
• Transcripts
• Simple letters

Use only Advanced IDP Tools skill

For some use cases, you don't need the Generative AI skill's reasoning capabilities. The Advanced IDP Tools skill is effective on its own for tasks that depend on a document's structure. To use this pattern, create an Advanced IDP Tools AI Skill and call it using the Advanced IDP Tools smart service in a process model.

Use this pattern to:

• Index documents for search: Use the Text tool to extract all content from a set of documents and write it to a database field that is enabled for smart search.
• Verify signatures or checkboxes: Use the Signatures or Checkboxes tool to confirm if a document is signed or which options are selected.
• Extract from fixed forms: For highly structured documents that never change like certain government forms.
• Leverage Confidence Scores: Use the confidence scores returned by the AI skill to support automated decisions, such as straight-through processing, when the score exceeds a defined threshold.

Use Advanced IDP Tools skill with Generative AI skill

This is the most robust pattern, designed for complex or semi-structured documents where you need both high-precision OCR and intelligent reasoning. To use this pattern, select Advanced Layout when you configure your document extraction model in AI Document Center.

Use this pattern for documents like:

• Invoices with varying formats
• Complex contracts
• Utility bills

How it works

When you use the Advanced Layout option, the extraction process happens in two steps:

1. Advanced IDP Tools skill scans the document to create a detailed, structured map of all its content, including checkboxes, tables, and signatures. Appian then automatically formats this map into a clearly structured phrase that AI can understand more easily.
2. Generative AI skill receives the formatted text from the first step. Your prompt then guides the model to reason over this high-quality text to find and extract the data you need.

Highlighting extracted data

When you enable the Bounding option in a document extraction model, Appian uses the x- and y-coordinate data from the Advanced IDP Tools skill. This allows you to show users exactly where on the page each piece of data was found on the reconciliation task interface.

AI action usage consideration

This combined pattern provides the highest accuracy but also uses more AI actions than other patterns. Converting the detailed structural data from the Advanced IDP Tools skill into a formatted text string can result in a large input for the generative AI skill. However, the improved accuracy often outweighs the additional AI action cost.

Feedback