Roman Klimenko
DA

Searching for entities with the CluedIn Python SDK

cluedinpython

In the latest release of the CluedIn Python SDK, I improved its support for the GraphQL Search API. This is a good time to explain how to retrieve entities, or golden records, directly through GraphQL and through the SDK, which uses the same API.

We'll start with the GraphQL playground in the CluedIn interface and finish with a one-line Python query.

GraphQL Search API

GraphQL supports most interactions with CluedIn. Ingestion endpoints are one of the few exceptions. The official documentation has a useful introduction to the CluedIn GraphQL API.

Open the Consume section in your CluedIn instance to find a playground where you can run GraphQL queries.

My instance contains some /Duck entities from DuckTales. To find them, I can run a query like this:

{
  search(query:"+entityType:/Duck")
  {
    entries {
      id
      name
      entityType
    }
  }
}

The query returns the first 20 /Duck entities. The query parameter filters the response by entity type, while the entries block selects the properties to include in the response: id, name, and entityType.

Next, change the query to accept GraphQL variables:

query ($query: String, $pageSize: Int) {
  search(
    query: $query
    pageSize: $pageSize
    sort: FIELDS
    sortFields: {field: "id", direction: ASCENDING}
  ) {
    cursor
    entries {
      id
      name
      entityType
    }
  }
}

Variables:

{
  "query": "+entityType:/Duck",
  "pageSize": 10000
}

Several parts of this query are worth noting:

  • query ($query: String, $pageSize: Int) defines the parameters. You can also name the query: query searchEntities($query: String, $pageSize: Int).
  • sort: FIELDS sortFields: {field: "id", direction: ASCENDING} sorts by a unique field, which keeps pagination predictable.
  • cursor asks CluedIn for the value needed to fetch the next page.
  • "pageSize": 10000 raises the default page size from 20 to the maximum of 10,000. Use a smaller value when you only need a few entities, as the query will finish faster.
GraphQL

CluedIn Python SDK

Any programming language can send a GraphQL request to CluedIn. Here is how to do it in Python.

Install the latest version of the CluedIn Python SDK:

%pip install cluedin

Import it together with Pandas, which we'll use to load the results into DataFrames:

import pandas as pd
import cluedin

You also need an API token. Copy or create one under Administration > API Tokens in CluedIn.

My CluedIn instance is installed at https://foobar.klimenko.dk/. To create a context for it, I provide the organization name (foobar), domain (klimenko.dk), and the access token copied from the CluedIn interface:

ctx = cluedin.Context.from_dict({
    'domain': 'klimenko.dk',
    'org_name': 'foobar',
    'access_token': '{paste_your_token_here}'
})

We can now run the earlier GraphQL query from Python:

query = """
query searchEntities($query: String, $pageSize: Int) {
  search(
    query: $query
    pageSize: $pageSize
    sort: FIELDS,
    sortFields: {field: "id", direction: ASCENDING}
  ) {
    cursor
    entries {
      id
      name
      entityType
    }
  }
}
"""

variables = {
    'query': '+entityType:/Duck',
    'pageSize': 3
}

cluedin.gql.gql(ctx, query=query, variables=variables)

The result contains the first three entities because the example uses a page size of three. It also includes the cursor needed to fetch the next page:

{'data': {'search': {'cursor': 'ewAiAFAAYQBnAGUAIgA6ADEALAAiAFAAYQBnAGUAUwBpAHoAZQAiADoAMwAsACIAQwBvAG0AcABvAHMAaQB0AGUAQQBmAHQAZQByACIAOgB7AH0ALAAiAFMAZQBhAHIAYwBoAEEAZgB0AGUAcgAiADoAWwAiADYAMwA1ADMAOAAzAGEAOQAtADkAYwA3ADUALQA1AGQANgAxAC0AOABmADIAYgAtAGYAZQA0ADkANgBmAGQAOAAyAGIAZQA3ACIALAAiADYAMwA1ADMAOAAzAGEAOQAtADkAYwA3ADUALQA1AGQANgAxAC0AOABmADIAYgAtAGYAZQA0ADkANgBmAGQAOAAyAGIAZQA3ACIAXQB9AA==',
   'entries': [{'id': '145afb55-4e78-5dad-b208-633b5b6d19cf',
     'name': 'Donald Duck',
     'entityType': '/Duck'},
    {'id': '17bad60e-6782-5ae5-84bf-7efe05e78e58',
     'name': 'Jake McDuck',
     'entityType': '/Duck'},
    {'id': '635383a9-9c75-5d61-8f2b-fe496fd82be7',
     'name': 'Dewey Duck',
     'entityType': '/Duck'}]}}}

Pass that cursor as a parameter in the next request:

query = """
query searchEntities($cursor: PagingCursor, $query: String, $pageSize: Int) {
  search(
    query: $query
    cursor: $cursor
    pageSize: $pageSize
    sort: FIELDS,
    sortFields: {field: "id", direction: ASCENDING}
  ) {
    cursor
    entries {
      id
      name
      entityType
    }
  }
}
"""

variables = {
    'query': '+entityType:/Duck',
    'pageSize': 3
    'cursor': 'ewAiAFAAYQBnAGUAIgA6ADEALAAiAFAAYQBnAGUAUwBpAHoAZQAiADoAMwAsACIAQwBvAG0AcABvAHMAaQB0AGUAQQBmAHQAZQByACIAOgB7AH0ALAAiAFMAZQBhAHIAYwBoAEEAZgB0AGUAcgAiADoAWwAiADYAMwA1ADMAOAAzAGEAOQAtADkAYwA3ADUALQA1AGQANgAxAC0AOABmADIAYgAtAGYAZQA0ADkANgBmAGQAOAAyAGIAZQA3ACIALAAiADYAMwA1ADMAOAAzAGEAOQAtADkAYwA3ADUALQA1AGQANgAxAC0AOABmADIAYgAtAGYAZQA0ADkANgBmAGQAOAAyAGIAZQA3ACIAXQB9AA=='
}

cluedin.gql.gql(ctx, query=query, variables=variables)

The result is the next three entities:

{'data': {'search': {'cursor': 'ewAiAFAAYQBnAGUAIgA6ADIALAAiAFAAYQBnAGUAUwBpAHoAZQAiADoAMwAsACIAQwBvAG0AcABvAHMAaQB0AGUAQQBmAHQAZQByACIAOgB7AH0ALAAiAFMAZQBhAHIAYwBoAEEAZgB0AGUAcgAiADoAWwAiADkAMwBiADkAMgA4ADMANQAtADkANgBmADIALQA1ADYAYQA5AC0AOQA4AGMAMAAtAGMAOAA0ADgAMgAzADYANQAyADEAYQA5ACIALAAiADkAMwBiADkAMgA4ADMANQAtADkANgBmADIALQA1ADYAYQA5AC0AOQA4AGMAMAAtAGMAOAA0ADgAMgAzADYANQAyADEAYQA5ACIAXQB9AA==',
   'entries': [{'id': '6ae43a44-81b4-5fd7-9c7b-47cb24d407ea',
     'name': 'Angus McDuck',
     'entityType': '/Duck'},
    {'id': '9353b703-13d8-59a1-886c-f40b95283c06',
     'name': 'Hortense McDuck',
     'entityType': '/Duck'},
    {'id': '93b92835-96f2-56a9-98c0-c848236521a9',
     'name': 'Matilda McDuck',
     'entityType': '/Duck'}]}}}

You get the idea.

To avoid passing each cursor manually, use cluedin.gql.entries. It returns a generator that you can iterate directly or convert to a list:

...

# this is where you need a smaller page size
# if you don't want to iterate to the end
variables = {
    'query': '+entityType:/Duck',
    'pageSize': 2
}

generator = cluedin.gql.entries(ctx, query=query, variables=variables)
print(next(generator))
print(next(generator))

Result:

{'id': '145afb55-4e78-5dad-b208-633b5b6d19cf', 'name': 'Donald Duck', 'entityType': '/Duck'}
{'id': '17bad60e-6782-5ae5-84bf-7efe05e78e58', 'name': 'Jake McDuck', 'entityType': '/Duck'}

You can also load all entities into a DataFrame. In this case, use the maximum page size (10000) to reduce the number of server calls:

query = """
query searchEntities($cursor: PagingCursor, $query: String, $pageSize: Int) {
  search(
    query: $query
    cursor: $cursor
    pageSize: $pageSize
    sort: FIELDS,
    sortFields: {field: "id", direction: ASCENDING}
  ) {
    cursor
    entries {
      id
      name
      entityType
    }
  }
}
"""

variables = {
    'query': '+entityType:/Duck',
    'pageSize': 10_000
}

print(pd.DataFrame(cluedin.gql.entries(ctx, query=query, variables=variables)))

Result:

                                      id             name entityType
0   145afb55-4e78-5dad-b208-633b5b6d19cf      Donald Duck      /Duck
1   17bad60e-6782-5ae5-84bf-7efe05e78e58      Jake McDuck      /Duck
2   635383a9-9c75-5d61-8f2b-fe496fd82be7       Dewey Duck      /Duck
3   6ae43a44-81b4-5fd7-9c7b-47cb24d407ea     Angus McDuck      /Duck
4   9353b703-13d8-59a1-886c-f40b95283c06  Hortense McDuck      /Duck
5   93b92835-96f2-56a9-98c0-c848236521a9   Matilda McDuck      /Duck
6   a388a77d-7d43-51d1-87b2-efb4f854b5ad    Fergus McDuck      /Duck
7   b2fb05cb-e806-5088-955b-2ff3f9261236   Scrooge McDuck      /Duck
8   b8fc5baf-b679-5e26-abb5-50ca77467992        Huey Duck      /Duck
9   cd8fe1dd-5637-5037-931e-f8bf1a15c0b4       Della Duck      /Duck
10  f5bf5d66-5698-515a-800e-9d778d916dcd       Louie Duck      /Duck

Starting with version 2.5.0 of the CluedIn Python SDK, you can reduce the code above to one line and get nearly the same result. The method also returns every code and property for each entity, which is often what you need, without requiring you to copy the same GraphQL query each time:

# this will return all the queried entities with all properties and codes
print(pd.DataFrame(cluedin.gql.search(ctx, '+entityType:/Duck')))

If you only need a subset, use itertools.islice and set a smaller page_size so the SDK doesn't query more data than necessary:

from itertools import islice

# get a generator that queries three entities at a time
gen = cluedin.gql.search(ctx, '+entityType:/Duck', page_size=3)

# wrap in an iterator that stops after three iterations
iter = islice(gen, 3)

# convert to DataFrame
df = pd.DataFrame(iter)

print(df)

The same code can be written in one expression:

pd.DataFrame(islice(cluedin.gql.search(ctx, '+entityType:/Duck', 3), 3))