The magic of CluedIn

https://youtu.be/WrVayWXPUOo?si=XIUybRa5jS6w0Bmk

IPython, the command shell behind Jupyter notebooks, provides a useful feature called magics. Magics let you use command-line-style syntax instead of writing Python for every operation.

This approach can simplify repetitive tasks, including work with the CluedIn Python SDK. I built CluedIn Magic to provide that kind of interface for the CluedIn API.

CluedIn Magic depends on the CluedIn Python SDK, so installing it gives you both packages:

%pip install cluedin-magic

When working in Microsoft Fabric, Synapse Analytics, or Databricks, I usually install packages in the environment in advance so this step isn't needed in every notebook.

Load CluedIn Magic with %load_ext:

%load_ext cluedin_magic

Once loaded, you can call the %cluedin magic. With no parameters or invalid ones, it displays a short usage guide:

Available commands: get-context, search
Usage:
%cluedin get-context --jwt <jwt>
%cluedin search --context <context> --query <query> [--limit <limit>]

Get CluedIn context

Calls to the CluedIn API need a context with details such as the domain, organization name, credentials, or API token. CluedIn Magic can derive the other details from an API token.

You can create an API token from Administration -> API Tokens in CluedIn.

In this example, I store the token in an environment variable and read it into a notebook variable:

access_token = %env ACCESS_TOKEN

Pass the variable to CluedIn Magic:

ctx = %cluedin get-context --jwt $access_token

You can also pass the token directly:

ctx = %cluedin get-context --jwt eyJhbGci...5Odvpr1g

The resulting context works with both the CluedIn Python SDK and CluedIn Magic.

Search

To load all /Infrastructure/User Entities, provide a context and a query. The command returns the data as a pandas DataFrame:

%cluedin search --context ctx --query +entityType:/Infrastructure/User

For a large dataset, use a limit to return a sample. This example selects ten Entities of any type:

%cluedin search --context ctx --query * --limit 10

This query returns ten /IMDb/Name records where the imdb.name.birthYear Vocabulary Key does not equal \\N:

%cluedin search --context ctx --query +entityType:/IMDb/Name -properties.imdb.name.birthYear:"\\\\N" --limit 10

You can also store the result in a variable and use it as a regular pandas DataFrame:

pd = %cluedin search --context ctx --query +entityType:/IMDb/Name +properties.imdb.name.birthYear:1981
pd.head()