Roman Klimenko
DA

CluedIn data enrichment

cluedinpythonazure

youtu.be/EHFraghW71Q?si=UBKnzONb7EosEcUJ

The Enrich Rule Action lets you enrich data in CluedIn with external sources.

It calls an API that accepts a list of Vocabulary Keys and returns a list of properties. CluedIn saves those properties with a specified Vocabulary Prefix. You can implement the API as an Azure Function, a REST API, or any other service accessible over HTTP.

The API retrieves data from the external source, processes it, and returns the properties to CluedIn.

The Rule Action takes the following parameters:

  • URL: The URL of the enrichment API.
  • Payload: Comma-separated list of vocabulary keys to send to the API.
  • Vocabulary Prefix: The vocabulary prefix used to save properties returned by the API.

When invoked, the action sends the payload to the API in an HTTP POST request and saves the returned properties with the specified Vocabulary Prefix.

Example

In this example, customer.postcode is a Vocabulary Key that contains UK postcodes.

The external source is the open postcodes.io API.

A request looks like this:

curl --location 'api.postcodes.io/postcodes/CA2 6PJ'

Response:

{
    "status": 200,
    "result": {
        "postcode": "CA2 6PJ",
        "quality": 1,
        "eastings": 338102,
        "northings": 554389,
        "country": "England",
        "nhs_ha": "North West",
        "longitude": -2.966283,
        "latitude": 54.880414,
        "european_electoral_region": "North West",
        "primary_care_trust": "Cumbria Teaching",
        "region": "North West",
        "lsoa": "Carlisle 009D",
        "msoa": "Carlisle 009",
        "incode": "6PJ",
        "outcode": "CA2",
        "parliamentary_constituency": "Carlisle",
        "parliamentary_constituency_2024": "Carlisle",
        "admin_district": "Cumberland",
        "parish": null,
        "admin_county": null,
        "date_of_introduction": "198001",
        "admin_ward": "Morton",
        "ced": "Morton",
        "ccg": "NHS North East and North Cumbria",
        "nuts": "Carlisle",
        "pfa": "Cumbria",
        "codes": {
            "admin_district": "E06000063",
            "admin_county": "E99999999",
            "admin_ward": "E05014205",
            "parish": "E43000295",
            "parliamentary_constituency": "E14000620",
            "parliamentary_constituency_2024": "E14001152",
            "ccg": "E38000215",
            "ccg_id": "01H",
            "ced": "E58000165",
            "nuts": "TLD12",
            "lsoa": "E01019231",
            "msoa": "E02003995",
            "lau2": "E07000028",
            "pfa": "E23000002"
        }
    }
}

We can put an API between CluedIn and postcodes.io. Here is a JavaScript Azure Function that accepts the customer.postcode key, gets the postcode data, and returns it to CluedIn:

const { app } = require("@azure/functions");
const querystring = require("node:querystring");

app.http("postcodes", {
  methods: ["POST"],
  authLevel: "anonymous",
  handler: async (request, context) => {
    const parsedBody = querystring.parse(await request.text());

    const postcode = parsedBody["customer.postcode"];

    const response = await fetch(
      `https://api.postcodes.io/postcodes/${postcode}`,
      {
        method: "GET",
        headers: {
          "Content-Type": "application/json",
        },
      }
    );

    return {
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify(await response.json()),
      status: response.status,
    };
  },
});

Call the Azure Function with this request:

curl --location 'https://postcodes.azurewebsites.net/api/postcodes' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'customer.postcode=CA2 6PJ'

The function returns the response from the postcodes.io API.

This intermediate API keeps the enrichment logic outside CluedIn. You can write it in any language and host it on any platform that supports HTTP requests.

Now create a Data Part or Golden Record Rule in CluedIn:

  • URL: The URL of the Azure Function.
  • Payload: customer.postcode, which is sent in the request body.
  • Vocabulary Prefix: postcode, which is used to save the returned properties.

When invoked, the Rule Action sends the payload to the Azure Function and flattens the properties in its JSON response. The response above is saved in the enriched Entity as follows:

{
  "postcode.status": 200,
  "postcode.result.postcode": "CA2 6PJ",
  "postcode.result.quality": 1,
  "postcode.result.eastings": 338102,
  "postcode.result.northings": 554389,
  "postcode.result.country": "England",
  "postcode.result.nhs_ha": "North West",
  "postcode.result.longitude": -2.966283,
  "postcode.result.latitude": 54.880414,
  "postcode.result.european_electoral_region": "North West",
  "postcode.result.primary_care_trust": "Cumbria Teaching",
  "postcode.result.region": "North West",
  "postcode.result.lsoa": "Carlisle 009D",
  "postcode.result.msoa": "Carlisle 009",
  "postcode.result.incode": "6PJ",
  "postcode.result.outcode": "CA2",
  "postcode.result.parliamentary_constituency": "Carlisle",
  "postcode.result.parliamentary_constituency_2024": "Carlisle",
  "postcode.result.admin_district": "Cumberland",
  "postcode.result.parish": null,
  "postcode.result.admin_county": null,
  "postcode.result.date_of_introduction": "198001",
  "postcode.result.admin_ward": "Morton",
  "postcode.result.ced": "Morton",
  "postcode.result.ccg": "NHS North East and North Cumbria",
  "postcode.result.nuts": "Carlisle",
  "postcode.result.pfa": "Cumbria",
  "postcode.codes.admin_district": "E06000063",
  "postcode.codes.admin_county": "E99999999",
  "postcode.codes.admin_ward": "E05014205",
  "postcode.codes.parish": "E43000295",
  "postcode.codes.parliamentary_constituency": "E14000620",
  "postcode.codes.parliamentary_constituency_2024": "E14001152",
  "postcode.codes.ccg": "E38000215",
  "postcode.codes.ccg_id": "01H",
  "postcode.codes.ced": "E58000165",
  "postcode.codes.nuts": "TLD12",
  "postcode.codes.lsoa": "E01019231",
  "postcode.codes.msoa": "E02003995",
  "postcode.codes.lau2": "E07000028",
  "postcode.codes.pfa": "E23000002"
}

Additional Scenarios and Best Practices

Data Normalization

If the API returns a key that already exists in the enriched Entity, the Rule Action overwrites its value. To normalize phone numbers or addresses, for example, implement an API that returns the normalized data under the same keys sent for enrichment.

Data Validation

You can also use the Rule Action to validate data. For example, an API can check whether an email address is valid and return a boolean. The Rule Action saves that value with the specified Vocabulary Prefix.

Caching

If the external source is slow or rate-limited, your API can cache responses and reuse them when it receives the same request within a specified period.

Rate Limiting

If the external source enforces rate limits, your API can implement its own rate-limiting logic and return an error when a limit is reached. You can reprocess the affected Entities later.

Include the Retry-After response header to specify when to retry, and return the 429 Too Many Requests status code.

Alternatively, your API can accept the data and put it on a queue for later processing.

Error Handling

Always return JSON, even when an error occurs. The Rule Action saves the error message in the Entity, where it can be used for debugging or monitoring.

Security

The Rule Action sends the API key configured in the CLUEDIN_RULE_ACTION_API_KEY variable. Your API can validate it before processing the request.

Preview

Previewing the Rule Action sends a request to the API with is_preview set to true. The CluedIn UI displays the preview response returned by the API.