Skip to main content
Skip table of contents

IDM REST Bulk API Specification

Introduction

The IDM bulk API allows to make multiple create, patch, or delete operations on IM objects in a single "bulk request". The client sends a bulk request containing multiple operations to the IDM service. When received by the IDM service, bulk operations are subdivided in batches, each batch corresponding to a DB transaction. By default the transaction size is 5000, but this number is configurable: ${citadel.idm.batch.transaction-size}.

The bulk API is synchronous; the response is returned after all operations are complete. Each atomic operation is executed independently, so the failure of one operation won’t affect the success of the others. The bulk execution may be prematurely interrupted if too many consecutive errors occur. The number of consecutive errors triggering the interruption is configurable (10 by default): ${citadel.idm.batch.abort-consecutive-errors}.

If any of the operations fails, the error details are reported in the response.

It is possible to mix operations of different natures (such as create, patch or delete) in the same bulk request.

However, it is currently not possible to mix operations on different IM object classes, such as Identities or Organizations. A bulk request concerns a single kind of IM objects; Identities only, or Organizations only.

The response format can be controlled through the "options" of the batch request.

Currently 3 possible response formats are supported:

  • "object id": the batch response only contains object identifiers ("light" response)

  • "object full": the batch response contains full object representations, including deleted objects ("heavy" response)

  • "object projection": the batch response contains partial object representations

Overall JSON Structure of a Batch Request

The overall JSON structure of a batch request is:

CODE 
{
  "operations" : [ operation1, operation2 ],
  "options" : { "key1" : "value1", "key2" : "value2" }
}

A batch request is thus composed of "operations" and "options". Those are detailed below.

JSON Representation of Request Operations

The possible operations submitted in a bulk request are:

  • CREATE

  • CREATE_OR_UPDATE (QQQ should be named CREATE_OR_PATCH for consistency's sake?)

  • PATCH

  • DELETE

A bulk request is a JSON array of operations.

A single operation is composed of:

  • its nature: CREATE, DELETE, etc.

  • an identification of the targeted object(s), either one of (mutually exclusive):

    • a single identifier, 

    • a list of identifiers, for "patch" or "delete" operations (not relevant for "create" operations)

    • a search Expression, for "patch" or "delete" operations (not relevant for "create" operations)

  • a list of attributes, for "create" or "patch" operations  (not relevant for "delete" operations)

  • an optional context, which is a list of key-values (technically it is a Map<String, String>). The context is not interpreted by Memority. The context is sent back unmodified in the batch response, it may help the caller resume some processing after receiving the batch response

The JSON format of each operation is represented below.

JSON Create Operation Request Representation

CODE 
{
  "operation" : "CREATE",
  "type" : "employee",
  "id" : "jdoe",
  "attributes" : [ 
    { "id" : "firstName", "values" : [ "John" ] },
    { "id" : "lastName", "values" : [ "Doe" ] },
    { "id" : "company", "values" : [ "Memority" ] },
  ],
  "context" : {
    "key1" : "value1",
    "key2" : "value2"
  }
}

The identifier field is optional; if not provided then it will be auto-generated.

JSON Create or Update Request Operation Representation

Same as above, except the value of the operation field is CREATE_OR_UPDATE and the identifier field is mandatory for an "update".

JSON Patch Operation Request Representation

The same patch can be applied to several IM objects. The list of IM objects to patch can be provided in 3 ways, which are mutually exclusive:

  • a single IM object identifier (i.e. a single object to patch)

  • an explicit array of IM object identifiers

  • a search Expression; each IM object returned by the search will be patched

QQQ Need a "preview" or "dry run" function showing the entries returned by the search before applying the patch?

When providing an explicit array of IM object identifiers to patch, the JSON representation of a patch operation is:

CODE 
{
  "operation" : "PATCH",
  "objects" : [ 
    { "id" : "jdoe" },
    { "id" : "mjordan" },
    { "id" : "spippen" },
  ],
  "attributes" : [ 
    { "operation" : "SET", "id" : "active", "values" : [ "true" ] } 
    { "operation" : "SET", "id" : "expiration_date", "values" : [ "2020-12-31" ] } 
  ]
}

When providing a search Expression of IM objects to patch, the JSON representation of a patch operation is:

CODE 
{
  "operation" : "PATCH",
  "expression" : {
    "prop" : "email",
    "op" : "ENDS_WITH",
    "values" : [ "@memority.com" ]
  },
  "attributes" : [ 
    { "operation" : "SET", "id" : "active", "values" : [ "true" ] } 
    { "operation" : "SET", "id" : "expiration_date", "values" : [ "2020-12-31" ] } 
  ]
}

When providing a single IM object identifier to patch, the JSON representation of a patch operation is:

CODE 
{
  "operation" : "PATCH",
  "id" : "jdoe",
  "attributes" : [ 
    { "operation" : "SET", "id" : "active", "values" : [ "true" ] } 
    { "operation" : "SET", "id" : "expiration_date", "values" : [ "2020-12-31" ] } 
  ]
}

One could also use the first representation, with only one entry in the objects array.

JSON Delete Operation Request Representation

Like the patch operation, the list of IM objects to delete can be provided in 3 ways, which are mutually exclusive:

  • a single IM object identifier (i.e. a single object to delete)

  • an explicit array of IM object identifiers

  • a search Expression; each IM object returned by the search will be deleted

When providing an explicit array of IM object identifiers to delete, the JSON representation of a delete operation is:

CODE 
{
  "operation" : "DELETE",
  "objects" : [ 
    { "id" : "jdoe" },
    { "id" : "mjordan" },
    { "id" : "spippen" },
  ]
}

When providing a search Expression of IM objects to delete, the JSON representation of a delete operation is:

CODE 
{
  "operation" : "DELETE",
  "expression" : {
    "prop" : "email",
    "op" : "ENDS_WITH",
    "values" : [ "@memority.com" ]
  }
}

When providing a single IM object identifier to delete, the JSON representation of a delete operation is:

CODE 
{
  "operation" : "DELETE",
  "id" : "jdoe",
}

One could also use the first representation, with only one entry in the objects array.

JSON Representation of Request Options

Here is an example of options submitted in a bulk request:

CODE 
{
  "responseFormat" : "OBJECT_PROJECTION",
  "projectedAttributes" : [ "attr1",  "attr3 ]
}

The possible values of responseFormat are:

  • OBJECT_ID (the default): the batch response only contains object identifiers

  • OBJECT_FULL : the batch response contains whole object representations

  • OBJECT_PROJECTION : the batch response contains partial object representations. Only the "projected attributes" are returned.

The projectedAttributes option only makes sense when the requested format if OBJECT_PROJECTION.

JSON Representation of a whole Bulk Request

A bulk request contains a JSON array of operations and possibly options. A "general case" example is represented below, mixing operations of different natures (CREATE , PATCH, DELETE):

CODE 
{
  "operations" : [ {
    "operation" : "CREATE",
    "type" : "employee",
    "id" : "identityToCreate",
    "attributes" : [ {
      "id" : "firstName",
      "values" : [ "John" ]
    }, {
      "id" : "lastName",
      "values" : [ "Doe" ]
    }, {
      "id" : "login",
      "values" : [ "jdoe" ]
    }, {
      "id" : "status",
      "values" : [ "ACTIVE" ]
    }, {
      "id" : "enabledFrom",
      "values" : [ "2018-02-01T14:31:47.532+0000" ]
    }, {
      "id" : "enabledUntil",
      "values" : [ "2018-03-01T14:31:47.532+0000" ]
    } ],
    "context" : {
      "key1" : "value1",
      "key2" : "value2"
    }
  }, {
    "operation" : "PATCH"
    "id" : "identityToPatch",
    "attributes" : [ {
      "id" : "firstName",
      "values" : [ "JACK" ],
      "operation" : "REPLACE"
    }, {
      "id" : "company",
      "values" : [ "Memority" ],
      "operation" : "REPLACE"
    }, {
      "id" : "enabledFrom",
      "values" : [ "1970-01-01T00:00:00.001+0000" ],
      "operation" : "SET"
    }, {
      "id" : "status",
      "values" : [ "FAILED" ],
      "operation" : "SET"
    }, {
      "id" : "enabled",
      "values" : [ false ],
      "operation" : "SET"
    }, {
      "id" : "flags",
      "values" : [ 4 ],
      "operation" : "SET"
    }, {
      "id" : "locked",
      "values" : [ true ],
      "operation" : "SET"
    }, {
      "id" : "password",
      "values" : [ "password" ],
      "operation" : "SET"
    } ]
  }, {
    "operation" : "DELETE",
    "id" : "identityToDelete"
  } ],
  "options" : {
    "responseFormat" : "OBJECT_ID"
  }
}

The format of the bulk response is represented later.

The next sections describe the URI on which REST bulk operations are invoked.

REST Bulk API

The following table presents the general format of the IDM REST bulk API for the "Identity" case.

Resource

POST

/api/identities/~bulk

General purpose bulk API handling all natures of operation: CREATE, PATCH, DELETE

JSON Representation of a Bulk Response

A bulk response is represented below. The response only contains object identifiers.

CODE 
{
  "status" : "PARTIAL_ERROR",
  "numberOfObjectsToProcess" : 6,
  "aborted" : false,
  "startDate" : "2017-03-21T10:34:22",
  "createdObjects" : [ 
    { "id" : "jdoe", "kind" : "IDENTITY", "context" : { "key1" : "value1", "key2" : "value2" },
    { "id" : "mjordan", "kind" : "IDENTITY" },
  ],
  "patchedObjects" : [ 
    { "id" : "ljames", "kind" : "IDENTITY" },
    { "id" : "spippen", "kind" : "IDENTITY" },
    { "id" : "jharden", "kind" : "IDENTITY" },
  ],
  "deletedObjects" : [ 
    { "id" : "drodman", "kind" : "IDENTITY" },
    { "id" : "jkidd", "kind" : "IDENTITY" },
  ],
  "processingErrors" : [ {
    "operation" : "CREATE",
    "id" : "jdoe",
    "error" : {
      "@class" : "com.memority.toolkit.core.error.SimpleErrorRepresentation",
      "errorId" : "CTD-E1030004",
      "label" : "AGGREGATE_ALREADY_EXISTS",
      "description" : "Aggregate 'jdoe' already exists",
      "properties" : {
        "id" : "jdoe"
      },
      "timestamp" : "2017-03-21T10:34:22",
      "logTrackingId" : "6b265ae7-c92f-4045-ae18-35b4aeeee448"
    },
  }, {
    "operation" : "DELETE",
    "id" : "gpayton",
    "error" : {
      "@class" : "com.memority.toolkit.core.error.SimpleErrorRepresentation",
      "errorId" : "CTD-E1030005",
      "label" : "AGGREGATE_NOT_FOUND",
      "description" : "Aggregate with id 'gpayton' could not be found!",
      "properties" : {
        "id" : "gpayton"
      },
      "timestamp" : "2017-03-21T10:34:22",
      "logTrackingId" : "7b265ae7-c92f-4045-ae18-35b4aeeee449"
    },
  } ],
  "processingTimeMillis" : 3
}

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.

Contact Support Close