SCIM Stream

Author: Jarle Elshaug

Overview

SCIM Stream is the modern way of user provisioning letting clients subscribe to messages instead of traditional IGA top-down provisioning. SCIM Stream collects information from authoritative sources and publish SCIM-formatted messages to channels according to predefined rules. Endpoints/Applications can then subscribe to relevant stream channels and use their own logic for message handling.

SCIM Stream includes SCIM Stream Gateway, the next generation SCIM Gateway that supports message subscription and automated provisioning. See SCIM Gateway for standard functionality (the none streaming version).

Latest news:

  • Dynamic groups, roles and profile attributes giving full ABAC/RBAC functionality

Highlights:

  • Pub/Sub having subscription-based provisioning initiated by client
  • Multi Tenancy
  • One binary and a configuration file built in a modern programming language, Go
  • Docker, Kubernetes and Unikernel-friendly
  • Using NATS for sending and receiving messages
  • No client inbound traffic, only outbound initiated by client which gives the most secure communication flow e.g. Cloud => On-Premise
  • Currently Azure AD (AAD) is supported as authoritative collection source
  • Replace or complements AAD Provisioning engine
  • Collects information from AAD which might have max 5 minutes internal Azure log delay (99% latency)
  • Much faster (max 5 minutes delay) than standard Azure provisioning that may be delayed up to 40 minutes
  • All changes to users and their group/application membership are processed
  • SCIM formatted message inludes a full user object (all user attributes including groups and application roles), Operations (changed attributes), and some general information like type of operation, when changed, who did the change etc.
  • Using a pre-defined AAD to SCIM attribute mapping which can be extended and customized by configuration.
  • Rules are defined per streaming channel (subscriber) supporting all attributes:
    • Filter: one or more advanced regular expression filters to user/group/role e.g.:
      (user.groups.display -match "App4.*") and (user.title -match "(Dog Trainer|Project Manager)")
  • Only process messages for users that match streaming rules
  • The user object in message only includes groups/applications according to rules (other groups/applications that user is member of are excluded)
  • Application roles are mapped to SCIM roles giving final message like: {"roles":[{type":"Application1","display":"Employees access","value":"Employees"}]
  • Nested groups are supported also linked to Application roles
  • Includes rulebased dynamic groups, roles and profile attributes for ABAC/RBAC. Gives IGA functionality e.g.: gives dynamic groups and Application roles when using Free Azure AD that do not have this functionality.
  • Supports initial load having all users in AAD processed according to both streaming and dynamic rules
  • Includes SCIM Stream Gateway
    • The next generation SCIM Gateway supporting message subscription and automated provisioning. This means users will be automatically created, modified, and deleted based on push notifications - streaming messages
    • Existing SCIM Gateway plugins can be used out-of-the-box
    • All groups/access definitions at the target endpoint can be exported and imported into Azure as Application Roles for out-of-the-box access management in Azure
    • Option for converting roles to groups. Plugins existing group-logic can then be used instead of handling roles

Message format

SCIM Stream use NATS message technology having a SCIM formatted message that includes following:

  • General information: activityOperation (modifyUser/createUser/deleteUser/initialLoad), initiatedByUserName, targetUserName, …
  • Operations: standard SCIM v2.0 Operations object that includes what have been changed
  • user: SCIM formatted user object including all user attributes according to attribute mapper and stream rules (groups/roles)

SCIM formatted message below shows an example of a user that have been assigned the Azure AD group “Employee” (can be found in Operations and user.groups)

{
  "activityOperation": "modifyUser",
  "activityDateTime": "2022-07-01T10:27:55.403989Z",
  "initiatedById": "2b47d2fd-a98e-4051-b053-93d2da7f834e",
  "initiatedByUserName": "[email protected]",
  "targetId": "da33c8bd-4511-5058-b79e-7a45938b1b08",
  "targetUserName": "[email protected]",
  "Operations": [{
          "op": "add",
          "value": [{
                  "displayName": "Employees",
                  "id": "Employees"
              }
          ],
          "path": "groups"
      }
  ],
  "user": {
      "active": true,
      "addresses": [{
              "type": "work",
              "region": "CA",
              "city": "Hollywood",
              "postalCode": "91608",
              "streetAddress": "100 Universal City Plaza1",
              "country": "USA"
          }
      ],
      "displayName": "John Smith",
      "emails": [{
              "type": "other",
              "value": "[email protected]"
          }, {
              "type": "work",
              "value": "[email protected]"
          }
      ],
      "externalId": "johns",
      "groups": [{
              "display": "Employees",
              "value": "Employees"
          }, {
              "display": "Admins",
              "value": "Admins"
          }
      ],
      "name": {
          "givenName": "John",
          "familyName": "Smith",
          "formatted": "John Smith"
      },
      "phoneNumbers": [{
              "type": "mobile",
              "value": "555-555-6521"
          }, {
              "type": "work",
              "value": "555-555-1256"
          }
      ],
      "roles": [{
              "type": "Application1",
              "display": "User",
              "value": "User"
          }
      ],
      "title": "Consultant",
      "userName": "[email protected]",
      "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User": {
          "organization": "Universal Studios",
          "employeeNumber": "991999",
          "manager": {
              "value": "[email protected]",
              "displayName": "Barbara Jensen"
          },
          "department": "Tour Operations"
      },
      "schemas": ["urn:ietf:params:scim:schemas:core:2.0:User", "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User"]
  }
}

Access Management

How can we administer user access like groups on endpoints/applications using SCIM Stream and Azure AD as authoritative collection source?

There are several ways to do this like using AAD groups and/or Application roles.

The best fit is probably using Application roles having the value set to the actual endpoint group id/name. When user is assigned an Application role in AAD, the Application name and the role value will be included in the SCIM message. Example message below shows that Application role value Employees have been added (user already have Admins):

{
  ...
  "Operations":[{"op":"add","value":[{"type":"Application1","display":"Employee access","value":"Employees"}],"path":"roles"}]
  ...
  "user":{
    ...
    "roles":[{"type":"Application1","display":"Employees access","value":"Employees"},
             {"type":"Application1","display":"Admins access","value":"Admins"}],
    ...
  }
}

We could then have one stream rule telling to include Application1

Subscriber or the SCIM Stream Gateway plugin then have logic for handling roles and will do a one-to-one match with endpoint groups using the role value content.

SCIM Stream Gateway have optional logic for converting roles to groups. Plugins existing group-logic can then be used instead of handling roles

SCIM Stream Gateway includes logic for exporting all target endpoint groups/accesses into Application roles that can be imported into Azure. This way we can ensure that Application roles in Azure reflects all corresponding accesses on endpoints and we get out-of-the-box management from Azure.

We could also use AAD Groups, having same group names defined in AAD as we have on the actual endpoint. There is also another way like prefixing AAD group with endpoint/application name e.g., App1_Employee

We could then have a SCIM Stream filtering rule like:
- filter: (user.groups.display -match "App1_.*")

Subscriber or the SCIM Stream Gateway plugin then have logic for removing “App1_” prefix from the group name and will do a one-to-one match towards endpoint having the same group name.

Configuration

SCIM Stream includes:

  • a binary: scim-stream
  • a configuration file: config.yaml

Below is an example of config.yaml configuration setup for collecting information from one AAD tenant every 5 second and streams to 5 different channels: App1, App2, App3, App4 and App5

azure:
- auth:
    clients:
    - client_id: <client-id>
      client_secret: <client-secret>
    tenant_id: <tenant-id>
  general:
    collect_interval: 5
    do_initial_load: false
    use_original_group_id: false
    stream_subscribers:
    - user_public_key: <user-public-key>
    stream_tenant_id: <stream-tenant-id>
  streams:
  - channel: App1
    comment: Includes one application App1, using special attribute typeId that corresponds with Azure Application ID
    rules:
    - filter: (user.roles.typeId -match "App1-id")
  - channel: App2
    comment: Includes two groups
    rules:
    - filter: (user.groups.value -match "group1-id")
    - filter: (user.groups.value -match "group2-id")
  - channel: App3
    comment: Includes all groups having name starting with "App3" and Azure Application name "Application 1" with role value "Employees"
    rules:
    - filter: (user.groups.display -match "App3.*")
    - filter: (user.roles.Application 1.value -match "Employees")
  - channel: App4
    comment: Includes all users having email address suffix "@mycompany.com" and all users member of group name startig with App3 having user title Dog Trainer or Project Manager
    rules:
    - filter: (user.emails.work.value -match "*[email protected]")
    - filter: (user.groups.value -match "App4.*") and (user.title -match "(Dog Trainer|Project Manager)")
  - channel: App5
    comment: All included (all users/groups/applications)
    rules:
    - filter: (.*)
log:
  console: false
  level: debug
  max_age: 0
  max_backups: 10
  max_size: 10
stream_server:
  encryption_at_rest_key: <secret-key>
  port: <port>
  certificate:
    certfile: <cert.pem>
    keyfile: <key.pem>
    cafile: <ca.pem>

Configuration may include rulebased logic for applying dynamic groups, roles and profile attributes giving ABAC/RBAC functionality:

dynamics:
- comment: ABAC/RBAC matching title Consultant and will update user with 2 groups, 2 roles including some profile attributes
  dynamic_id: dynamic-1
  rules:
  - filter: (user.title -match ".*Consultant.*")
  assignments:
    user.groups:
    - display: Grp1-name
      value: Grp1-id
    - display: Grp2-name
      value: Grp2-id
    user.roles:
    - display: Admin access
      value: Admin
      type: App1
    - display: Employee access
      value: Employee
      type: App1
    user.addresses:
    - postalCode: 1234
      streetAddress: Back Office Street
      type: work
    user.type: External

Configuration may also include a mapper section for mapping AAD attributes to SCIM attributes. Any mapper definition included will be merged with default mapper configuration that have following settings:

  mapper:
    group:
    - displayName:
        map_to: displayName
    user:
    - userPrincipalName:
        map_to: userName
    - accountEnabled:
        map_to: active
    - displayName:
        map_to: displayName
    - givenName:
        map_to: name.givenName
    - surname:
        map_to: name.familyName
    - Join(" ", [givenName], [surname]):
        map_to: name.formatted
    - mail:
        map_to: emails[type eq "work"].value
    - otherMails:
        map_to: emails[type eq "other"].value
    - businessPhones:
        map_to: phoneNumbers[type eq "work"].value
    - jobTitle:
        map_to: title
    - preferredLanguage:
        map_to: preferredLanguage
    - physicalDeliveryOfficeName:
        map_to: addresses[type eq "work"].formatted
    - streetAddress:
        map_to: addresses[type eq "work"].streetAddress
    - state:
        map_to: addresses[type eq "work"].region
    - city:
        map_to: addresses[type eq "work"].city
    - postalCode:
        map_to: addresses[type eq "work"].postalCode
    - country:
        map_to: addresses[type eq "work"].country
    - mobilePhone:
        map_to: phoneNumbers[type eq "mobile"].value
    - facsimileTelephoneNumber:
        map_to: phoneNumbers[type eq "fax"].value
    - mailNickname:
        map_to: externalId
    - companyName:
        map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:organization
    - employeeId:
        map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:employeeNumber
    - department:
        map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:department
    - employeeOrgData.costCenter:
        map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:costCenter
    - employeeOrgData.division:
        map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:division
    - manager.userPrincipalName:
        map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:manager.value
    - manager.displayName:
        map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:manager.displayName
    - groups:
        map_to: groups
    - appRoles:
        map_to: roles

Note, following mapper settings cannot be overridden:

  mapper:
    group:
    - displayName:
        map_to: displayName
    user:
    - groups:
        map_to: groups
    - appRoles:
        map_to: roles

Note, the most important mapper configuration is the unique naming attribute - SCIM mandatory userName:

  mapper:
    user:
    - userPrincipalName:
        map_to: userName

Details

Config key Description
azure one or more azure authoriative collection sources
azure.auth authentication configuration
azure.auth.clients one or more azure applications, api permissions = AuditLog.Read.All + Directory.Read.All. Having more than one client/application configured will reduce azure throttling limitations
azure.auth.clients.client_id azure application client id
azure.auth.clients.client_secret azure application client secret, will be encrypted on startup
azure.auth.tenant_id azure tenant id
azure.general general configuration
azure.general.collect_interval collection interval in seconds, minimum and default is 5 seconds
azure.general.do_initial_load optional, true or false, default false. true will process all users in AAD according to rules, value automatically reverted back to false
azure.general.stream_subscribers one or more subscriber configurations - private/public key (NATS nkey) generated by command scim-stream -user, public key defiend as user_public_key and private key defined at the subscriber/client (SCIM Stream Gateway)
azure.general.stream_subscribers.user_public_key user public key (the private key is used by subscriber)
azure.general.stream_tenant_id unique tenant id for stream. This could be the name of the tenant (no spaces or special characters) or for example the azure tenant id. This unique tenant id must also be used in the subscriber/client subject having syntax: <tenantId>.<collector>.<channel>
azure.general.use_original_group_id optional, true or false, default false. true will set SCIM group-id to Azure group-id instead of using displayName as id
azure.dynamics optional, one or more rules for dynamic groups, roles and profile attributes giving ABAC/RBAC functionality
azure.dynamics.dynamicId self defined uniqe id for dynamic rule - this id is also used in stream rules for including dynamic users e.g., (user.dynamicIds -match "dynamic-1")
azure.dynamics.rules one or more filter rules for dynamic assignments, several rules gives “OR” logic
azure.dynamics.rules.filter advanced regular expression supporting the -match operator related to user/group/role (SCIM-formatted) e.g:
(user.title -match ".*Consultant.*")
azure.dynamics.assignments contains assignments definitions on what will be applied when rule(s) are fulfilled and also will be revoked when previous have been fulfilled. Note, assignment key always relates to “user.attribute” e.g. user.group, user.roles, user.addresses, user.name.givenName, …
azure.dynamics.assignments.user.groups one or more groups to be assigned
azure.dynamics.assignments.user.groups.display group display name
azure.dynamics.assignments.user.groups.value group value (id)
azure.dynamics.assignments.user.roles one or more scim roles to be assigned
azure.dynamics.assignments.user.roles.type role type, e.g: App1 - note for Azure Application roles the type corresponds to Azure Application name
azure.dynamics.assignments.user.roles.display display name that correspond to value e.g. “Admin access”
azure.dynamics.assignments.user.roles.value value e.g. “Admin”
azure.dynamics.assignments.user.xxx all user attrbutes may be used e.g. user.addresses, user.emails, user.title, user.name.givenName, …
azure.mapper optional, default Azure to SCIM attributes mapping is included. Configuration defined will override or extend defaults
azure.mapper.user one or more user mapping (group mapping cannot be overridden)
azure.mapper.user.aad-attribute aad-attribute is the name of the AAD attribute to be mapped, also supporting virtual attributes like Join(" ", [givenName], [surname])
azure.mapper.user.aad-attribute.map_to contains the SCIM attribute name like: userName, name.givenName, emails[type eq "work"].value, urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:organization
azure.streams azure streams configuration
azure.streams one or more azure streams configuration
azure.streams.channel mandatory channel name used by SCIM Stream publisher. Same channel name must also be used in the subscriber/client subject having syntax: <tenantId>.<collector>.<channel>
azure.streams.comment optional
azure.streams.rules one or more filter rules for message publishing, several rules gives “OR” logic
azure.streams.rules.filter advanced regular expression supporting the -match operator related to user/group/role (SCIM-formatted) e.g:
(user.groups.value -match "App1.*") and (user.title -match "(Dog Trainer|Project Manager)")
other filter examples:
(.*) includes all users/groups/applications
(user.name.givenName -match ".*enr.*")
(user.emails.work.value -match ".*@mycompany.com")
(user.addresses.work.postalCode -match "91608")
(user.groups.value -match "9936d283-2ac4-4c78-a8a0-52631fb20344")
(user.groups.display -match "App1")
(user.roles.typeId -match "a12f063b-1c8b-40e0-9988-01b46f478349")
(user.roles.Azure Application Test.value -match ".*")
(user.roles.Azure Application Test.value -match "role-value1")
(user.dynamicIds -match "dynamic-1")
log log configuration
log.console true or false, default false. true sends log to stdout instead of file
log.level debug, error, info or disabled
log.max_age number of days to retain old log files, default 0 - disabled
log.max_backups number of logfiles to keep, default 10
log.max_size max logfile size in Mega bytes before rollover to new file, default 10
streamserver NATS stream server configuration
streamserver.encryption_at_rest_key define your own secret key used for encryption at rest
streamserver.port port used by SCIM Stream publisher and client subscribers, default 9012
streamserver.certificate optional, files must be located in ./config/certs. If not defiend, all files will be autogenerated (ca.pem, cert.pem and key.pem)
streamserver.certificate.certfile name of certificate file e.g. cert.pem, file
streamserver.certificate.keyfile name of private key file e.g. key.pem
streamserver.certificate.cafile name of certificate authority file e.g. ca.pem, set to same as certfile if using self-signed certificate. This CA file should also be copied to the client/subscriber (SCIM Stream Gateway)

Environments

Configuration may also be set using environments or external file.

Syntax is variable names like above dotted notation, but starting with SCIM-STREAM_, all uppercase, and underscore instead of dots. Attributes having underscore e.g. user_private_key must be used without underscore like USERPRIVATEKEY. Arrays like Azure using AZURE[index]

Example 1:

azure:
- auth:
    clients:
    - client_secret: my-client-secret

Becomes:

export SCIM-STREAM_AZURE[0]_AUTH_CLIENTS[0]_CLIENTSECRET=my-client-secret

Example 2:

stream_server:
  publisher:
    user_private_key: my-private-key

Becomes:

export SCIM-STREAM_STREAMSERVER_PUBLISHER_USERPRIVATEKEY=my-private-key

Environments may also be defined in a vault file having only the initial vaultfile defined as environment

Example:

export SCIM-STREAM_VAULTFILE=/var/run/vault/.vaultfile

file "/var/run/vault/.vaultfile" having content:
SCIM-STREAM_AZURE[0]_AUTH_CLIENTS[0]_CLIENTSECRET=my-client-secret
SCIM-STREAM_STREAMSERVER_PUBLISHER_USERPRIVATEKEY=my-private-key

Secrets like azure[].auth.clients[].client_secret and streamserver.publisher.user_private_key defined in configuration file will automatically become encrypted on startup, and this encrypted configuration file cannot be copied from one machine to another. Seed logic used for encryption may be overridden by your own seed defined as environment SEED e.g.:

export SEED=SomeRandomCharacters:-)

SCIM Stream Gateway

SCIM Stream Gateway is the next generation SCIM Gateway for message subscription and automated provisioning. This Gateway is the latest version of the SCIM Gateway, but not public available.

Having subscriber enabled and messages are received will result in automated provisioning through any existing plugin - everything out-of-the-box.

Configuration

SCIM Stream Gateway requires the following subscriber section to be added in the plugin configuration:

{
  "scimgateway": {
    ...
    "subscriber": {
      "enabled": true,
      "servers": ["nats://<scim-stream-host>:<port>"],
      "certificate": {
        "ca": "cert.pem"
      },
      "entity": {
        "undefined": {
          "subject": "<tenantId>.<collector>.<channel>",
          "queueGroup": null,
          "userPrivateKey": "<private-key>",
          "deleteUserOnLastGroupRoleRemoval": false,
          "convertRolesToGroups": false,
          "generateUserPassword": false,
          "replaceDomains": [{
            "from": "@my-company.onmicrosoft.com",
            "to": "@my-company.com"
           }]
        }
      }
    },
	...
  },
  "endpoint": {
    ...
      "entity": {
        "undefined": {
          ...
        }
      }
    ...
  }
}
  • subscriber - contains configuration needed for activating SCIM Stream subscription
  • subscriber.enabled - true/false, true enables subscription
  • subscriber.servers - array of one or more SCIM Stream servers to connect, syntax: ["nats://<scim-stream-host>:<port>"]. Using NATS streaming technology.
  • subscriber.certificate.ca - CA file name that have been copied from the SCIM Stream server and should be located in /config/certs directory
  • subscriber.entity.xxx - enity corresponds with url-baseEntity and may be used in the endpoint section for multi-tenant or multi-endpoint support. Entities defined in the subscriber sections will be linked to corresponding entities defined in the endpoint sections. If no entities are defined in the endpoint section, we use the default entity named “undefined”.
  • subscriber.entity.xxx.subject - <tenantId>.<collector>.<channel>. must match SCIM Stream server general.stream_tenant_id configuration. must match SCIM Stream server suported collector type e.g. AZURE. must match SCIM Stream server streams.channel.name configuration. e.g.: tenant1.AZURE.App1. Note, subject is case sensitive.
  • subscriber.entity.xxx.queueGroup - queueGroup ensures messages are processed by only one of the clients in the same queueGroup. If defined, all clients subscribing to same subject must use the same queueGroup name e.g “plugin-xxx”
  • subscriber.entity.xxx.userPrivateKey - User private key (NATS nkey). Will become encrypted on startup. scim-stream -user generates a user private/public key. Corresponding public key should be defined in the SCIM Stream azure tenant configuration azure.general.subscribers.user_public_key
  • subscriber.entity.xxx.deleteUserOnLastGroupRoleRemoval - optional true/false, true will delete user when the last group/role become removed
  • subscriber.entity.xxx.convertRolesToGroups - optional true/false, true will convert roles to groups. Operations are converted and user.roles becomes user.groups having correct syntax. Existing plugins group-logic can then be used instead of building new logic related to roles.
  • subscriber.entity.xxx.generateUserPassword - optional true/false, true will generate a random user password on createUser operation if password not included (Azure do not include password)
  • subscriber.entity.xxx.replaceDomains - optional, array of upn/email domains to be replaced (from/to objects)
  • subscriber.entity.xxx.replaceDomains[].from - domain name to be replaced, must start with “@” e.g. “@my-company.onmicrosoft.com”
  • subscriber.entity.xxx.replaceDomains[].to - new domain name, must start with “@” e.g. “@my-company.com”

Azure Application Roles

SCIM Stream Gateway may retrieve all groups/accesses from target endpoints and create a result having these converted into Azure Application Roles. Result provided includes appRoles content supported by the Azure Application Manifest-appRoles definition. We may copy/paste this appRoles result into Azure Application Manifest and start administring all our existing target endpoint groups/accesses in Azure through Application roles, everything out-of-the-box and unique uuid’s preserved on future run that might include new accesses definitions.

Azure Application Roles can be generated by using GET /AppRoles

Example using defult loki-plugin:

http://localhost:8880/AppRoles

After having updated the Azure Manifest with appRoles, all we have to do is configuring SCIM Stream to include the Azure Application ID to a stream channel rule:

  streams:
  - channel: App1
    comment: Includes one application App1
    rules:
    - application_id: <app1-id>

And configure SCIM Stream Gateway plugin to use that stream channel:

{
  "scimgateway": {
    ...
    "subscriber": {
      ...
      "entity": {
        "undefined": {
          "subject": "AZURE.App1",
          "convertRolesToGroups": true
          ...
        }
      }
    },
  ...
}

Note, by setting "convertRolesToGroups": true we can use plugins existing group-logic instead of using roles

License

© Jarle Elshaug