SCIM Stream


Author: Jarle Elshaug

Overview

SCIM Stream is the modern way of user provisioning letting clients subscribe to messages instead of traditional IGA top-down provisioning. SCIM Stream collects information from authoritative sources and publish SCIM-formatted messages to channels according to predefined rules. Endpoints/Applications can then subscribe to relevant stream channels and use their own logic for message handling.

SCIM Stream includes SCIM Stream Gateway, the next generation SCIM Gateway that supports message subscription and automated provisioning. See SCIM Gateway for standard functionality (the none streaming version).

Latest news:

  • Dynamic groups and roles (where roles mean grouping of resources) giving ABAC/RBAC functionality

Highlights:

  • Subscription-based provisioning initiated by client
  • One binary and a configuration file built in Go
  • Docker, Kubernetes and Unikernels-friendly
  • Using NATS JetStream for sending and receiving messages
  • No inbound, only outbound traffic which gives the most secure communication flow e.g. Cloud => On-Premise
  • Currently Azure AD (AAD) is supported as authoritative collection source
  • Replace or complements AAD Provisioning engine
  • Collects information from AAD which might have max 5 minutes internal Azure log delay (99% latency)
  • Much faster (max 5 minutes delay) than standard Azure provisioning that may be delayed up to 40 minutes
  • All changes to users and their group/application membership are processed
  • SCIM formatted message inludes a full user object (all user attributes including groups and application roles), Operations (changed attributes), and some general information like type of operation, when changed, who did the change etc.
  • Using a pre-defined AAD to SCIM attribute mapping which can be extended and customized by configuration.
  • Rules are defined per streaming channel (subscriber) supporting all attributes:
    • Filter: one or more advanced regular expression filters to user/group/role e.g.:
      (user.groups.display -match "App4.*") and (user.title -match "(Dog Trainer|Project Manager)")
  • Only process messages for users that match streaming rules
  • The user object in message only includes groups/applications according to rules (other groups/applications that user is member of are excluded)
  • Application roles are mapped to SCIM roles giving final message like: {"roles":[{type":"Application1","display":"Employees access","value":"Employees"}]
  • Nested groups are supported also linked to Application roles
  • Also includes rulebased dynamic groups and roles for ABAC/RBAC. Gives IGA functionality e.g.: gives dynamic groups and Application roles when using Free Azure AD that do not have this functionality.
  • Supports initial load having all users in AAD processed according to both streaming and dynamic rules
  • Includes SCIM Stream Gateway
    • The next generation SCIM Gateway for message subscription and automated provisioning. This means users will be automatically created, modified, and deleted based on incoming messages
    • Existing SCIM Gateway plugins can be used out-of-the-box
    • All groups/access definitions at the target endpoint can be exported and imported into Azure as Application Roles for out-of-the-box access management in Azure
    • Option for converting roles to groups. Plugins existing group-logic can then be used instead of handling roles

Message format

SCIM Stream use NATS message technology having a SCIM formatted message that includes following:

  • General information: activityOperation (modifyUser/createUser/deleteUser/initialLoad), initiatedByUserName, targetUserName, …
  • Operations: standard SCIM v2.0 Operations object that includes what have been changed
  • user: SCIM formatted user object including all user attributes according to attribute mapper and stream rules (groups/roles)

SCIM formatted message below shows an example of a user that have been assigned the Azure AD group “Employee” (can be found in Operations and user.groups)

{
  "activityOperation": "modifyUser",
  "activityDateTime": "2022-07-01T10:27:55.403989Z",
  "initiatedById": "2b47d2fd-a98e-4051-b053-93d2da7f834e",
  "initiatedByUserName": "[email protected]",
  "targetId": "da33c8bd-4511-5058-b79e-7a45938b1b08",
  "targetUserName": "[email protected]",
  "Operations": [{
          "op": "add",
          "value": [{
                  "displayName": "Employees",
                  "id": "Employees"
              }
          ],
          "path": "groups"
      }
  ],
  "user": {
      "active": true,
      "addresses": [{
              "type": "work",
              "region": "CA",
              "city": "Hollywood",
              "postalCode": "91608",
              "streetAddress": "100 Universal City Plaza1",
              "country": "USA"
          }
      ],
      "displayName": "John Smith",
      "emails": [{
              "type": "other",
              "value": "[email protected]"
          }, {
              "type": "work",
              "value": "[email protected]"
          }
      ],
      "externalId": "johns",
      "groups": [{
              "display": "Employees",
              "value": "Employees"
          }, {
              "display": "Admins",
              "value": "Admins"
          }
      ],
      "name": {
          "givenName": "John",
          "familyName": "Smith",
          "formatted": "John Smith"
      },
      "phoneNumbers": [{
              "type": "mobile",
              "value": "555-555-6521"
          }, {
              "type": "work",
              "value": "555-555-1256"
          }
      ],
      "roles": [{
              "type": "Application1",
              "display": "User",
              "value": "User"
          }
      ],
      "title": "Consultant",
      "userName": "[email protected]",
      "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User": {
          "organization": "Universal Studios",
          "employeeNumber": "991999",
          "manager": {
              "value": "[email protected]",
              "displayName": "Barbara Jensen"
          },
          "department": "Tour Operations"
      },
      "schemas": ["urn:ietf:params:scim:schemas:core:2.0:User", "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User"]
  }
}

Access Management

How can we administer user access like groups on endpoints/applications using SCIM Stream and Azure AD as authoritative collection source?

There are several ways to do this like using AAD groups and/or Application roles.

The best fit is probably using Application roles having the value set to the actual endpoint group id/name. When user is assigned an Application role in AAD, the Application name and the role value will be included in the SCIM message. Example message below shows that Application role value Employees have been added (user already have Admins):

{
  ...
  "Operations":[{"op":"add","value":[{"type":"Application1","display":"Employee access","value":"Employees"}],"path":"roles"}]
  ...
  "user":{
    ...
    "roles":[{"type":"Application1","display":"Employees access","value":"Employees"},
             {"type":"Application1","display":"Admins access","value":"Admins"}],
    ...
  }
}

We could then have one stream rule telling to include Application1

Subscriber or the SCIM Stream Gateway plugin then have logic for handling roles and will do a one-to-one match with endpoint groups using the role value content.

SCIM Stream Gateway have optional logic for converting roles to groups. Plugins existing group-logic can then be used instead of handling roles

SCIM Stream Gateway includes logic for exporting all target endpoint groups/accesses into Application roles that can be imported into Azure. This way we can ensure that Application roles in Azure reflects all corresponding accesses on endpoints and we get out-of-the-box management from Azure.

We could also use AAD Groups, having same group names defined in AAD as we have on the actual endpoint. There is also another way like prefixing AAD group with endpoint/application name e.g., App1_Employee

We could then have a SCIM Stream filtering rule like:
- filter: (user.groups.display -match "App1_.*")

Subscriber or the SCIM Stream Gateway plugin then have logic for removing “App1_” prefix from the group name and will do a one-to-one match towards endpoint having the same group name.

Configuration

SCIM Stream includes:

  • a binary: scim-stream
  • a configuration file: config.yaml

Below is an example of config.yaml configuration setup for collecting information from one AAD tenant every 5 second and streams to 5 different channels: App1, App2, App3, App4 and App5

azure:
- auth:
    client_id: <client-id>
    client_secret: <client-secret>
    tenant_id: <tenant-id>
  general:
    collect_interval: 5
    do_initial_load: false
    use_original_group_id: false
  streams:
  - channel: App1
    comment: Includes one application App1, using special attribute typeId that corresponds with Azure Application ID
    rules:
    - filter: (user.roles.typeId -match "App1-id")
  - channel: App2
    comment: Includes two groups
    rules:
    - filter: (user.groups.value -match "group1-id")
    - filter: (user.groups.value -match "group2-id")
  - channel: App3
    comment: Includes all groups having name starting with "App3" and Azure Application name "Application 1" with role value "Employees"
    rules:
    - filter: (user.groups.display -match "App3.*")
    - filter: (user.roles.Application 1.value -match "Employees")
  - channel: App4
    comment: Includes all users having email address suffix "@mycompany.com" and all users member of group name startig with App3 having user title Dog Trainer or Project Manager
    rules:
    - filter: (user.emails.work.value -match "*[email protected]")
    - filter: (user.groups.value -match "App4.*") and (user.title -match "(Dog Trainer|Project Manager)")
  - channel: App5
    comment: All included (all users/groups/applications)
    rules:
    - filter: (.*)
log:
  console: false
  level: debug
  max_age: 0
  max_backups: 10
  max_size: 10
stream_server:
  port: <port>
  publisher:
    user_private_key: <private-key>
    user_public_key: <public-key>
  subscriber:
    user_public_key: <public-key>
  certificate:
    certfile: cert.pem
    keyfile: key.pem
    cafile: ca.pem

Rulebased dynamic groups and roles for ABAC/RBAC functionality are supported:

  dynamics:
  - comment: ABAC/RBAC that apply 2 groups and 2 roles to all users having title containing "Consultant"
    dynamic_id: dynamic-1
    rules:
    - filter: (user.title -match ".*Consultant.*")
    groups:
    - display: Grp1-name
      id: Grp1-id
    - display: Grp2-name
      id: Grp2-id
    roles:
    - type: App1
      display: Admin access
      value: Admin
    - type: App1
      display: Employee access
      value: Employee
  streams:
  - channel: App1
    comment: Rules using the special dynamicId attribute instead of having several individual group/role filters for matching dynamic rules
    rules:
    - filter: (user.roles.dynamicId -match "dynamic-1")
    - filter: (user.groups.dynamicId -match "dynamic-1")

Configuration may also include a mapper section for mapping AAD attributes to SCIM attributes. Any mapper definition included will be merged with default mapper configuration that have following settings:

  mapper:
    group:
    - displayName:
        map_to: displayName
    user:
    - userPrincipalName:
        map_to: userName
    - accountEnabled:
        map_to: active
    - displayName:
        map_to: displayName
    - givenName:
        map_to: name.givenName
    - surname:
        map_to: name.familyName
    - Join(" ", [givenName], [surname]):
        map_to: name.formatted
    - mail:
        map_to: emails[type eq "work"].value
    - otherMails:
        map_to: emails[type eq "other"].value
    - businessPhones:
        map_to: phoneNumbers[type eq "work"].value
    - jobTitle:
        map_to: title
    - preferredLanguage:
        map_to: preferredLanguage
    - physicalDeliveryOfficeName:
        map_to: addresses[type eq "work"].formatted
    - streetAddress:
        map_to: addresses[type eq "work"].streetAddress
    - state:
        map_to: addresses[type eq "work"].region
    - city:
        map_to: addresses[type eq "work"].city
    - postalCode:
        map_to: addresses[type eq "work"].postalCode
    - country:
        map_to: addresses[type eq "work"].country
    - mobilePhone:
        map_to: phoneNumbers[type eq "mobile"].value
    - facsimileTelephoneNumber:
        map_to: phoneNumbers[type eq "fax"].value
    - mailNickname:
        map_to: externalId
    - companyName:
        map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:organization
    - employeeId:
        map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:employeeNumber
    - department:
        map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:department
    - manager:
        map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:manager
    - groups:
        map_to: groups
    - appRoles:
        map_to: roles

Note, following mapper settings cannot be overridden:

  mapper:
    group:
    - displayName:
        map_to: displayName
    user:
    - groups:
        map_to: groups
    - appRoles:
        map_to: roles

Note, the most important mapper configuration is the unique naming attribute - SCIM mandatory userName:

  mapper:
    user:
    - userPrincipalName:
        map_to: userName

Details

Config key Description
azure one or more azure authoriative collection sources
azure.auth authentication configuration
azure.auth.client_id azure application client id
azure.auth.client_secret azure application client secret, will be encrypted on startup
azure.auth.general general configuration
azure.auth.general.collect_interval collection interval in seconds, minimum and default is 5 seconds
azure.auth.general.do_initial_load true or false, default false. true will process all users in AAD according to rules, value automatically reverted back to false
azure.auth.general.use_original_group_id true or false, default false. true will set SCIM group-id to Azure group-id instead of using displayName as id
azure.dynamics optional, one or more rules for dynamic groups and roles giving ABAC/RBAC functionality
azure.dynamics.dynamicId self defined uniqe id for dynamic rule
azure.dynamics.rules one or more filter rules for dynamic assignments, several rules gives “OR” logic
azure.dynamics.rules.filter one or more advanced regular expression filters related to user/group/role (SCIM-formatted) e.g:
(user.title -match ".*Consultant.*")
azure.dynamics.groups one or more groups to be assigned when rules are fulfilled
azure.dynamics.groups.display group display name
azure.dynamics.groups.id group id
azure.dynamics.roles one or more scim roles to be assigned when rules are fulfilled
azure.dynamics.roles.type: role type, e.g: App1 - note for Azure Application roles the type corresponds to Azure Application name
azure.dynamics.roles.display display name that correspond to value e.g. “Admin access”
azure.dynamics.roles.value value e.g. “Admin”
azure.mapper optional, default Azure to SCIM attributes mapping is included. Configuration defined will override or extend defaults
azure.mapper.user one or more user mapping (group mapping cannot be overridden)
azure.mapper.user.aad-attribute aad-attribute is the name of the AAD attribute to be mapped, also supporting virtual attributes like Join(" ", [givenName], [surname])
azure.mapper.user.aad-attribute.map_to contains the SCIM attribute name like: userName, name.givenName, emails[type eq "work"].value, urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:organization
azure.streams azure streams configuration
azure.streams one or more azure streams configuration
azure.streams.channel mandatory channel name used by SCIM Stream publisher and corresponding client subscribers
azure.streams.comment optional
azure.streams.rules one or more rules for message publishing, several rules gives “OR” logic
azure.streams.rules.filter one or more advanced regular expression filters related to user/group/role (SCIM-formatted) e.g:
(user.groups.value -match "App1.*") and (user.title -match "(Dog Trainer|Project Manager)")
other filter examples:
(.*) includes all users/groups/applications
(user.name.givenName -match ".*enr.*")
(user.emails.work.value -match ".*@mycompany.com")
(user.addresses.work.postalCode -match "91608")
(user.groups.value -match "9936d283-2ac4-4c78-a8a0-52631fb20344")
(user.groups.display -match "App1")
(user.roles.typeId -match "a12f063b-1c8b-40e0-9988-01b46f478349")
(user.roles.Azure Application Test.value -match ".*")
(user.roles.Azure Application Test.value -match "role-value1")
log log configuration
log.console true or false, default false. true sends log to stdout instead of file
log.level debug, error, info or disabled
log.max_age number of days to retain old log files, default 0 - disabled
log.max_backups number of logfiles to keep, default 10
log.max_size max logfile size in Mega bytes before rollover to new file, default 10
streamserver NATS stream server configuration
streamserver.port port used by SCIM Stream publisher and client subscribers, default 9012
streamserver.publisher publisher configuration - private/public key (NATS nkey) generated by command scim-stream -user
streamserver.publisher.user_public_key user public key
streamserver.publisher.user_private_key user private key, will become encrypted on startup
streamserver.subscriber subscriber configuration - private/public key (NATS nkey) generated by command scim-stream -user
streamserver.subscriber.user_public_key user public key (the private key is used by subsciber)
streamserver.certificate optional, files must be located in ./config/certs. If not defiend, files will be autogenerated (ca.pem, cert.pem and key.pem)
streamserver.certificate.certfile name of certificate file e.g. cert.pem, file
streamserver.certificate.keyfile name of private key file e.g. key.pem
streamserver.certificate.cafile name of certificate authority file e.g. ca.pem, set to same as certfile if using self-signed certificate

Environments

Configuration may also be set using environments or external file.

Syntax is variable names like above dotted notation, but starting with SCIM-STREAM_, all uppercase, and underscore instead of dots. Attributes having underscore e.g. user_private_key must be used without underscore like USERPRIVATEKEY. Arrays like Azure using AZURE[index]

Example (linux using “export”, windows using “set”):

export SCIM-STREAM_AZURE[0]_AUTH_CLIENTSECRET=xxxx
export SCIM-STREAM_STREAMSERVER_PUBLISHER_USER_PRIVATEKEY=yyyy

Environments may also be defined in a vault file having only the initial vaultfile defined as environment

Example:

export SCIM-STREAM_VAULTFILE=/var/run/vault/.vaultfile

file "/var/run/vault/.vaultfile" content:
SCIM-STREAM_AZURE[0]_AUTH_CLIENTSECRET=xxxx
SCIM-STREAM_STREAMSERVER_PUBLISHER_USER_PRIVATEKEY=yyyy

Secrets like azure.auth.client_secret and streamserver.publisher.user_private_key defined in configuration file will automatically become encrypted on startup, and this file cannot be copied from one machine to another. Seed logic used for encryption may be overridden by your own seed defined as environment SEED e.g.:

export SEED=SomeRandomCharacters:-)

SCIM Stream Gateway

SCIM Stream Gateway is the next generation SCIM Gateway for message subscription and automated provisioning. This Gateway is the latest version of the SCIM Gateway, but not public available.

Having subscriber enabled and messages are received will result in automated provisioning through any existing plugin - everything out-of-the-box.

Configuration

SCIM Stream Gateway requires the following subscriber section to be added in the plugin configuration:

{
  "scimgateway": {
    ...
    "subscriber": {
      "enabled": true,
      "servers": ["nats://<scim-stream-host>:<port>"],
      "userPrivateKey": "<private-key>",
      "certificate": {
        "ca": "cert.pem"
      },
      "entity": {
        "undefined": {
          "subject": "AZURE.<channel>",
          "deleteUserOnLastGroupRoleRemoval": false,
          "convertRolesToGroups": false
        }
      }
    },
	...
  },
  "endpoint": {
    ...
      "entity": {
        "undefined": {
          ...
        }
      }
    ...
  }
}
  • subscriber - contains configuration needed for activating SCIM Stream subscription
  • subscriber.enabled - true/false, true enables subscription
  • subscriber.servers - array of one or more SCIM Stream servers to connect, syntax: ["nats://<scim-stream-host>:<port>"]. Using NATS streaming technology.
  • subscriber.userPrivateKey - User private key (NATS nkey). Will become encrypted on startup. scim-stream -user generates a user private/public key.
  • subscriber.certificate.ca - CA file name that have been copied from the SCIM Stream server and should be located in /config/certs directory
  • subscriber.entity.xxx - enity corresponds with url-baseEntity and may be used in the endpoint section for multi-tenant or multi-endpoint support. Entities defined in the subscriber sections will be linked to corresponding entities defined in the endpoint sections. If no entities are defined in the endpoint section, we use the default entity named “undefined”.
  • subscriber.entity.xxx.subject - Channel name for subscription. Azure channel name must be prefixed with “AZURE.” (uppercase and a dot) and include the channel name as defined at the SCIM Stream server - streams configuration
  • subscriber.entity.xxx.deleteUserOnLastGroupRoleRemoval - true/false, true will delete user when the last group/role become removed
  • subscriber.entity.xxx.convertRolesToGroups - true/false, true will convert roles to groups. Operations are converted and user.roles becomes user.groups having correct syntax. Existing plugins group-logic can then be used instead of building new logic related to roles.

Azure Application Roles

SCIM Stream Gateway may retrieve all groups/accesses from target endpoints and create a result having these converted into Azure Application Roles. Result provided includes appRoles content supported by the Azure Application Manifest-appRoles definition. We may copy/paste this appRoles result into Azure Application Manifest and start administring all our existing target endpoint groups/accesses in Azure through Application roles, everything out-of-the-box and unique uuid’s preserved on future run that might include new accesses definitions.

Azure Application Roles can be generated by using GET /AppRoles

Example using defult loki-plugin:

http://localhost:8880/AppRoles

After having updated the Azure Manifest with appRoles, all we have to do is configuring SCIM Stream to include the Azure Application ID to a stream channel rule:

  streams:
  - channel: App1
    comment: Includes one application App1
    rules:
    - application_id: <app1-id>

And configure SCIM Stream Gateway plugin to use that stream channel:

{
  "scimgateway": {
    ...
    "subscriber": {
      ...
      "entity": {
        "undefined": {
          "subject": "AZURE.App1",
          "convertRolesToGroups": true
          ...
        }
      }
    },
  ...
}

Note, by setting "convertRolesToGroups": true we can use plugins existing group-logic instead of using roles

License

© Jarle Elshaug