SCIM Stream

Author: Jarle Elshaug

Overview

SCIM Stream is the contemporary solution that redefines user provisioning by replacing traditional IGA top-down processing with cutting-edge message subscription capabilities. Clients can now subscribe to real-time messages, allowing for dynamic and automated provisioning, unlike anything you’ve experienced before.

SCIM Stream effortlessly gathers data from authoritative sources, formatting it into SCIM messages that are intelligently published to channels based on predefined rules. Endpoints and applications can then subscribe to relevant stream channels, ensuring swift and accurate message handling.

SCIM Stream includes SCIM Gateway that supports message subscription and automated provisioning, providing a robust foundation for your provisioning needs. In addition, gateway can become a publisher. In this mode, standard incoming SCIM requests from your Identity Provider (IdP) or API are directed and published to the stream. Subsequently, one of the gateways subscribing to the channel utilized by the publisher will manage the SCIM request, and response back.

Key takeaway: In cases where ingress or inbound traffic is restricted, provisioning may require specific measures. By deploying the SCIM Gateway subscriber within these restricted zones, we facilitate provisioning through egress/outbound communication.

Highlights:

  • Pub/Sub streaming using NATS having subscription-based provisioning initiated by client
  • Well-suited and tailored to meet all kind of needs and conditions; cloud, on-premises, at the edge and even on a Raspberry Pi or offline operations e.g., submarine
  • Yes, offline clients are supported, by default messages not processed by client remains in the stream for 14 days before purged
  • Multi Tenancy support for versatile usage
  • One binary and a configuration file, built using modern programming language, Go
  • Docker, Kubernetes and Unikernel-friendly
  • Security-first, only egress/outbound traffic initiated by the client ensuring the utmost communication security, leveraging JWT-based decentralized, zero trust principles that require both authentication and authorization
  • Currently Microsoft Entra ID and REST connected HR Systems are supported as authoritative collection sources, typically using Entra for IGA and HR for Joiner-Mover-Leaver
  • HR collector can be configured to use users most current employment when multiple entries exist, something that is a known limitation for other products
  • Replace traditional IGA tools like Microsoft Identity Manager (MIM), Okta, OneLogin, and more by using Entra ID Governance via SCIM Stream
  • Outperforms Entra ID standard provisioning; more flexibility and deliver faster results with minimal latency
  • Resolves Entra ID limitations of unmanaged read-only Active Directory synced groups, enabling effective group management of on-premises Active Directory from Entra ID
  • All changes to users and their group/application membership are processed
  • Using pre-defined Entra ID to SCIM attribute mapping, which can be extended and customized by configuration
  • Rules are configured per streaming channel (subscriber)
    Defined by one or more filter having advanced regular expression capabilites for all user/group/application attributes e.g.:
    - filter: (user.groups.display -match "App4.*") and (user.title -match "(Dog Trainer|Project Manager)")
  • Only process messages for users who meet the specified streaming rules criteria
  • User object in message only includes groups/applications according to rules, other groups/applications that user is member of are excluded
  • Application roles are mapped to SCIM roles giving final message like:
    {"roles":[{type":"Application1","display":"Employees access","value":"Employees"}]
  • Support nested groups, also linked to Application roles
  • Rule-based dynamic groups, roles, and profile attributes for advanced ABAC/RBAC functionality. E.g. gives dynamic groups and application roles when using the free version of Entra ID that do not include this functionality.
  • Supports initial load having all or filtered users in Entra ID or HR Systems processed according to both streaming and dynamic rules
  • Hot reload feature on configuration changes, eliminating the need for restarts
  • Option for sending e-mail on error
  • Builtin methods like GetUniqueValue(), Increment(), Normalize(), ElementNumber(), Join(), FirstN(), Replace(), LowerCase() and UpperCase(). These can be used in attribute mappings, typically when creating new users, unique value must be found and set for upn, samAccountName, e-mail,…
  • Includes SCIM Gateway
    • Existing gateway plugins can be used out-of-the-box, ensuring seamless integration with your infrastructure
    • Facilitates message subscription and automated provisioning
    • In publisher mode, standard incoming SCIM requests to the gateweay will be published to stream and one of the gateways subscribing to the channel will manage the request and response back. Note, since scimgateway also is an api-gateway that may handle all kind of request through the /api url, we now also have the possibility to stream whatever we want
    • Gives loadbalancing and failover by adding more gateways subscribing to same channel
    • Option for converting roles to groups, allowing the use of existing group-logic
    • All groups/roles at target endpoint can be imported to Entra ID as Application Roles for centralized management

Message format

SCIM Stream use NATS message technology having a SCIM formatted message that includes following:

  • General information: type of operation, when changed, who did the change etc
  • Operations: SCIM v2.0 Operations object that includes what have been changed
  • user: SCIM formatted user object including all user attributes according to attribute mapper and stream rules

SCIM formatted message below shows an example of a user that have been assigned the Entra ID group “Employee” (can be found in Operations and user.groups)

{
  "activityOperation": "modifyUser",
  "activityDateTime": "2022-07-01T10:27:55.403989Z",
  "initiatedById": "2b47d2fd-a98e-4051-b053-93d2da7f834e",
  "initiatedByUpn": "[email protected]",
  "targetId": "da33c8bd-4511-5058-b79e-7a45938b1b08",
  "targetUpn": "[email protected]",
  "Operations": [{
          "op": "add",
          "value": [{
                  "displayName": "Employees",
                  "id": "Employees"
              }
          ],
          "path": "groups"
      }
  ],
  "user": {
      "active": true,
      "addresses": [{
              "type": "work",
              "region": "CA",
              "city": "Hollywood",
              "postalCode": "91608",
              "streetAddress": "100 Universal City Plaza1",
              "country": "USA"
          }
      ],
      "displayName": "John Smith",
      "emails": [{
              "type": "other",
              "value": "[email protected]"
          }, {
              "type": "work",
              "value": "[email protected]"
          }
      ],
      "externalId": "johns",
      "groups": [{
              "display": "Employees",
              "value": "Employees"
          }, {
              "display": "Admins",
              "value": "Admins"
          }
      ],
      "name": {
          "givenName": "John",
          "familyName": "Smith",
          "formatted": "John Smith"
      },
      "phoneNumbers": [{
              "type": "mobile",
              "value": "555-555-6521"
          }, {
              "type": "work",
              "value": "555-555-1256"
          }
      ],
      "roles": [{
              "type": "Application1",
              "display": "User",
              "value": "User"
          }
      ],
      "title": "Consultant",
      "userName": "[email protected]",
      "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User": {
          "organization": "Universal Studios",
          "employeeNumber": "991999",
          "manager": {
              "value": "[email protected]",
              "displayName": "Barbara Jensen"
          },
          "department": "Tour Operations"
      },
      "schemas": ["urn:ietf:params:scim:schemas:core:2.0:User", "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User"]
  }
}

Collector - Entra ID

Users, groups and application access can be provisioned from Microsoft Entra ID to all types of endpoints/applications using SCIM Stream in combination with SCIM Gateway.

The most suitable approach of doing this would likely involve utilizing Entra ID Applications. Within the Application, roles can be established, each with a distinctive display and value. The crucial step lies in configuring the role value to correspond with the actual ID used for the respective group/access at the endpoint. Example SCIM message below shows that Application role value Employees have been added and user already have Admins:

{
  ...
  "Operations":[{"op":"add","value":[{"type":"Application1","display":"Employee access","value":"Employees"}],"path":"roles"}]
  ...
  "user":{
    ...
    "roles":[{"type":"Application1","display":"Employees access","value":"Employees"},
             {"type":"Application1","display":"Admins access","value":"Admins"}],
    ...
  }
}

We could then have one stream rule telling to include Application1 and all roles for this application e.g.,

rules:
  - filter: (user.roles.Application1.value -match ".*")

or even better, using the Application1 Enterprise Application Object ID:

rules:
  - filter: (user.roles.<Application1-EnterpriseAppObjectID>.value -match ".*")

Subscriber or the SCIM Gateway plugin then have logic for handling roles

Entra ID groups can be used. Groups can also be linked to an Application role using above mentioned role logic. Groups have display and value. Using standard group logic, we cannot define our own value. Instead SCIM Stream will set group value to the same value as display. We can optionally keep original group ID as value by configuration use_original_group_id.

To avoid having rules for all groups we want to include, we may prefix Entra ID groups e.g., App1_Employee and use the prefix in stream filtering rule:

rules:
  - filter: (user.groups.display -match "App1_.*")

We may have use-cases where we want to provision users based on group or role membership, but we do not want to include group/role in stream message. Reason could be we do not allow group/role provisioning, or it might not be supported by endpoint/plugin. Using -exclude in filter rule will skip actual groups/roles from message.

rules:
  - filter: (user.groups.value -match "dc474382-b3b6-4d7b-a6da-e9721c4ba150" -exclude)

One rule may have several filters defined giving OR-logic. E.g., we want all users having title=Consultant to be included, in addition we want members of any Application1 roles to be included and also include application role provisioning for those

rules:
  - filter: (user.title -match "Consultant")
  - filter: (user.roles.<Application1-EnterpriseAppObjectID>.value -match ".*")

For AND-logic we may use

rules:
  - filter: (user.title -match "Consultant") and (user.roles.<Application1-EnterpriseAppObjectID>.value -match ".*")

Note, when provisioning to Active Directory (AD), and AD is using Microsoft Entra Connect synchronization, we might have user and group updates in Entra that have been initiated by changes in AD. These changes do not need a loopback to AD, and we therefore may define an and filter-rule for excluding everything that have been initiated by the Entra connect Sync-user like shown below

rules:
  - filter: (user.roles.<Application1-EnterpriseAppObjectID>.value -match ".*") and (initiatedByUpn -match "(?!Sync_.*)")

Preferable strategy:

Use Entra ID Application roles instead of groups

SCIM Gateway includes logic for exporting all target endpoint groups/accesses into Application roles that can be imported to Entra. This way we can ensure that Application roles in Entra reflects all corresponding accesses on endpoints and we get out-of-the-box management from Entra.

SCIM Gateway have optional logic for converting roles to groups. Plugin existing group-logic can then be used instead of implementing new logic for handling roles

Collector - HR

HR collector is typically used for Joiner-Mover-Leaver purposes. When a new employee starts, we want user to be created on misc. endpoints/applications. Same when employee leave company, user should be disabled or removed. This logic is based on employee startDate and endDate, supporting most common date formats like: <unix-time-milli-seconds>, /Date(<unix-time-milli-seconds>)/, YYYY-MM-dd, YYYY/MM/dd, dd/MM/YYYYY. Collector is equipped with built-in logic to publish only those messages that have changed since the previous collection. For republishing all messages, we could startup with do_initial_load=true

HR collector is a general REST collector. If we need collection from non-REST e.g. direct SQL/LDAP/SOAP or complex logic is needed, we could use a SCIM Gateway in the middle providing the REST API.

Both collector types, hr and entra, use a common configuration structure. But, there are some additional configuration options needed for hr:

  • auth - there are several authentication options e.g., oauth_saml_assertion used for SAP SuccessFactors
  • collect_interval - using crontab expression instead of seconds e.g. 15 1 * * 1-5
  • collect_query_string - query used for collecting all users
  • collect_query_result_path - where users can be found in collected result
  • pre_activate_days - number of days we want users to be created/activated before startDate

In addition hr collector needs attribute mapping configuration, HR to SCIM:

  • attribute mapping based on attributes from collect_query_string result
  • SCIM attributes startDate and endDate must be included

HR collector includes an optional mapper rule configuration -most_current, e.g.,

mapper:
  rules:
    - filter: (employmentNav.results -most_current "startDate")

Above mapper rule will ensure all attribute mapper definitions with regards to employmentNav.results will use the most current employment in the list/array. User might have several employments and we want to use the most current one, excluding historical and future.

Stream rule for hr will typically include everything, all users e.g.,

rules:
  - filter: (.*)

Configuration

SCIM Stream includes:

  • a binary: scim-stream
  • a configuration file: ./config/config.yaml

Starting scim-stream will check for configuration file ./config/config.yaml. If not found, it will be created with default settings and we will be forced to startup with additional arguments for auto-generating missing nats configuration including entra/hr-collectors tenant name definition.

Below is an example configuration:
Using entra collector having one tenant mycompany defined
Collects from tenant mycompany every 5 second
Streams to 5 different channels: APP1, APP2, APP3, APP4 and APP5
Sending email on error

entra:
  mycompany:
    auth:
      oauth:
        clients:
          - client_id: <client-id>
            client_secret: <client-secret>
    general:
      collect_interval: 5
      do_initial_load_filter: null
      entra:
        include_guest_users: false
        tenant_id: <tenant-id>
    streams:
      - channel: APP1
        comment: Includes all roles for Enterprise Application App1
        rules:
          - filter: (user.roles.<App1-EnterpriseAppObjectID>.value -match ".*")
      - channel: APP2
        comment: Includes two groups
        rules:
          - filter: (user.groups.value -match "<group1-id>")
          - filter: (user.groups.value -match "<group2-id>")
      - channel: APP3
        comment: Includes all groups having name starting with "App3" and Enterprise Application name "Application 1" with role value "Employees"
        rules:
          - filter: (user.groups.display -match "App3.*")
          - filter: (user.roles.Application 1.value -match "Employees")
      - channel: APP4
        comment: Includes all users having email address suffix "@mycompany.com" or all users member of group name startig with App4 having user title Dog Trainer or Project Manager
        rules:
          - filter: (user.emails.work.value -match "*[email protected]")
          - filter: (user.groups.display -match "App4.*") and (user.title -match "(Dog Trainer|Project Manager)")
      - channel: APP5
        comment: All included (all users/groups/applications)
        rules:
          - filter: (.*)
    dynamics: null
    mapper: null
hr: null
log:
    console: false
    level: info
email:
    on_error:
        enabled: true
        to: [email protected],[email protected]
    smtp:
        host: smtp.office365.com
        port: 587
        username: [email protected]
        password: xxx
nats:
    <...>

Tenant configuration may include rule based logic for applying dynamic groups, roles and profile attributes giving ABAC/RBAC functionality:

dynamics:
- comment: ABAC/RBAC matching title Consultant and will update user with 2 groups, 2 roles including some profile attributes
  dynamic_id: dynamic-1
  rules:
  - filter: (user.title -match ".*Consultant.*")
  assignments:
    user.groups:
    - display: Grp1-name
      value: Grp1-id
    - display: Grp2-name
      value: Grp2-id
    user.roles:
    - display: Admin access
      value: Admin
      type: App1
    - display: Employee access
      value: Employee
      type: App1
    user.addresses:
    - postalCode: 1234
      streetAddress: Back Office Street
      type: work
    user.type: External

Tenant configuration may also include a mapper section for mapping collector attributes (entra/hr) to SCIM attributes. Mapper configuration is mandatory for HR. For Entra any mapper definition included will be merged with default Entra mapper configuration that have following settings:

  mapper:
    group:
    - displayName:
        map_to: displayName
    user:
    - userPrincipalName:
        map_to: userName
    - accountEnabled:
        map_to: active
    - displayName:
        map_to: displayName
    - givenName:
        map_to: name.givenName
    - surname:
        map_to: name.familyName
    - Join([name.givenName], " ", [name.familyName]):
        map_to: name.formatted
    - mail:
        map_to: emails[type eq "work"].value
    - otherMails:
        map_to: emails[type eq "other"].value
    - businessPhones:
        map_to: phoneNumbers[type eq "work"].value
    - jobTitle:
        map_to: title
    - preferredLanguage:
        map_to: preferredLanguage
    - physicalDeliveryOfficeName:
        map_to: addresses[type eq "work"].formatted
    - streetAddress:
        map_to: addresses[type eq "work"].streetAddress
    - state:
        map_to: addresses[type eq "work"].region
    - city:
        map_to: addresses[type eq "work"].city
    - postalCode:
        map_to: addresses[type eq "work"].postalCode
    - country:
        map_to: addresses[type eq "work"].country
    - mobilePhone:
        map_to: phoneNumbers[type eq "mobile"].value
    - faxNumber:
        map_to: phoneNumbers[type eq "fax"].value
    - mailNickname:
        map_to: externalId
    - employeeHireDate:
        map_to: startDate
    - companyName:
        map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:organization
    - employeeId:
        map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:employeeNumber
    - employeeType:
        map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:employeeType
    - department:
        map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:department
    - employeeOrgData.costCenter:
        map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:costCenter
    - employeeOrgData.division:
        map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:division
    - manager.userPrincipalName:
        map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:manager.value
    - manager.displayName:
        map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:manager.displayName
    - groups:
        map_to: groups
    - appRoles:
        map_to: roles

Note, following mapper settings cannot be overridden:

  mapper:
    group:
    - displayName:
        map_to: displayName
    user:
    - groups:
        map_to: groups
    - appRoles:
        map_to: roles

Note, the most important mapper configuration is the unique naming attribute - SCIM mandatory userName:

  mapper:
    user:
    - userPrincipalName:
        map_to: userName

For hr we must specify baseUrls and type of authentication. Configuration includes a section general.hr that is unique for the hr collector, like we have general.entra that is unique for entra. We also need to specify other details like collect query, result path and mapper. Using crontab expression for scheduled collector intervals. Mapper definintion like map_to: onCreate.<attribute> for create user only mapping, typically used in combination with built-in functions like shown in SAP SuccessFactors example below

Note, using explicite yaml style that includes line-prefix ?key :value on complex syntax like GetUniqueValue()

Additionally, it’s important to highlight that there’s no requirement to utilize built-in mapper functions if SCIM Gateway plugin already incorporates the necessary logic

hr:
  mycompany:
    auth:
      base_urls:
        - https://api2.successfactors.eu
      oauth_saml_assertion:
        client_id: <...>
        company_id: <...>
        name_id: <...>
        key_file: sf-key.pem
        token_url: https://api2.successfactors.eu/oauth/token
    general:
      collect_interval: 15 1 * * 1-5
      do_initial_load: false
      hr:
        pre_activate_days: 0
        collect_query_result_path: d.results
        collect_query_string: /odata/v2/PerPerson?$count=true&customPageSize=100&$format=json&$expand=employmentNav/userNav,employmentNav/jobInfoNav,personalInfoNav,personEmpTerminationInfoNav,phoneNav,emailNav,employmentNav/jobInfoNav/companyNav/countryOfRegistrationNav,employmentNav/jobInfoNav/divisionNav,employmentNav/jobInfoNav/departmentNav&$filter=employmentNav/startDate ne null
    streams:
      - channel: SF
        comment: All included
        rules:
          - filter: (.*)
    dynamics: null
    mapper:
      rules:
        - filter: (employmentNav.results -most_current "startDate")
      user:
        - personIdExternal:
            map_to: userName
        - ? GetUniqueValue(Normalize(Join(LowerCase(Replace([name.givenName]," ",".")),".",LowerCase(Replace([name.familyName]," ",".")),Increment("01",false),"@mycompany.com")))
          : map_to: onCreate.userPrincipalName
        - ? GetUniqueValue(Normalize(Join(FirstN(LowerCase([name.familyName]),3),FirstN(LowerCase([name.givenName]),2),Increment("01",true))))
          : map_to: onCreate.sAMAccountName
        - employmentNav.results.[0].startDate:
            map_to: startDate
        - employmentNav.results.[0].endDate:
            map_to: endDate
        - personalInfoNav.results.[0].firstName:
            map_to: name.givenName
        - personalInfoNav.results.[0].lastName:
            map_to: name.familyName
        - Join([name.givenName], " ", [name.familyName]):
            map_to: name.formatted
        - phoneNav.results.[{"isPrimary":true}].phoneNumber:
            map_to: phoneNumbers[type eq "mobile"].value
        - employmentNav.results.[0].userNav.city:
            map_to: addresses[type eq "work"].city
        - employmentNav.results.[0].userNav.country:
            map_to: addresses[type eq "work"].country
        - employmentNav.results.[0].userNav.state:
            map_to: addresses[type eq "work"].state
        - employmentNav.results.[0].userNav.zipCode:
            map_to: addresses[type eq "work"].postalCode
        - employmentNav.results.[0].userNav.addressLine1:
            map_to: addresses[type eq "work"].streetAddress
        - employmentNav.results.[0].jobInfoNav.results.[0].jobTitle:
            map_to: title
        - employmentNav.results.[0].jobInfoNav.results.[0].managerId:
            map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User.manager.value
        - employmentNav.results.[0].jobInfoNav.results.[0].department:
            map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User.department
        - employmentNav.results.[0].jobInfoNav.results.[0].division:
            map_to: urn:ietf:params:scim:schemas:extension:enterprise:2.0:User.division

Details

Configuration file ./config/config.yaml, encompasses the following primary objects:

  • entra - collectors for Entra ID
  • hr - collectors for REST based HR systems
  • log - logging
  • email - sending email onerror
  • nats - builtin NATS streaming server
Config key Description
[entra][hr] collector type, both entra and hr are supported. Typically using entra collector for IGA and hr for joiner-mover-leaver
[entra][hr].tenant tenant is self-defined tenant name, aka mycompany. This is the unique name of the specific collector type configuration. We may have one or more tenants configured for each type of collector. Name must correspond with nats.publisher.tenant and nats.server.auth.account.tenant. Note, tenant names are defined and automatically updated during initial startup/configuration procedure
[entra][hr].tenant.auth Entra using OAuth, HR may have other requirements, following are supported:
OAuth
OAuthSamlAssertion
BearerToken
Basic Auth
General Headers
hr.tenant.auth.base_urls one or more (failover) endpoint urls. Not applicable for Entra that will use hardcoded url based on entra.tenant.general.entra.tenant_id
[entra][hr].tenant.auth.oauth oauth authentication configuration
[entra][hr].tenant.auth.oauth.clients array of one or more oauth clients. For Entra, auth client refer to Entra Application that require following api permissions defined: AuditLog.Read.All + Directory.Read.All + User.Read.All. Multiple clients can be defined to alleviate throttling or rate-limit constraints
[entra][hr].tenant.auth.oauth.clients.client_id client id, for Entra this is application client id
[entra][hr].tenant.auth.oauth.clients.client_secret client secret, will be encrypted on startup - for Entra this is application client secret,
hr.tenant.auth.oauth_saml_assertion OAuth SamlAssertion authentication, not applicable for Entra
hr.tenant.auth.oauth_saml_assertion.audience default “scim-stream”, value used to tag the SAML assertion
hr.tenant.auth.oauth_saml_assertion.key_file file name, file located in ./config/certs - containing the private key of the key pair used to sign the SAML assertion
hr.tenant.auth.oauth_saml_assertion.cert_file file name, file located in ./config/certs - containing public certificate corresponding to the key
hr.tenant.auth.oauth_saml_assertion.client_id client id, for SAP SuccessFactors this is also called as API key
hr.tenant.auth.oauth_saml_assertion.company_id custom SAML attribute used by SAP SuccessFactors representing id of company
hr.tenant.auth.oauth_saml_assertion.issuer default “scim-stream”, assertion is issued by scim-stream
hr.tenant.auth.oauth_saml_assertion.name_id user identifier to be used to access endpoint
hr.tenant.auth.oauth_saml_assertion.token_url the URL of the API server from which you request the OAuth token e.g., https://api2.successfactors.eu/oauth/token
hr.tenant.auth.oauth_saml_assertion.user_identifier_format default userName, it can be one of the three following types: userName, userId or email
hr.tenant.auth.oauth_saml_assertion.life_time default 3600, lifetime of the SAML Assertion in seconds
hr.tenant.auth.basic basic authentication, not applicable for Entra
hr.tenant.auth.basic.username username used for authentication
hr.tenant.auth.basic.password password used for authentication, will be encrypted on startup
hr.tenant.auth.bearer_token bearer authentication, not applicable for Entra
hr.tenant.auth.bearer_token.token token used for authentication, will be encrypted on startup
hr.tenant.auth.header optional headers to be included, not applicable for Entra
hr.tenant.auth.header.xxx xxx is the name of header and configuration defines the value to be set
[entra][hr].tenant.general general configuration
[entra][hr].tenant.general.collect_interval collect interval in seconds, minimum and default is 5. Also supporting crontab scheduler expression <min> <hour> <day> <month> <weekday> e.g., 15 2 * * 1-5 will run 02:15 Monday-Friday. Crontab expression typically used by HR collector
[entra][hr].tenant.general.do_initial_load optional, true or false, default false. true will collect and process all users in Entra/HR during startup or hot reload on configuration changes. Value true will automatically be reverted back to false
entra.tenant.general.do_initial_load_filter optional and preferred way of initial load for Entra
Same as do_initial_load but instead using a single regular expression filter, like stream rules filter, for collect and process users. Filter will automatically be reverted back to blank/null after processing
examples:
do_initial_load_filter: (user.roles.25a131d0-af63-443b-976f-ac0076734627.value -match ".*")
do_initial_load_filter: (user.roles.App1.value -match ".*")
do_initial_load_filter: (user.groups.value -match "dc384382-b3b6-4d6b-a6da-e9801c4ba150")
do_initial_load_filter: (user.groups.value -match "(dc384382-b3b6-4d6b-a6da-e9801c4ba150|5f2a9c65-5d65-4e66-a9d6-379dd92974ca)")
do_initial_load_filter: (user.groups.display -match "Group1")
do_initial_load_filter: (user.userName -match "[email protected]")
do_initial_load_filter: (user.title -match "Consultant")
do_initial_load_filter: (user.emails.work.value -match "*[email protected]")

Note: Regular expressions .* and | are supported, but there are some limitations for intial load filter using .*:
"xxx.*" - startsWith is OK, but not for user.groups
".*xxx" - endsWith is OK only for attributes user.userName (userPrincipalName), user.emails.work.value (mail) and user.emails.other.value (otherMails)
".*xxx.*" - contains is not supported
[entra][hr].tenant.general.request_timeout default 90 seconds, when timeout occures we will see a context deadline exceeded message in error log
entra.tenant.general.entra Entra specific settings
entra.tenant.general.entra.tenant_id Entra ID tenant ID (GUID) or domain name
entra.hr.tenant.general.entra.include_guest_users default false, true will include processing of guest users
entra.tenant.general.entra.skip_upn_startswith optional, array of upn prefix, users having upn prefix will not be processed e.g. svc will not process upn starting with svc
entra.tenant.general.entra.use_original_group_id default false, true will use Entra group id instead of group displayName as SCIM id
hr.tenant.general.hr HR specific settings
hr.tenant.general.hr.collect_query_string REST query used for collecting all users and needed information. Must correspond with hr mapper definitions
hr.tenant.general.hr.collect_query_result_path path relative to the result telling where users can be found (array/list). For SAP SuccessFacters path will be d.results. If path not defined, result should be root array/list of users
hr.tenant.general.hr.pre_activate_days default 0, number of days we want users to be created/activated before startDate
[entra][hr].tenant.dynamics optional, one or more rules for dynamic groups, roles and profile attributes giving ABAC/RBAC functionality
[entra][hr].tenant.dynamics.dynamicId self-defined uniqe id for dynamic rule - this id is also used in stream rules for including dynamic users e.g., (user.dynamicIds -match "dynamic-1")
[entra][hr].tenant.dynamics.rules one or more filter rules for dynamic assignments, several rules gives “OR” logic
[entra][hr].tenant.dynamics.rules.filter advanced regular expression supporting the -match operator related to user/group/role (SCIM-formatted) e.g:
(user.title -match ".*Consultant.*")
[entra][hr].tenant.dynamics.assignments contains assignments definitions on what will be applied when rule(s) are fulfilled and also will be revoked when previous have been fulfilled. Note, assignment key always relates to “user.attribute” e.g. user.group, user.roles, user.addresses, user.name.givenName, …
[entra][hr].tenant.dynamics.assignments.user.groups one or more groups to be assigned
[entra][hr].tenant.dynamics.assignments.user.groups.display group display name
[entra][hr].tenant.dynamics.assignments.user.groups.value group value (id)
[entra][hr].tenant.dynamics.assignments.user.roles one or more scim roles to be assigned
[entra][hr].tenant.dynamics.assignments.user.roles.type role type, e.g: App1 - note for Entra Application roles the type corresponds to Entra Application name
[entra][hr].tenant.dynamics.assignments.user.roles.display display name that correspond to value e.g. “Admin access”
[entra][hr].tenant.dynamics.assignments.user.roles.value value e.g. “Admin”
[entra][hr].tenant.dynamics.assignments.user.xxx all user attrbutes may be used e.g. user.addresses, user.emails, user.title, user.name.givenName, …
[entra][hr].tenant.streams streams configuration
[entra][hr].tenant.streams one or more entra streams configuration
[entra][hr].tenant.streams.channel mandatory channel name used by SCIM Stream publisher. The dot-character . may be used to create a subject hierarchy e.g,:
AD
AD.OPERATOR
AD.USER
AD.USER.APP1
Subscriber, the SCIM Gateway plugin, subscribes to channel(s).
Note, SCIM Stream will automatically prefix channel with collector type ENTRA or HR. Subscriber then use subject (channel) that must include this prefix e.g. HR.SF, ENTRA.AD, ENTRA.AD.USER.APP1
Wildcards may also be used for subscribing to a hierarchy
ENTRA.> includes all ENTRA channels/subjects, ENTRA.AD.* includes single subjects under ENTRA.AD e.g. ENTRA.AD.OPERATOR and ENTRA.AD.USER, but not ENTRA.AD.USER.APP1
[entra][hr].tenant.streams.comment optional
[entra][hr].tenant.streams.rules one or more filter rules for message publishing, several rules gives “OR” logic
[entra][hr].tenant.streams.rules.filter advanced regular expression supporting the -match operator related to user/group/role (SCIM-formatted) e.g:
(user.groups.display -match "App1.*") and (user.title -match "(Dog Trainer|Project Manager)")
other filter examples:
(.*) includes all users/groups/applications
(user.name.givenName -match ".*enr.*")
(user.emails.work.value -match ".*@tenant.com")
(user.addresses.work.postalCode -match "91608")
(user.groups.value -match "9936d283-2ac4-4c78-a8a0-52631fb20344")
(user.groups.value -match "9936d283-2ac4-4c78-a8a0-52631fb20344" -exclude)
(user.groups.display -match "App1")
(user.roles.a12f063b-1c8b-40e0-9988-01b46f478349.value -match ".*")
(user.roles.Entra Application Test.value -match ".*")
(user.roles.Entra Application Test.value -match "role-value1")
(user.dynamicIds -match "dynamic-1")
[entra][hr].tenant.mapper optional for Entra, default Entra to SCIM attributes mapping is included. Configuration defined will override or extend defaults. For HR, mapper is mandatory
[entra][hr].tenant.mapper.user one or more user mapping (group mapping cannot be overridden)
[entra][hr].tenant.mapper.user.attr attr is the name of the Entra ID or HR attribute to be mapped, also supporting virtual attributes defined as functions e.g., Join([name.givenName], " ", [name.familyName]).

Following functions are supported:
LowerCase(arg)
UpperCase(arg)
Join(arg1,arg2,…)
Replace(arg, org, new)
Normalize(arg)
FirstN(arg, number)
ElementNumber(arg, seperator, number)
Increment(number, true/false)
GetUniqueValue(arg)

Functions can be nested.
Below is an example of defining a unique upn; removing spaces from givenName and familyName, Join them both with “.”, normalize for replacing country special chracters e.g. å=>a, also join with Increment() that will add number specified and will be increased on next try when combined with GetUniqueValue() - using Increment(xx, true) will include xx on first GetUniqueValue attempt

GetUniqueValue(Normalize(Join(LowerCase(Replace([name.givenName]," “,”.")),".",LowerCase(Replace([name.familyName]," “,”.")),Increment(“01”,false),"@mycompany.com")))

Note, function like GetUniqueValue() could be used only when creating user. We then set a custom onCreate prefix in the map_to definitions e.g.

- GetUniqueValue(…)
map_to: onCreate.userPrincipalName
[entra][hr].tenant.mapper.user.attr.map_to defines the SCIM attribute name e.g.: userName, name.givenName, emails[type eq "work"].value, urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:organization

for hr startDate and endDate must be included
hr.tenant.mapper.rules only applies to hr
hr.tenant.mapper.rules.filter supporting only one filter rule -most_current e.g.,

- filter: (employmentNav.results -most_current “startDate”)

purpose of this rule is to ensure all attribute mapper definitions related to employmentNav.results will use the most current employment in the list/array, a user might have several employments and we want to use the most current, excluding historical and future. startDate referes to SCIM startDate attribute that must be included in attribure mappings
log log configuration
log.console true or false, default false. true logs to stdout instead of file
log.level debug, error, info or disabled
log.max_age number of days to retain old log files, default 0 - disabled
log.max_backups number of logfiles to keep, default 10
log.max_size max logfile size in Mega bytes before rollover to new file, default 10
email email configuration
email.smtp.host mailserver, e.g. smtp.office365.com
email.smtp.port port used by mailserver e.g. 587, 25 or 465
email.smtp.username mail account username for authentication and also the sender of the email, e.g. [email protected]
email.smtp.password mail account password, will be automatically encrypted on startup. If not defined, user/password authentication not in use
email.smtp.proxy optional, mailproxy e.g. http://proxy-host:1234
email.smtp.proxy_username proxy authentication username
email.smtp.proxy.proxy_password proxy authentication password, will be automatically encrypted on startup
email.on_error sending email notification when error occure
email.on_error.enabled true or false, true will enable email onerror notifications
email.on_error.send_interval default 15, email notifications are deferred until send_interval minutes have passed since the last notification
email.on_error.subject default SCIM Stream error - autogenerated email
email.on_error.to comma separated list of recipients email addresses e.g: [email protected]
nats NATS streaming configuration
All configurations are automated through scim-stream startup arguments
tenant names must correspond with entra.tenant and hr.tenant names
See Downloads for details
nats.server NATS server configuration
SCIM Stream have builtin NATS server, but also support using external
nats.server.port default 9012, port used by server
nats.server.trace true or false, default false
nats.server.encryption_at_rest_key optional, secret for data encryption at rest; NATS internal message storage will be encrypted. Provided clear text secret will become encrypted on startup
nats.server.use_external_server true or false, default false. If true, external NATS server will be used and nats.publisher.urls must be using external server. External server should also be using an exported configuration, see Downloads for details
nats.server.certificate optional, files will be autogenerated if not defined. File name defined must be located in ./config/certs
nats.server.certificate.certfile certificate file name, default autogenerated cert.pem
nats.server.certificate.keyfile private key file name, default autogenerated key.pem
nats.server.certificate.cafile CA file name, default autogenerated ca.pem. Note, subscribers needs a copy of this file
nats.server.auth NATS server authentication configuration
nats.server.auth.operator NATS operator configuration
nats.server.auth.operator.jwt operator JSON Web Token
nats.server.auth.operator.secret operator secret for jwt
nats.server.auth.operator.sys_account_jwt NATS system account jwt
nats.server.auth.operator.sys_account_secret NATS system account secret
nats.server.auth.account NATS Account configuration
nats.server.auth.account.tenant tenant is self-defined tenant name, aka mycompany
nats.server.auth.account.tenant.jwt tenant JSON Web Token
nats.server.auth.account.tenant.secret tenant secret for jwt
nats.server.auth.account.tenant.signing signing for pub/sub auth
nats.server.auth.account.tenant.signing.pub_secret secret for signing tenant SCIM Stream publisher
nats.server.auth.account.tenant.signing.sub_entra_secret secret for signing tenant entra subscribers
nats.server.auth.account.tenant.signing.sub_hr_secret secret for signing tenant hr subscribers
nats.publisher publisher configuration, SCIM-Stream is the only publisher of messages
nats.publisher.tenant tenant is self-defined tenant name, aka mycompany
nats.publisher.tenant.jwt publisher JSON Web Token
nats.publisher.tenant.secret publisher secret for signing jwt
nats.publisher.tenant.urls default nats://127.0.0.1:9012using builtin internal NATS server, one or more nats urls (cluster servers), external NATS server could be used
nats.publisher.tenant.max_age default 14, number of days messages will remain in the stream before purged. 14 means subscriber can be maximum 14 days offline before starting loosing messages

Environments

Configuration may also be set using environments or external file.

Syntax is variable names like above dotted notation, but starting with SCIM-STREAM_, all uppercase, and underscore instead of dots. Attributes having underscore e.g. user_private_key must be used without underscore like USERPRIVATEKEY. Arrays like Entra using CLIENTS[index]

Example 1:

entra:
  mycompany:
    oauth:
      clients:
        - client_secret: my-client-secret

Becomes:

export SCIM-STREAM_ENTRA_MYCOMPANY_OAUTH_CLIENTS[0]_CLIENTSECRET=my-client-secret

Example 2:

nats:
  publisher:
    mycompany:
      secret: generated-secret

Becomes:

export SCIM-STREAM_NATS_PUBLISHER_MYCOMPANY_SECRET=generated-secret

An external vault file may also be defined as environment. All configuration environments may be then be included in this file.

Example:

export SCIM-STREAM_VAULTFILE=/var/run/vault/.vaultfile

file "/var/run/vault/.vaultfile" having content:
SCIM-STREAM_ENTRA_MYCOMPANY_OAUTH_CLIENTS[0]_CLIENTSECRET=my-client-secret
SCIM-STREAM_NATS_PUBLISHER_MYCOMPANY_SECRET=generated-secret

Secrets defined in configuration file will automatically become encrypted on startup. Encrypted configuration file cannot be copied from one machine to another unless all secrets are reverted back to clear text. To overcome this, seed logic used for encryption may be overridden by your own seed defined as environment SEED e.g.:

export SEED=SomeRandomCharacters:-)

SCIM Gateway

SCIM Gateway offers enhanced functionality with support for message subscription. Enabling one or more subscribers initiates seamless automated provisioning through your plugins

Configuration

SCIM Gateway requires a stream configuration section to be found in the plugin configuration file. Since streaming do not have any url and corresponding baseEntity, the baseEntity have to be configured at the subscriber: scimgateway.stream.subscriber.entity.xxx which normally corresponds with endpoint.entity.xxx. If baseEntity for multi tenancy/endpoint support not being used, we define baseEntity as undefined: scimgateway.stream.subscriber.entity.undefined.

Below is subscriber configuration example:

{
  "scimgateway": {
    "port": 0,
    "scim": {
      "usePutSoftSync": true,
      ...
    }
    ...
    "stream": {
      "baseUrls": ["nats://<scim-stream-host>:9012"],
      "certificate": {
        "ca": "ca.pem"
      },
      "subscriber": {
        "enabled": true,
        "entity": {
          "undefined": {
            "nats": {
              "tenant": "mycompany"
              "subject": "ENTRA.APP1",
              "jwt": "<...>",
              "secret": <...>,
            },
            "deleteUserOnLastGroupRoleRemoval": false,
            "skipConvertRolesToGroups": false,
            "generateUserPassword": false,
            "modifyOnly": false,
            "replaceDomains": [{
              "from": "@mycompany.onmicrosoft.com",
              "to": "@mycompany.com"
            }]
          }
        }
      },
      "publisher": null,
    },
	...
  },
  "endpoint": {
    ...
      "entity": {
        "undefined": {
          ...
        }
      }
    ...
  }
}

Below is gateway publisher configuration example: Note, subscribers must use same channel (subject) GW.APPX as publisher and they must also have a valid jwt/secret for tenant being used.

{
  "scimgateway": {
    "port": 8880,
    ...
    "stream": {
      "baseUrls": ["nats://<scim-stream-host>:9012"],
      "certificate": {
        "ca": "ca.pem"
      },
      "subscriber": null,
      "publisher": {
        "enabled": true,
        "entity": {
          "undefined": {
            "nats": {
              "tenant": "mycompany"
              "subject": "GW.APPX",
              "jwt": "<...>",
              "secret": <...>
            },
          "company1": {
            "nats": {
              "tenant": "company1"
              "subject": "GW.APPX",
              "jwt": "<...>",
              "secret": <...>
            }
          }
        }
      }
    },
	...
  },
  "endpoint": {
    <not used by publisher>
  }
}
  • stream - stream configuration for pub/sub - using streaming we only allow egress/outbound traffic
  • stream.publisher - publisher configuration using GW prefixed subjects - one gateway may have several publishers for multi tenancy/endpoint. When enabled, standard incoming SCIM requests will routed and published to stream, and one of the gateways subscribing to channel used by publisher will handle the SCIM request and response back to publisher. Also note when gateway is running in publisher mode, plugin methods (getUsers, modifyUser, getApi,…) are not in use - everything will be handled by the subscriber gateway
  • stream.subscriber - subscriber configuration - one gateway may have several subscribers for multi tenancy/endpoint. For loadbalancing and failover, we may have several gateways running same plugins with same subscriber subject, and only one of them will be processing stream message.
  • port - note, setting port to 0 will prevent standard inbound listener to be started, not mandatory but optional
  • scim.usePutSoftSync - must be set to true when subscribing to ENTRA or HR prefixed subjects which is published by SCIM Stream and not the gateway (gateway is publishing using GW prefixed subjects). There will be an startup error if not set to true in case of ENTRA/HR
  • stream.baseUrls - one or more nats connection urls (cluster servers) e.g. ["nats://<scim-stream-host>:9012"]
  • stream.certificate - certificate configuration
  • stream.certificate.ca - CA filename or absolute path to the file that have been copied from the SCIM Stream server ref. nats.server.certificate.cafile. This file should be located in ./config/certs unless absolute path being used
  • stream.subscriber/publisher - subscriber/publisher configuration
  • stream.subscriber/publisher.enabled - true/false, true enables subscriber/publisher
  • stream.subscriber/publisher.entity - entity definition
  • stream.subscriber/publisher.entity.xxx - xxx is baseEntity, set to undefiend means baseEntity not applicable. subscriber.entity.xxx must correspond with endpoint.entity.xxx if endpoint baseEntity being used. This ensures message subcriptions will be linked to correct baseEntity for multi tenancy/endpoints support. For gateway as publisher, baseEntity will be included in message and used by subscriber.
  • stream.subscriber/publisher.entity.xxx.nats - nats configuration for entity
  • stream.subscriber/publisher.entity.xxx.nats.tenant - tenant name aka mycompany, must correspond with SCIM Stream tenant name. Not used by publisher, but should be included for information purpose
  • stream.subscriber/publisher.entity.xxx.nats.subject - syntax: <collector-type>.<channel> e.g. ENTRA.APP1, ENTRA.AD, HR.SF. Subject is the stream channel having collector type prefix and is case sensitive. <collector-type> is either ENTRA, HR or GW. <channel> corresponds with SCIM Stream tenant.streams.channel configuration or the GW type channel used by gateway publisher. The dot-character . is used for subject hierarchy, see SCIM Stream [entra][hr].tenant.streams.channel description. Subscriber is only authorized for subscribing to the tenant and collector-type (account) that have signed the jwt and generated the secret used by subscriber. Not allowing cross tenant nor cross ENTRA/HR/GW subscriptions.
  • stream.subscriber/publisher.entity.xxx.nats.jwt - user JSON Webtoken (JWT) generated by SCIM Stream NATS. JWT is linked to a specific NATS account/tenant allowing subscription to either ENTRA, HR or GW messages for that tenant. JWT will be signed by user secret with final verification/authorization by SCIM Stream NATS server.
  • stream.subscriber/publisher.entity.xxx.nats.secret - user secret generated by SCIM Stream NATS for signing jwt, will become encrypted on startup
  • stream.subscriber.entity.xxx.deleteUserOnLastGroupRoleRemoval - true/false, default false - true will delete user when the last group/role become removed
  • stream.subscriber.entity.xxx.skipConvertRolesToGroups - true/false, default false - true keeps roles as-is
  • stream.subscriber.entity.xxx.generateUserPassword - true/false, default false - true will generate a random user password on createUser operation
  • stream.subscriber.entity.xxx.modifyOnly - true/false, default false - true will only allow modifyUser/modifyGroup and ignore createUser/deleteUser E.g., we do not want Entra messages to create users in AD when having HR messages for that purpose.
  • stream.subscriber.entity.xxx.replaceDomains - optional, array of upn/email domains to be replaced (from/to objects)
  • stream.subscriber.entity.xxx.replaceDomains[].from - domain name to be replaced, must start with “@” e.g. “@mycompany.onmicrosoft.com”
  • stream.subscriber.entity.xxx.replaceDomains[].to - new domain name, must start with “@” e.g. “@mycompany.com”

Entra Application Roles

SCIM Gateway may retrieve all groups/accesses from target endpoints and create a result having these converted into Entra Application Roles. Result provided includes appRoles content supported by the Entra Application Manifest-appRoles definition. We may copy/paste this appRoles result into Entra Application Manifest and start administring all our existing target endpoint groups/accesses in Entra through Application roles, everything out-of-the-box and unique uuid’s preserved on future run that might include new accesses definitions.

Entra Application Roles can be generated by using GET /AppRoles

Example using defult loki-plugin:

http://localhost:8880/AppRoles

After having updated Entra Application Manifest appRoles[…] content, all we must do is configuring SCIM Stream to include this Entra Enterprise Application ObjectID to a stream channel rule:

  streams:
  - channel: App1
    comment: Includes one application App1
    rules:
      - filter: (user.roles.<App1-EnterpriseAppObjectID>.value -match ".*")

And configure SCIM Gateway plugin to use this Entra collector stream channel ENTRA.App1:

{
  "scimgateway": {
    ...
    "subscriber": {
      ...
      "entity": {
        "undefined": {
          "nats": {
            "subject": "ENTRA.App1",
            ...
          },
          "skipConvertRolesToGroups": false,
          ...
        }
      }
    },
  ...
}

Note, by setting "skipConvertRolesToGroups: true, roles will be as-is and plugin needs logic for handling roles. This can also be tested using default plugin-loki

License

© Jarle Elshaug

Downloads are exclusively available for licensed clients

Downloads