DataSilos 3.8.0

Welcome to DataSilos - created and maintained by Sascha Wildgrube.

The DataSilos application helps to separate data between different organizational units - so called data silos.

In contrast to domain separation DataSilos allows the exchange of data (in form of individual records) between data silos and makes them visible to users on a need-to-know basis.

In addition to the organizational dimension, access to records can also be controlled based on security clearance levels. Each record and each user has a security clearance level. A user can only access a record if their security clearance level is equal or higher than the record's security clearance level.

Disclaimer

DataSilos is NOT an officially supported ServiceNow product.

DataSilos does NOT come with any kind of warranty. If you use it, you own it!

Use DataSilos if

  • Domain Separation is too strict
  • Domain Separation is already implemented for a different purpose and more detailed visibility rules must be enforced WITHIN a domain
  • There is a certain level of collaboration taking place between organizational units
  • Visibility of individual records can be defined by a user's assignment to groups

Do NOT use DataSilos if

  • The different organizational units shall NEVER exchange any records (in that case Domain Separation may be considered)
  • Visibility of data should be controlled by specific user roles

Supported Processes

  1. Incident
  2. Change Request
  3. Problem
  4. Service Catalog Requests (not the Catalog itself)

How DataSilos controls access

Organizational Dimension

In the organizational dimension user access to records is controlled on the basis of a user's group memberships and whether the user is directly associated to a record (e.g. because the user created the record).

Users are members of groups. Groups can be associated to data silos by a silo manager. Each association between a group and a data silo has a type. Whether an individual user can see a record of a supported process depends on the type of their group's association with a silo.

Member: Users in groups that are associated to a silo as type "Member" can only see those records that are or were directly assigned to that group.

Fulfiller: Users in groups that are associated to a silo as type "Fulfiller" can see all records that have been initially assigned to groups that are associated to the same silo. And they see all records that have ever been assigned directly assigned to their group. As a result, fulfillers have an overview of all work that originates from the same silo. However, fulfillers do not see any records that have been created in another silo and assigned to another team within the fulfiller's silo.

Supervisor: Users in groups that are associated to a silo as type "Supervisor" can see all records that have ever been assigned to any group of the same silo. As a result, supervisors have an overview of all work originating and/or that is conducted in their silos. This includes all records that have been created in other silos and that have been assigned to the supervisor's silo.

A hierarchy or network of silos can be defined. Silo-to-silo relationships can be defined between "super" silos to "sub" silos. Supervisors in the "super" silo are granted access to all records whitelisted for the "sub" silo. These relationships can be bi-directional, so both strict hierarchies and all other forms of networks can be described.

Note that silo-to-silo relationships are not inherited. If silo "A" supervisors can access sub silo "B" records, and silo "B" supervisors have access to silo "C" records, silo "A" supervisor do not automatically get access to silo "C" records - unless a relationship between silo "A" and silo "C" is explicitly defined.

Security Clearance Dimension

Access to records can also be controlled based on individual user security clearance levels. The security clearance level is a numeric value. All users and all records have level 0 by default. If a record is set to a higher level, it can only be seen by users who have an equal or higher security clearance level.

DataSilos does not actively set the security clearance level when records or users are created. This should be done by a customer specific customization. But whenever two records are linked (like an incident task being associated to an incident) the highest security clearance level requirement of all related record is inherited. As a result, all related records will then have the highest security clearance level requirement of all related records.

Security clearance-based filtering and the organizational dimension are combined. If one of the two denies access, no access is granted. This may result in records not being visible even to the user who created it.

Why DataSilos is probably the better solution

Many solutions have been built over time by ServiceNow customers and ServiceNow consultants. Some of them at some point in time ran into either performance or maintenance issues. Here is why DataSilos is likely superior to many comparable solutions in the past:

  1. Many existing solutions reflected the complexity of the rules directly in the modified query with severe impact on performance. DataSilos separates the complexity of access control rules from the actual query. The rules are applied when records change, not when they are queried by the user.
  2. With an increasing number of records in a table, queries take longer - that can't be avoided. DataSilos always uses the same query pattern which remains unchanged even if the access rules change. As a result, indices and query hints can be applied on database level to improve performance.
  3. DataSilos also covers corner cases when field values from records are included in a request to another table (dot walking) or when records are presented to ACLs incompletely. DataSilos considers these cases.
  4. DataSilos can handle the derived tables of tables in scope for filtering.
  5. DataSilos also protects database views which contain tables that should be filtered. Most existing solutions did not cover that aspect and eventually left records unprotected when database views were in use.
  6. Business rules and ACLs must be used in combination to cover all scenarios. Many solutions miss one of the two.
  7. When ACLs are used to control access based on a general access rule, these rules must be checked in every ACL on a given table. If just one ACL does not also check the filtering rule, access might be granted to users which should not access the record. DataSilos uses ATF tests to verify that all ACLs of a table are configured correctly and can help to prevent that future changes open up security issues.

System Requirements

Protection Mechanisms

The following mechanism are applied to enforce the filtering:

Before Query Business Rules

DataSilos has business rules that modify the query which is used to retrieve data for a table in scope from the database. Before query business rules are triggered whenever a user or background job requests data explicitly from a table. However, before query business rules are not triggered if data from a table is integrated into the response for a different table - e.g. if fields from a referenced record are used in a list view or form.

ACLs

Since before query business rules are not always triggered when retrieving data from a given table, ACLs are in place to perform a check on every single record that is delivered back from the database. This inlcudes cases where fields are joined from other tables via dot-walking or reference fields.

It's in the nature of ACLs to allow access to a table, record, or specific field if there is at least one ACL that grants access. This means that the DataSilos specific logic must be considered in every single ACL of a table in scope. And that implies that the protection mechanism is always in danger to be compromised by later changes to the platform. New or changed features might require new ACLs to be added or existing ones to be modified. A developer who is not aware of the role ACLs play for DataSilos filtering might easily remove (or not add) the required logic to the ACL script. To mitigate this, DataSilos contains ATF tests that check the configuration of all relevant ACLs. However, it is crucial that these tests are execute as part of the regular development and deployment process.

Database View Protection

Database views need to be protected explicitly. The before query business rule of a table are not triggered if a database view is used to retrieve data from a table. ACLs are triggered when records are retrieved through a database view - but the records being presented to the ACL script are incomplete, so specific logic must be applied when checking records within database views.

In order to apply the filtering on records when using database views DataSilos uses a background job to configure business rules for all databse views automatically. In addition it adds the fields to the table view configuration of a databse view if required. As a result, no additional manual configuration is required to protect data delivered via database views.

Implementation Guide

Analysis

The following aspects and topics should be considered when implementing the DataSilos application:

  1. What data (which specific tables) should be separated and what data should be shared across the organization?
  2. How should access be sliced and structured? What silos will exist?
  3. Who should have which access within a silo?
  4. HR, information security or other special stakeholder groups might have additional requirements. Have these been considered?
  5. Are security clearance levels of individual users to be considered to access records?
  6. How should external suppliers and vendors be affected? The specific rights and roles of users associated to external suppliers and vendors should be considered.
  7. If Domain Separation is already installed or the plan is to use both solutions in combination: Which specific security aspects should be covered by Domain Separation and which should be covered by DataSilos?
  8. Does the existing group structure support the later silo configuration? Are groups missing? Are there duplicates?
  9. Are users are assigned to multiple groups? Will that result in expected and intended behavior?
  10. Are there users without any assignment groups? Such users might lose access that they currently have.
  11. Who is responsible to define the data silos and can provide a mapping to groups? In large organizations this might be a challenging task - especially if there are multiple systems of record from which organization data is retrieved and imported into the ServiceNow instance.
  12. Who should be responsible to maintain the silo configuration? Which processes might be affected? What must happen if new groups are being created in the future?
  13. Should data silo configuration be automated - maybe as part of existing processes that integrate organizational data into the platform?
  14. Should all groups be considered for filtering? Are there groups which should be ignored?
  15. Does a process exist to manage groups? Consider the process implications once a group must also be assigned to a silo.
  16. Do naming conventions exist? How should group silos be named?
  17. What should happen with records in supported tables that do not have an assignment group - and hence may be invisible to most users?
  18. Which impact will DataSilos have on the onboarding of parts of the organization that do not yet use ServiceNow yet (or not yet actively)? Can plans and arrangements be made already for that future scenario?
  19. Are there any before/query business rules that serve comparable purposes? Should they be replaced?
  20. Are there any customized ACLs in place?
  21. Which integrations exist and how will (or should) they be impacted? Keep in mind that technical users used for integrations will also be affected by access restrictions applied by DataSilos.
  22. Which database views exist that contain tables which are in scope? Should data segregation be applied to views?
  23. How many records are in the tables in scope? Any number higher than approx. two Million should raise concerns as queries might become slow. Can records be archived?
  24. How many database indexes are "left" on the tables in scope? Often there is a maximum of 64 indexes per physical database table. Are there enough indexes left to apply recommended optimizations?
  25. What reports are maintained and run and by whom? Consider the impact of data segregation on reports!
  26. Is Performance Analytics being used? What impact on Performance Analytics dashboards is expected?
  27. DataSilos introduces new roles (admin, manager etc.). What implication does it have to your license and user access management processes?
  28. Consider that before DataSilos is activated, existing records must be initialized - i.e. their whitelists must be populated so that visibility is not prevented for users who should have access. Do existing records have sufficient information to apply the visibility rules? Can all records be updated? Would such updates trigger unnecessary notifications?

User Stories

The following list of user stories is a recommendation based on previous DataSilos implementations. It should be extended and modified based on the findings of the analysis.

  1. As the CISO I must ensure that incidents/problems/changes/... are only visible to specific groups of users.
  2. As the CISO I must ensure that specific rules are applied to control the initial visibility of records.
  3. As the CISO I must ensure that specific groups are NOT considered to control visibility.
  4. As the CISO I want to allow users who raised an incident/problem/change/... to access it.
  5. As the CISO I must ensure that users can only access records according to their security clearance level.
  6. As the CHRO I demand that only entitled users can access records that contain sensitive HR-related data.
  7. As the platform owner I want to ensure that all relevant groups are set up and users are assigned to groups.
  8. As the platform owner I want to ensure that all relevant silos are set up and groups are assigned to silos.
  9. As the platform owner I want to have a comprehensive documentation on DataSilos, its dependencies and customizations.
  10. As the platform owner I want ServiceNow developers from my organization to be trained to maintain and support the solution.
  11. As the platform owner I want to ensure that future changes do not break the data segregation logic (e.g. by adding additional ACLs that bypass filtering).
  12. As the platform owner I want to reduce customizations. When upgrading to future releases we need checks if DataSilos can be replaced by OOTB features.
  13. As the platform owner I want to ensure that users have access from day one. This requires initializing all records before go-live.
  14. As the platform owner I want to ensure that no records become totally inaccessible to everyone. Maintenance jobs should always run.
  15. As the platform owner I want to ensure that scheduled jobs are run using a technical user of my choice.
  16. As the platform owner I want to ensure maintenance job performance. Database indices for the last_maintenance fields should be added to all tables.
  17. As the platform owner I must ensure that all records in all tables in scope can be updated by the maintenance jobs.
  18. As the platform owner I want to reduce manual work. Where possible users should be assigned to groups automatically.
  19. As the platform owner I want to reduce manual work. Where possible groups should be assigned to silos automatically.
  20. As the platform owner I want to reduce manual work. User's security clearance levels should be set automatically.
  21. As the platform owner I want to ensure platform performance. One aspect is the consolidation of multiple ACLs into one.
  22. As the platform owner I want to ensure platform performance. One option is to add database indexes to optimize query performance.
  23. As the platform owner I want to ensure that existing and future database views do not bypass access restrictions.
  24. As the platform owner I want to ensure stability by running all related ATF tests as part of the development and deployment process.
  25. As the platform owner I want to ensure stability and consider running all related ATF tests to monitor production.
  26. As the platform owner I want to ensure that existing integrations continue to work as expected.
  27. As the platform owner I want to ensure that maintenance jobs do not trigger unnecessary notifications.
  28. As the platform owner I want to automate the steps required during installation.
  29. As the QA manager I need existing tests (both manual and automated) are updated to reflect the new expected behavior.
  30. As the QA manager I need new (manual and/or automated) tests to verify the new access restrictions work as expected.
  31. As an operations user I want to be informed and trained properly so that I can operate the solution going forward.
  32. As an operations user I need documented incident models and operational procedures to deal with issues being raised during operations.
  33. As an operations user I need a process in place to manage user accounts that considers the new roles and eventually the need for new group memberships.
  34. As an operations user I need a process in place to manage groups and their association to silos on a day-to-day basis.
  35. As a user I need access to records even if I have not been part of any group in the past.
  36. As a user I want to be informed about the introduction of data segregation and to know what that means to me and my work.
  37. As the incident manager I want to make specific incidents (e.g. major incidents or incidents with the highest priority) visible to everyone in the organization.
  38. As the one responsible for reporting and performance analytics I want to make sure that reports and dashboards work as expected.

Preparing Validation

The following spects should be considered for testing and validation:

  1. DataSilos modifies queries being sent to the database in order to implement the visibility filtering. These modifications come at a performance cost. Depending on how many teams (and silos) a user is a member of and depending on how many records are in the table, the performance impact might be visible to users. IT IS STRONGLY RECOMMENDED TO PERFORM A REALISTIC LOAD TEST! A simulation of actual user behavior in a production-like instance with production-like configuration can show if the performance impact is acceptable.
  2. It is recommended to identify pages and views that are most frequently used by users - both for automated and manual user acceptance testing.
  3. Functional test cases should proof that specific records are visible to specific user groups AND that other records are explicitly NOT visible to other user groups.
  4. Functional test cases should cover scenarios in which records are passed from one silo to another.
  5. Functional and performance tests for dashboards.
  6. Functional tests on existing reports and Performance Analytics based dashboards.
  7. Any existing test cases (that relate to the supported tables) should be executed as part of the QA. Behavior that was expected in the past may no longer be working with active filtering. In some cases, the silo configuration might have to be reviewed, in other cases, the test cases might have to be updated.
  8. It is strongly recommended to use specific test users and test groups for functional testing. Do not reuse users or groups from other tests!

Installation

Before installing, verify if there is a plugin scoped app that contains specific customizations and additions relevant to your organization's instance and which may come with its own installation instructions. In such cases, follow the installation instructions shipped with the custom scoped app.

Please note that is it NOT RECOMMENDED to deploy DataSilos and related scoped apps using update sets. Use source control instead!

  1. Create an account on GitHub - if not done already.
  2. Create a personal access token for your GitHub account.
  3. Add credentials to access GitHub - use "Basic Auth".
  4. Fork the repository https://github.com/saschawildgrube/servicenow-devtools.
  5. Go to Studio and import the DevTools application from source control.
  6. Perform all installation steps for the DevTools application documented here: https://www.wildgrube.com/servicenow-devtools 
  7. Fork the repository https://github.com/saschawildgrube/servicenow-datasilos.
  8. Go to Studio and import the DataSilos application from source control.
  9. Elevate roles to security_admin
  10. Go to "DataSilos/Access Control (ACL)" and check if there are any ACLs in the "DataSilos" application scope that refer to OOTB tables (e.g. incident, incident_task, etc.) with a name like e.g. incident.* (This is "incident DOT asterix"). If you see some, they must be deleted. Do NOT delete any ACLs that refer to any x_snc_datasilos* fields (like e.g. x_snc_datasilos_group_whitelist). A platform bug eventually creates them during installation and associates them to the app scope. Note that they cannot be deleted directly, because the platform denies any change to an ACL that refers to a table in the global scope from within an application scope. Use the following trick: Change the table to "Data Silos" then "Save" and then "Delete" them.
  11. Switch to global scope
  12. Create a new update set in the global scope to capture all changes related to the DataSilos installation. In case some of the ACLs reside in different scopes, you might have to create additional update sets for these scopes to capture all changes.
  13. WARNING: In the following steps, a script is going to make changes to existing scripts within ACLs or add scripts to ACLs. Verify that you have sufficient test coverage in place to confirm regression safety.
  14. The InstallApp() function must be executed.
    Run the following script as a background script in scope x_snc_datasilos:
  15. x_snc_devtools.InstallApp("x_snc_datasilos");
  16. Check out if there are any errors or warnings in the script output.
  17. If there are ACLs which have not been modified automatically, then perform the steps listed below for each of the supported tables (including but not limited to the following):
  18. incident
    incident_task
    change_request
    change_task
    problem
    problem_task
    sc_request
    sc_req_item
    sc_task
    1. At least two ACLs for the read operation must exist for each table: table_name and table_name.*. Ideally only ONE ACL exists for table_name and table_name.* (Note it is "table_name DOT asterix") for the read operation. You may have to create or change existing ACLs or consolidate multiple ACLs into one. Either way: make sure that all active ACLs contain the following script logic.
    2. Activate "Advanced" (if not active already) and add the following script:
    3. answer = x_snc_datasilos.ExecuteACL(current);
    4. If the ACL already contains a script, you will need to combine the DataSilos specific code with the existing script. You should use the following template:
    5. var bAnswer = false;
      
      /* Enter your (existing) code here! Make sure you set bAnswer (and not answer)! */
      bAnswer = your_code;	
      
      if (bAnswer == true)
      {
      	bAnswer = x_snc_datasilos.ExecuteACL(current);
      }
      answer = bAnswer;
    6. CAUTION: setting the response variable "answer" sometimes only works ONCE during a script. To avoid any issues, it is recommended to set the "answer" value only once at the end of the script.
  19. In case the default "System Administrator" user does no longer exist, is deactivated or should not be used to run scheduled jobs, identify a technical user that should be used to run scheduled jobs and configure the user for all DataSilos related jobs.
  20. If possible, activate the DataSilos app in the options and then go to "Test Suite" and execute all tests. Fix any issues reported by the tests. If needed deactivate the DataSilos app after running all tests. The following list may help you to troubleshoot failing tests:
    • If tests fail directly on the first step it means that DataSilos (or the specific table) has not been configured to be active.
    • Tests may fail because the scoped app "DevTools" has not been installed or not properly set up. Have all relevant tables been configured to allow cross-scope access from all scopes? Check out the DevTools installation guide for more details.
    • In rare occasions the ATF engine fails to remove test data that has been created during a test. This may result in test data being left over in several tables. The test "Test data sanity" can help to find such records. These records must be deleted manually.
    • Tests may fail because test data records cannot be created. One reason are customized fields which are configured to be required when creating a record. To resolve this, you can create business rules that populate these fields when new records are created. You may want to check if the name or short description starts with "_ATF_DATASILOS_" to determine records that are created during DataSilos specific tests.
    • Tests may also fail to to restrictions that result from other before query business rules. Try deactivating them to see what happens. Maybe additional configuration on users, groups or roles are needed.
    • Also consider restrictions resulting from Domain Separation. DataSilos can be used in combination with Domain Separation - however make sure that test data records are all created in the same domain.
  21. Set all update sets to complete.
  22. Optimize the database:
  23. There are two options:

    1. Let DataSilos automatically create recommended indexes by setting the system property x_snc_datasilos.database_indexes.active to true.
    2. Raise a case with ServiceNow support (or in case you run your instance on-premise with your internal database support team) and ask for the following:
    3. Hello,
      	
      as part of the installation of DataSilos, optimization on database level is required.
      
      Please execute the following background script in the x_snc_datasilos scope to trigger queries that may need optimization:
      	
      x_snc_datasilos.Log(x_snc_devtools.RenderValue(x_snc_datasilos.GetStatusPerformance()));
      
      Check out the time required to execute the different queries.
      	
      The "backmarker" queries return exactly one record and should not take more than 50ms.
      This can be achieved by adding a database index on the "Last Maintenance" fields.
      On tables that are derived from the "task" table, a combined index with the "sys_class_name" field is recommended.
      	
      All other queries should not take more than 500ms each.
      Please check if additional indices and query hints can be used to optimize these queries.
      	
      Please note that all options for code-level optimization have already been exhausted.
      The queries are already the fastest and most efficient known way to fulfill customer requirements. 
      
      Thank you very much.
      	
      Best regards,
  24. After the database optimization is in place and the maintenance scripts are running at full speed, navigate to the DataSilos log and check if there are messages that indicate aborting maintenance scripts. The log entry will state the sys_id of the record that is blocking the maintenance script. All data issues in existing records must be fixed so that the maintenance script can process all records in the table continuously.
  25. Navigate to "DataSilos/Status" and verify that there are no active tables that are marked "incomplete". Do not proceed until all records have been processed at least once.

Configuration

  1. Assign the x_snc_datasilos.manager role to users who are tasked to configure the silos.
  2. As a data silos manager go to the "Data Silos" module and configure all data silos. Add the corresponding groups to each data silo. Groups associated to a data silo are either Member, Fulfiller or Supervisor groups. This can also be done automatically or via data import.
  3. Setup silo-to-silo relationships where appropriate. Supervisors in a super silo also get access to the associated sub silo.
  4. Assign the x_snc_datasilos.sentinel role to technical users where applicable. They will not be affected by any filtering.
  5. Assign the x_snc_datasilos.sentinel role to users who are tasked to create reports and hence must access all data where applicable.
  6. Go to "Options" and choose if the maintenance jobs should be active or not. As a general rule of thumb, it's always a good idea to let them run UNLESS they cause a performance degradation. If there are already records in the tables in scope, the maintenance jobs need to run and process every record at least once to make sure all records are initialized (and hence visible). After initial processing of all existing records, the business rules on the affected tables should ensure that all records are visible to eligible users, however sometimes unexpected situations occur - like an import of data from an external system without triggering business rules etc. The maintenance jobs make sure that such events don't leave records invisible to users.
  7. Go to "ITSM Options" and configure the tables. Which tables should be affected by filtering? Which fields should be considered for instant access and whitelisting?
  8. Go to the "Status" page to verify the configuration settings.

Activation

  1. Go to "Options" and activate the maintenance jobs. Wait until all existing records have been initialized by checking the oldest date in the "Last Maintenance" field. The "Status" page provides an overview on the progress of the initial maintenance.
  2. Go to the "Options" module and activate the DataSilos application.
  3. Deactivate any existing conflicting business rules that may have been identified earlier.
  4. Go to "Test Suite" and execute all tests.

Rollback

In case anything goes terribly wrong after activation, follow these steps:

  1. Activate ATF testing and run all tests (in production!) to verify that the installation and functionality works from the perspective of the DataSilos application.
  2. Go to the "Options" module and DEACTIVATE the DataSilos application.
  3. If this does not resolve the issue, deactivate all "DataSilos" business rules and UNDO all changes done to ACLs.
  4. If this still does not resolve issues - the root cause is probably somewhere else. However, at this point one could remove the DataSilos application as a whole – BUT KEEP IN MIND THAT THIS WILL IRREVOCABLY DELETE ALL WHITELIST INFORMATION!

Reference

Roles

  • x_snc_datasilos.admin

    A user that has access to all DataSilo administration and configuration modules. Includes the x_snc_datasilos.manager role.

  • x_snc_datasilos.ghost

    Users with that role do not trigger business rules when creating or updating records.

  • x_snc_datasilos.manager

    A user that can configure data silos and assign groups to data silos.

  • x_snc_datasilos.sentinel

    A user that is NOT impacted by any filtering by DataSilos. This is suitable for users that should have global access (e.g. for reporting) or technical users (for integrations).

UI Actions

  • DataSilos Access Analysis (global)

    Enables users with the DataSilos manager role to check if a given user may or may not see a record according to the DataSilos filter.

  • DataSilos Maintenance (global)

    Enables users with the DataSilos manager role to trigger the maintenance of the current record to make sure that data changes are reflected in the record's visibility configuration.

Configuration Options

Scheduled Jobs

Table Columns

Each supported table is equipped with several new columns to support DataSilos filtering:

  • x_snc_datasilos_global The global visibility flag controls if this record should be globally visible - i.e. it can be seen by all users without any exception. This flag can be useful to make very important incidents public to everyone.
  • x_snc_datasilos_supervisor_whitelist The supervisor whitelist field stores a list of data silos. The users in the supervisor groups within the data silos will be able to access the record.
  • x_snc_datasilos_fulfiller_whitelist The fulfiller whitelist field also stores a list of data silos. The users in the fulfiller groups within the data silos will be able to access the record.
  • x_snc_datasilos_group_whitelist The group whitelist field stores a list of groups which should have access to the record.
  • x_snc_datasilos_security_clearance The security clearance field defines the required security clearance level that a user must have to be allowed to see a record.
  • x_snc_datasilos_ghost The ghost flag controls if the record's whitelists should be updated. If set to false, whitelists are populated. If the ghost flag is true, whitelists remain unchanged. This allows custom implementations to control the whitelist population. If this flag is set to true, the record behaves in the same way as if it was changed by a user with the x_snc_datasilos.ghost role.
  • x_snc_datasilos_last_maintenance The last maintenance field stores the last point in time when the record has been "maintained" - that is when the corresponding scheduled maintenance job applied the visibility rules to the record.

FAQ

What happens if a user is not associated to any group?

The user will only see records that they created (opened_by) or which have been created on their behalf (caller_id). Those users will not see any other records unless that user has the admin role or the x_snc_datasilos.sentinel role.

What happens if a record is not associated to any group?

Only users who created that record or have the admin role will see this record.

How does it actually work?

DataSilos modifies all queries sent to the database of the tables in scope. Once the data comes back, the same filter logic is applied in ACLs. DataSilos adds several fields to all relevant tables that contain whitelists of groups and silos, the flag for global visibility, the security clearance level requirement and other relevant information. Whenever records are created or changed, these fields are updated according to the configured rules - if the ghost flag is not set for the record.

How do I apply my own visibility rules on records?

DataSilos implements a default behavior whenever a record is created or updated. Whitelists are extended and the security clearance level requirement is increased if a record is related to another record with a higher security clearance level. Some of that behavior can be configured using system properties. If you need a completely different behavior you can create a business rule that is triggered before a record is inserted or updated. In this business rule, set the ghost flag to true and populate the whitelists, security clearance level, and global flag as required. When the ghost flag is set to true, DataSilos will not make any further changes to the record.

Runbook

Incident Models

Instance not responding / Severe performance issue

Symptoms
  • The instance is completely unavailable.
  • Some requests take extremely long to be answered.
  • Large amount of expensive database queries.
Procedure
  1. Independently of whether DataSilos is the root cause, deactivating DataSilos completely may be considered as an emergency option. This should be aligned with stakeholders as this will (temporarily) expose the organization to legal or compliance risks.
  2. Deactivate DataSilos by setting the x_snc_datasilos.active property to "false".

Technical integration does no longer work as expected

Symptoms
  • A technical integration has suddenly stopped to work as expected.
  • Some records that should be exposed to a 3rd party system can no longer be accessed.
Procedure
  1. Identify the technical user(s) which are used as part of the integration.
  2. Check if the technical user has the required access.
  3. If not done already, consider to assign the x_snc_datasilos.sentinel role to such technical users. Keep in mind that this may provide access to the technical user which may not be intended. However, this may be a temporary workaround.

Individual users complain about not being able to see specific records

Symptoms
  • A user reports that they do not have access to a specific record.
  • After creating a new record, errors are being displayed.
Procedure
  1. Identify the specific user record.
  2. Copy the user's Sys ID.
  3. Navigate to the record. From the form context menu, select "DataSilos Access Analysis" and enter the Sys ID (or user name or email address). The function will explain if and why a user can or cannot see a record based on DataSilos filter mechanisms.
  4. If the analysis states that the user should be able to see the record (based on DataSilos controlled criteria) but the user cannot access the record, then it is very likely that other (non-DataSilos) related business rules, domain separation or ACLs block the user from accessing that record.

ATF test "DataSilos - Test data sanity" fails

Symptoms
  • ATF test "DataSilos - Test data sanity" fails.
Procedure
  1. Run this script in scope x_snc_datasilos:
  2. // Delete DataSilos Test Data
    var testdata = new x_snc_datasilos.DataSilosTestDataAPI('_ATF_DATASILOS_');
    x_snc_datasilos.Log('Before DeleteAllData(): '+x_snc_devtools.RenderValue(testdata.GetAllData()));
    testdata.DeleteAllData(testdata.GetAllData());
    x_snc_datasilos.Log('After DeleteAllData(): '+x_snc_devtools.RenderValue(testdata.GetAllData()));

Developer Guide

Entry Points

The DataSilos application is implemented in a number of script includes. Most of which contain a single function. The following script includes (i.e. functions) serve as entry points to the application. All other functions are then called from these entry points:

  1. ExecuteBusinessRule
  2. ExecuteACL
  3. Maintenance
  4. MaintenanceDatabaseViews

For a more detailed description refer to the DataSilos API Reference below.

DataSilos API Reference

The following functions can be used to extend and modify out of the box behavior without making any changes to the core implementation. This allows for extensions which are compatible to future versions.

Script Includes

  • AccessAnalysis

    Performs an analysis on a list of users against a list of records. Can these users see these records?

  • AccessAnalysisForUserAndRecord

    Performs an access analysis for a single user on a single record.

  • AppBuilder

    The function turns an existing scoped app into a DataSilos plugin app and adds all components required to support a given set of tables.

  • AppGetDependencies

    Retrieves information about this app's dependencies to other apps.

  • AppGetProperty

    Gets a system property of this application.

  • AppInstall

    The function performs steps required for installation.

  • AppSetProperty

    Sets a system property of this application.

  • CheckACLScript

    Returns true if the given ACL script is supporting DataSilos filtering correctly.

  • CreateTestData

    Creates test data records for various scenarios. Do not use on production environments!

  • DataSilosClientAPI

    The DataSilosClientAPI exposes functions to be called from the bowser.

  • DataSilosTestDataAPI

    The class DataSilosTestDataAPI contains functions to create test data which is required for the many ATF tests that validate the DataSilos Application.

  • Debug

    Logs a debug output using DevTools' Debug function.

  • DevTools

    The DevTools extension point implementation to support DataSilos specific extensions to the DevTools "Up" button.

  • DevToolsGetLinkDirectory

    The DevToolsGetLinkDirectory function is called by the DevTools extension point instance class and returns an extended (or modified) link directory object that serves as the basis for the DevTools Link page.

  • ExecuteACL

    ExecuteACL is the entry point function used within script blocks of ACLs.

  • ExecuteBusinessRule

    This function serves as the main entry point to the DataSilos application and should be called from the script block of a business rule which responds to insert, update and query operations.

  • GetAllActiveTables

    Returns the name of all active tables as an array.

  • GetAllSupportedTables

    Returns an array containing the names of all supported tables.

  • GetBusinessRuleName

    Creates a name for a DataSilos business rule for a given table that does not exceed 40 characters but is very likely collision-free.

  • GetConfig

    This function returns the (cached) configuration object that contains all configuration settings of the DataSilos application. System properties may override default configuration. The function returns the configuration settings that are actually used by the application.

  • GetConfigBaseline

    Returns the baseline configuration for DataSilos.

  • GetConfigValue

    The function returns an individual value or sub-object from the configuration object provided by GetConfig().

  • GetGroupFilter

    Returns the configured encoded query string to filter for groups relevant for filtering. Use this to ignore e.g. permission/role groups.

  • GetGroupsFromUser

    Returns the groups of the given user considering the configured filter.

  • GetScopeFromRecord

    Returns the configured scope based on a record.

  • GetScopeFromTable

    Returns the configured scope based on a table.

  • GetSiloByName

    The function returns the sys_id of a silo based on a given silo name. It returns false if no silo can be found.

  • GetSiloRecordByName

    Returns the silo record based on a given name if it exists, otherwise false.

  • GetSilosFromUserByTypes

    Returns all silos that the user belongs to, filtered by the type of the group in the silo.

  • GetStatusPerformance

    Returns an object with current status information about database performance.

  • GetSupportedTable

    Gets the supported (parent) table based on a given table.

  • GetSupportedTableFromRecord

    Gets the supported table from the chain of parent tables based on a record.

  • IsActive

    Returns if the DataSilos application is currently active. Active means that the filtering of records is applied.

  • IsRecordComplete

    Checks a record if all fields relevant for the access check are contained.

  • IsSecurityClearanceActive

    Returns true if the filtering based on security clearance levels is active, otherwise false.

  • IsTableActive

    Returns true if the table is configured to be filtered, false if not.

  • IsTableSupported

    The function returns true if the given table is supported by the DataSilos application.

  • IsUserGhost

    The function returns true if the current user explicitly has the ghost role. It will return false for admin users who do not have the role explicitly.

  • IsUserManager

    The function returns true if the current user has the manager role. It will return false for admin users who do not have the role explicitly.

  • IsUserSentinel

    The function returns true if the current user explicitly has the sentinel role. It will return false for admin users who do not have the role explicitly.

  • Log

    Writes a message to the log using DevTools' Log function.

  • LogError

    Produces an error log output in the application log.

  • LogWarning

    Produces a warning log output in the application log.

  • Maintenance

    The Maintenance function is used to update records in supported tables based on the current values in fields. Essentially the Maintenance function iterates over records in the table specified by the given GlideRecord object and simulates a change to each record. It is used to initialize records after initial installation or when specific silo configuration changes were made. It is recommended to trigger the RecordMaintenance function on a regular basis. A GlideRecord object must be provided to choose a specific table and a predefined filter.

  • MaintenanceACLs

    Maintain existing ACLs and create missing ACLs required to protect the tables in scope.

  • PrepareQuery

    The PrepareQuery function is the heart of the DataSilos application. It takes a GlideRecord object and adds all filter criteria that apply to the current user when trying to access records in the given table. It is used by all entry point functions. However, it does not check for the roles of the current user - which means it can also be used to check if a record would be visible to a user even if the user has the sentinel or admin role.

  • RecordAddGroupToWhitelist

    Adds a group to the whitelist of the given record.

  • RecordAddSiloToFulfillerWhitelist

    The function adds a silo (identified by a sys_id) to the fulfiller whitelist field of a given record. If the silo is already whitelisted, no change is made.

  • RecordAddSiloToSupervisorWhitelist

    The function adds a silo (identified by a sys_id) to the supervisor whitelist field of a given record. If the silo is already whitelisted, no change is made.

  • RecordCheckAccess

    Checks if the given record would pass the DataSilos filter for a given user considering the user's roles. The function may return true if DataSilos does not filter the record, but the user might still not be able to access it due to other security contraints (e.g. missing roles etc.).

  • RecordCheckFilter

    Checks if the given record would pass the DataSilos filter for a given user but only strictly based on the query filter, ignoring the user's roles. The function may return true if DataSilos does not filter the record, but the user might still not be able to access it due to other security contraints (e.g. missing roles etc.). The function may return false, but if the user has the admin or sentinel role, they may still access the record.

  • RecordDistributeVisibilityToRelatedRecords

    Distributes the visibility of the given record to all related records.

  • RecordGetFulfillerWhitelistValue

    Retrieves the value of the fulfiller whitelist field from a record.

  • RecordGetGhostValue

    Retrieves the value of the ghost field from a record.

  • RecordGetGlobalValue

    Retrieves the value of the global field from a record.

  • RecordGetGroupWhitelistValue

    Retrieves the value of the group whitelist field from a record.

  • RecordGetSecurityClearanceValue

    Retrieves the value of the security clearance field from a record.

  • RecordGetSupervisorWhitelistValue

    Retrieves the value of the supervisor whitelist field from a record.

  • RecordGetVisibilityHash

    Generates a hexadecimal hash based on a record's whitelist and other visibility rules.

  • RecordInheritVisibilityFromRelatedRecords

    Inherits the visibility of the given record from related records.

  • RecordInsert

    The function populates the whitelists of a given record and its related records. The function is used by ExecuteBusinessRule when a record is being inserted.

  • RecordMaintenance

    The RecordMaintenance function is used to update a record in a supported table during a maintenance operation. It will update the whitelists based on the current values in fields. It is used to initialize records after initial installation or when specific silo configuration changes were made.

  • RecordSetFulfillerWhitelistValue

    Sets the value of the fulfiller whitelist field of a record.

  • RecordSetGhostValue

    Sets the value of the ghost field of a record.

  • RecordSetGlobalValue

    Sets the value of the global field of a record.

  • RecordSetGroupWhitelistValue

    Sets the value of the group whitelist field of a record.

  • RecordSetSecurityClearanceValue

    Sets the value of the security clearance field of a record.

  • RecordSetSupervisorWhitelistValue

    Sets the value of the supervisor whitelist field of a record.

  • RecordUpdate

    The function populates the whitelists of a given record and its related records. The function is used by ExecuteBusinessRule when a record is being updated.

Extension Points

  • DataSilos

    This extension point allows plugin apps to extend DataSilos' capabilities.

    var DataSilos = Class.create();
    DataSilos.prototype = {
        initialize: function() {
        },
    
        GetConfig: function(config)
    	{
    		var IsValidFunction = x_snc_devtools.IsValidFunction;
    		if (IsActive() && IsValidFunction(x_this_app_scope.DataSilosGetConfig))
    		{
    			config = DataSilosGetConfig(config);
    		}
            return config;
        },
    	
        GetRelatedRecords: function(record)
    	{
    		var IsValidFunction = x_snc_devtools.IsValidFunction;
    		if (IsActive() && IsValidFunction(x_this_app_scope.DataSilosGetRelatedRecords))
    		{
    			return DataSilosGetRelatedRecords(record);
    		}
            return false;
        },		
    
        RecordInsert: function(record)
    	{
    		var IsValidFunction = x_snc_devtools.IsValidFunction;
    		if (IsActive() && IsValidFunction(x_this_app_scope.DataSilosRecordInsert))
    		{
    			return DataSilosRecordInsert(record);
    		}
            return false;
        },		
    	
        RecordUpdate: function(record)
    	{
    		var IsValidFunction = x_snc_devtools.IsValidFunction;
    		if (IsActive() && IsValidFunction(x_this_app_scope.DataSilosRecordUpdate))
    		{
    			return DataSilosRecordUpdate(record);
    		}
            return false;
        },	
    	
        GetStatus: function(status)
    	{
    		var IsValidFunction = x_snc_devtools.IsValidFunction;
    		if (IsActive() && IsValidFunction(x_this_app_scope.DataSilosGetStatus))
    		{
    			status = DataSilosGetStatus(status);
    		}
            return status;
        },	
    
        type: 'DataSilos'
    };

Extension of Scope

You can extend the scope of the DataSilos application by adding the filtering logic to more tables. You might also modify the regular behavior of DataSilos when working with tables already supported by DataSilos. In either scenario you need to create a DataSilos plugin application. Before you start the implementation, perform the following steps:

  1. Check if there is a more recent version of DataSilos available that may already cover your requirements.
  2. Get in touch with the author of the DataSilos application and check if there is work in progress on a new version of DataSilos. Eventually analysis or development work can be re-used or at least knowledge can be shared. Maybe the feature you are considering is already being worked on.
  3. Identify the specific tables that require protection. Keep in mind that a parent table may already be supported.
  4. Identify the rules to be used to control access. Is all the information available that is required to apply these rules?
  5. Identify data sources. Is it based on manual input from users or is data provided via integrations?
  6. Verify if these sources provide the necessary details that are required to derive the visibility rules.

After a decision has been made to create a DataSilos plugin application perform the following steps:

  1. Create a new scoped application in Studio.
  2. Identify the affected tables
  3. Create a new script include "AppBuilder" in the new scoped application.
  4. Implement the AppBuilder function based on the following template:
  5. function AppBuilder()
    {
    	x_snc_datasilos.AppBuilder(
    		'x_abc_your_app_scope',
    		[
    			'table1',
    			'table2'
    		],
    		'Your App', // The title of the submenu section
    		20000 // The order where your submenu should start
    		);				
    }
  6. Replace "x_abc_your_app_scope" by the actual app scope identifier.
  7. Replace "Your App" with a suitable caption for the sub menu within the DataSilos menu.
  8. Replace "20000" with a reasonable order number where the sub menu should be positioned. Values must be equal or greater than 10000. Please note that if other developers who create DataSilos plugins make the same or very similar choices, there might be collisions - but they can easily be resolved later.
  9. Add the tables to be supported into the array. Note that you do not need to add tables which are already supported by DataSilos in order to implement deviating behavior for visibility rules. Also keep in mind that if a parent table is already supported, you should not add any derived tables either.
  10. Now build the boiler-plate code and prepare the implementation of your new DataSilos plugin application. Replace x_abc_your_app_scope by the actual scope identifier of your new application.
    Run the following script as a background script in scope x_abc_your_app_scope:
  11. x_abc_your_app_scope.AppBuilder();
  12. Modify or add your specific code in script include "DataSilosGetConfig". For example, you can specifiy fields that reference users or groups which should impact the visibility of records. You can also modify the configuration of tables which are supported by DataSilos directly. Please note that the modifications to the config object are applied AFTER system properties are read. That means that your implementation has full control over the actual configuration and can override system properties - that may be convenient during development, but it may be confusing for operations teams.
  13. Modify or add your specific code in script include "DataSilosGetRelatedRecords". Use this function to identify records that are related to a record. By specifying related records you can automatically derive visibility from one record to be applied to its related records (e.g. the task records associated to a story record, etc.).
  14. Modify or add your specific code in script include "DataSilosRecordInsert" to specify what should happen when a record is inserted. If the default behavior should be applied to a table, no implementation is required.
  15. Modify or add your specific code in script include "DataSilosRecordUpdate" to specify what should happen when a record is updated or maintained on a regular basis. If the default behavior should be applied to a table, no implementation is required.
  16. Modify the DataSilos UI list view for the supported tables. It is recommended at add all columns that can help operations teams to understand whether a record should or should not be visible to a user. Whatever column is being used in your visibility rules should be displayed in your UI list view.
  17. Create ATF tests. The following tests can (and should) be used as templates:
  18. DataSilos - installation
    DataSilos - implementation
    DataSilos - incident - installation
    DataSilos - incident - basic
    DataSilos - incident - opener, caller and sentinel
    DataSilos - incident - global
    DataSilos - incident - cross-silo
    DataSilos - incident_task - installation
    DataSilos - incident_task - basic
    DataSilos - incident_task - cross-silo
  19. Add test steps in the "Installation" test to check the ACL configuration for each supported table using the "DataSilos - Check ACLs" test step configuration.
  20. Add test steps in the "Implementation" test to check the table implementations for each supported table using the "DataSilos - Check table implementation" test step configuration.
  21. Write documentation. Your documentation should be targeted to both developers who might be tasked to extend the application later and to operations teams who need to resolve operational issues and user requests and questions.

Test Steps

DataSilos uses the following customized Test Step Configurations:

  • DataSilos - Check ACLs

    Checks all ACLs of a given table for compliance with DataSilos requirements.

  • DataSilos - Check table implementation

    Checks if DataSilos features are fully implemented for a given table. This includes business rules, fields, properties, ATF tests and more.

  • DataSilos - IsTableActive

    Checks if the given table is set active in the DataSilos configuration and if DataSilos is globally active. This test step should be used before any actual test steps are performed to verify that the test does not fail because of the configuration settings.

  • DataSilos - Log test data

    Displays all current test data records on the test step result log. This test step is intended to be used temporarily to analyze failing ATF tests.

License

Copyright 2020-2023 by Sascha Wildgrube

Licensed under the Apache License, Version 2.0 (the "License")

You may not use DataSilos except in compliance with the License.

You may obtain a copy of the License at: https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Release Notes

3.8.0 - 2023-07-26

  1. DevTools 1.65.0 is now required.
  2. DataSilos has been fully tested in Tokyo and Utah.
  3. Fixed numerous CodeSanity findings.
  4. The DataSilos test suite no longer contains the DevTools test suite as a child test suite.
  5. Log() and Debug() have been refactored to use the new PrepareLogInfo() and PrepareLogDebug() functions.
  6. The system property "x_snc_devtools.log.active" has been removed.
  7. Added scheduled jobs to the manual.
  8. Corrected some mistakes in the manual.

3.7.0 - 2022-11-07

  1. DevTools 1.45.0 is now required.
  2. Rome is no longer actively supported.
  3. Read ACLs for a table are now created during ACL maintenance if they do not exist yet.
  4. The test step configuration "DataSilos - Check table implementation" treats the "DataSilos Security Clearance" field on the sys_user table in a more flexible way. It is now allowed to use a deviating label for the field and define a choice list. In addition the read ACLs on the field are not checked on the sys_user table as clients may want to define other than standard access controls.
  5. The test step configuration "DataSilos - Check table implementation" input variable for the table was erroneously limited to 40 characters. This has been fixed. 80 characters are now allowed.
  6. The UI action "DataSilos Access Analysis" now uses the better looking DevTools' modal dialogs.
  7. The UI action "DataSilos Access Analysis" now uses DevTools' client script function ServerRequest().
  8. The status page now contains a link to the table's configuration form.
  9. Improved the layout of the status page.
  10. AppBuilder() now considers more implementation variants in script includes DataSilosRecordInsert() and DataSilosRecordUpdate() to make sure relevant changes are made when new tables are added.
  11. AppBuilder() now adds some helpful demo code to function DataSilosGetStatus().

3.6.0 - 2022-08-11

  1. DevTools 1.42.0 is now required.
  2. The extension point now also supports the GetStatus() function which can be used to provide additional status information to be displayed on the status page.
  3. The UI action "DataSilos Access Analysis" is now also available in the list context menu.
  4. The ATF test step config "DataSilos - Check ACLs" would also look at ACLs which are not of type "record" - which may lead to false negative test results.
  5. Added documentation about the extension point.
  6. Improved documentation on extending the scope: which tables should be added, which should not.

3.5.0 - 2022-07-19

  1. DevTools 1.41.0 is now required.
  2. The MainenanceACL() function unintentionally modified the script contained in non-record type ACLs for a filtered table. This has been fixed.
  3. A defect in AccessAnalysisForUserAndRecord() has been fixed that prevented to "DataSilos Access Analysis" UI action to fail.
  4. Added display of Domain Separation status per table on the Status page.
  5. The status page now requires the x_snc_datasilos.manager role.
  6. Added ATF test for function AccessAnalysisForUserAndRecord().
  7. Added system property "x_snc_datasilos.logging.verbosity" to control the log level for the application.
  8. Added guidance on adding test steps in the installation and implementation tests when extending the scope.

3.4.1 - 2022-04-25

  1. Regression defect caused extension points to be ignored. This has been fixed.

3.4.0 - 2022-04-25

  1. DevTools 1.37.0 is now required.
  2. Added function GetAllActiveTables().
  3. GetRelatedRecords() now processes extension point implementations in reverse order to make sure that the implementation with the highest order has ultimate control over the response.
  4. GetSecurityClearanceFromUser() is no longer triggering business rules when retrieving user information and uses the transaction cache to store the result.
  5. Refactoring of code to be more CodeSanity compliant.

3.3.0 - 2022-03-10

  1. DevTools 1.34.0 is now required.
  2. UI Action "DataSilos Maintenance" is now using DevTools' GlideRecordUpdate() in ghost mode to update records - which allows updating records in scoped tables which are not configured for cross-scope access.
  3. GetSupportedTable() now uses transaction caching to avoid repetitive database queries and function calls.
  4. Functions PrepareQuery(), RecordCheckFilter() and RecordCheckAccess() no longer default to the current users if no or an invalid user id is provided.
  5. AppBuilder() now considers tables that are in the scope of the application.
  6. The "Start" page has been renamed to "Manual".
  7. The label of the field storing a user's security clearance level has been renamed to "User Security Clearance" - otherwise one could confuse it with the field the stores the security clearance level of the record itself - if a plugin supports filtering the sys_user table.
  8. The group filter is now considered in the "Analysis" and "User Stories" chapters in the manual.

3.2.0 - 2022-02-03

  1. Paris is no longer actively supported.
  2. DevTools 1.32.0 is now required.
  3. AccessAnalysis() and the corresponding UI action "DataSilos Access Analysis" now also consider domain separation.
  4. Roles section in the documentation is now using DocumentationAPI::RenderRoles() function.
  5. AppBuilder() now checks if a sub menu title is specified and if the sub menu start order is 10000 or higher.
  6. AppSetDefaults() now sets the "log.active" system property properly.

3.1.0 - 2021-12-09

  1. DevTools 1.30.0 is now required.
  2. A new scheduled job collects status information and stores it in the new status table. The status page displays the status information from the status table if possible which reduces load time of the status page to a minimum.
  3. Added UI Action "DataSilos Maintenance" to trigger the update of a record's visibility configuration manually.
  4. GetRelatedRecords() now checks if the records returned from plugins are valid - thus preventing errors caused by invalid records.
  5. A regression defect caused the test step configuration "DataSilos - Implementation" not to be cross-scope-aware. This has been fixed.
  6. An unintendedly created table has been removed from the app.
  7. The function RecordInsert() in the example implementation of the DataSilos extension point was incorrectly named RecordInitialize(). This has been fixed.
  8. Added test "DataSilos - Ready" to check if all preconditions are met to run tests in the first place: The user running the tests must not have the x_snc_datasilos.ghost role.

3.0.0 - 2021-11-19

  1. DevTools 1.27.1 is now required.
  2. Introduced silo-to-silo relationships with visibility inheritance across silos for supervisors.
  3. Introduced silo based filtering for fulfillers.
  4. Record maintenance no longer triggers business rules, auditing and sys field updates, allowing seamless updated of whitelists on records without triggering any unintended processes.
  5. The "Status" page now contains links to ACLs, business rules, storage aliases and database indexes for each table.
  6. The new system property x_snc_datasilos.log.active controls if log output should be produced by DataSilos.
  7. The new function AppBuilder() can now be used to build scoped apps that extend DataSilos.
  8. The scheduled job "DataSilos - Maintenance" (formerly "DataSilos - Database view protection") is now maintaining both database view and ACL configuration.
  9. All relevant ACLs are now reconfigured automatically where required. Changes made are documented within the ACL description.
  10. The new ui action "DataSilos Access Analysis" helps DataSilos manager users to explain why a user can or cannot see a record.
  11. Removed scheduled job "DataSilos - CreateTestData".
  12. The function CreateTestData() is now available from all scopes and can be used to create records for demos, manual testing and automated testing scenarios.
  13. Table specific application menu modules have been removed and the test step configuration "DataSilos - Check table implementation" no longer checks for table specific application menu modules. The status page contains all the table specific links.
  14. Renamed function GetOptimizationStatus() to GetStatusPerformance().
  15. The "Options" area is now split up between general settings in "Options" and ITSM table specific options at "ITSM Options".
  16. Added function RecordCheckFilter() to check if the record's filter criteria would allow a user to access a record ignoring the user's roles.
  17. Added function RecordCheckAccess() to check if the record's filter criteria would allow a user to access a record considering the user's roles.

2.10.0 - 2021-09-22

  1. DataSilos has been fully tested on Rome.
  2. Orlando is no longer actively supported.
  3. DevTools 1.24.0 is now required.
  4. Introduced extension point instance for "DevTools" to support the "Up" button on silo-to-group mapping record forms.
  5. Added "DataSilos - DevTools" child test suite to the "DataSilos" test suite so that all relevant tests can be run in one go.
  6. Missing ACLs are now added into the global scope (instead of DataSilos) by MaintenanceACLs().
  7. AccessAnalysis() is now available from all application scopes and provides information on which user may see which records - but not yet why.
  8. AppInstall() is now available from all application scopes.
  9. AppInstall() now uses DevTools' InstallerAPI class for installation.
  10. Added function MaintenanceACLs() to encapsulate all ACL related configuration activities.
  11. GetOptimizationStatus() is now available from all application scopes.
  12. GetOptimizationStatus() now uses DevTools's PerformanceQueryMultiple() function.
  13. Updated installation instructions to always use AppInstall() to configure ACLs automatically.
  14. Updated installation instructions with the recommendation to use source control and to consider additional custom scoped apps that extend DataSilos' functionality.
  15. Updated installation instructions with specific guidance on database optimization.
  16. Updated installation instructions with guidance regarding records with data issues which cannot be updated during maintenance.
  17. Extended analysis agenda and added a story to verify that no unnessecary notifications are triggered by maintenance jobs.
  18. Added a story to cover the work required to ensure that all records in all tables in scope can be updated by maintenance jobs.
  19. Refactored test "DataSilos - Test data sanity" to use the new test step config "DevTools - Check test data sanity".
  20. Now using DocumentationAPI.RenderSystemRequirements() function to automatically render system requirements including dependencies to other apps.
  21. The new function AppGetDependencies() now provides information about dependencies to other apps.
  22. The "Test Suite" module now point to the DataSilos test suite explicitly.

2.9.0 - 2021-06-30

  1. DevTools 1.19.0 is now required.
  2. AppInstall() can now add missing ACLs and existing modify one-line scripts. The installation guide has been updated accordingly.
  3. Added function CheckACLScript() to check if an ACL script complies with DataSilos requirements. The function is being used in the test step configuration and the evolving yet experimental AppInstall() function.
  4. Added chapter "Protection mechanisms" to the documentation.
  5. The test step configuration "DataSilos - Check table implementation" no longer checks for a test step that checks for a cross scope configuration on tables as this is no longer required.
  6. The security clearance level query condition was added as a float value. This has been fixed. It is an integer value now.

2.8.2 - 2021-06-28

  1. Fixed a bug in GetOptimizationStatus() leading to slightly incorrect queries to be checked for performance.
  2. Added incident model for the failing ATF test "DataSilos - Test data sanity".

2.8.1 - 2021-06-24

  1. DevTools 1.18.0 is now required.
  2. Database views that use left joins are now supported.
  3. Duplicate business rules were created for database views if the database view name exceeded 28 characters. This has been fixed.
  4. DataSilosTestDataAPI.GetAllData() now also returns records from sys_db_view.
  5. The test "DataSilos - incident - database_view" checks if the system property "com.snc.sla.engine.async". If it is set to true, some of the test case would fail and hence are ignored.

2.8.0 - 2021-06-21

  1. DevTools 1.17.0 is now required.
  2. Added generic protection of database views including automatic configuration of required business rules and database view table fields.
  3. Added system property "database_views.active" to control if database views should be protected or not.
  4. Added module "DB Views" to show all database views and tables in scope.
  5. When distributing visibility between related records, the ghost flag was not considered - resulting in modifications of ghost records. This has been fixed.
  6. The test "DataSilos - incident - database views" now sets all required system properties to run the test.
  7. The test step configuration "DataSilos - Check table implementation" no longer checks for the run_as user and if the scheduled job is active.
  8. Added experimental function GetOptimizationStatus() to trigger and measure typical queries when modified by DataSilos. This function should help database administrators to apply suitable optimizations (like adding indices and query hints).
  9. Added steps in the installation guide for database level optimizations.
  10. Added stories to update and create new tests (manual and automated).
  11. Added documentation on why DataSilos might be the better solution.
  12. AppInstall() can now be executed directly from the application scope and runs MaintenanceDatabaseViews() to prepare database view protection.

2.7.0 - 2021-05-05

  1. WARNING: New York is no longer actively supported.
  2. Added system property "avoid_expensive_queries" for all supported tables and extended the test step configuration "DataSilos - Check table implementation" to check for the property.
  3. Defined the sort order in module "Logs".
  4. Added view "DataSilos" for scheduled jobs to show the "Run as" field.
  5. The Status page now displays the backmarker correctly for empty tables.
  6. Added stories for group and silo setup to the documentation.
  7. Added stories and extended installation manual to consider technical users for scheduled jobs.
  8. Added public function AppSetProperty() to allow other scoped apps to configure DataSilos via scripts.
  9. Added internal function AppSetDefaults() to set all DataSilos system properties to their defaults to prepare a release.
  10. DataSilos tables are now configured to allow cross scope configuration to enable further extensions of functionality.

2.6.0 - 2021-04-16

  1. Added the "Status" page which provides an overview on the current configuration and state of the DataSilos application on a per table level.
  2. Added property x_snc_datasilos.maintenance.active to control if the maintenance jobs should run (i.e. do their job) or not.
  3. Renamed module "Table" to "ITSM" and added a new module "CORE" to contain core system tables like sys_user etc. - another step to allow for further extensions of DataSilos.
  4. Performance optimization in GetGroupsFromUser() - special thanks to Pat Casey.
  5. Performance improvement in RecordUpdateWhitelistsFromGroupByTypes() for the case that there may be thousands of supervisor groups in a silo.
  6. All business rules responsible for whitelist population now run at order 10000 to avoid side-effects with other business rules which set fields that could impact filter rules.
  7. Extended test step configuration "DataSilos - Check table implementation" to check if the business rule order is set to 10000.
  8. Using DevTools' GetGroupsFromUser() as the group filter is now part of DevTools.
  9. GetGroupsFromUser() is now public so that it can be used by other scoped apps.
  10. Improved the code in the maintenance jobs and extended the test step configuration "DataSilos - Check table implementation" to enforce the change in future implementations.
  11. Extended runbook section in the documentation with several incident models.
  12. Added more stories to the documentation.

2.5.1 - 2021-03-22

  1. Fixed a defect in the test step configuration "DataSilos - Check table implementation".
  2. Updated documentation on the last maintenance field.

2.5.0 - 2021-03-22

  1. Madrid is no longer supported.
  2. DevTools version 1.12.2 or higher is required.
  3. Due to a potential platform defect, some removed ACLs might still be present after an update to DataSilos 2.5.0. This will cause the "DataSilos - Implementation" test to fail. Run the following script in GLOBAL scope to remove the useless ACLs:
    var grSSA = new GlideRecord('sys_security_acl');
    grSSA.addQuery('sys_scope','=','dafbffebdbb78450ba15a353059619eb');
    grSSA.addQuery('name','CONTAINS','.');
    grSSA.query();
    while (grSSA.next())
    {
    	if (grSSA.operation == 'delete' || grSSA.operation == 'create')
    	{   
    		gs.info('Deleting ACL: '+grSSA.name+' / '+grSSA.operation+' / '+grSSA.sys_id);
    		grSSA.deleteRecord();
    	}
    }
  4. For unknown reasons a duplicate of ACL "change_request.x_snc_datasilos_security_clearance" for the "write" operation exist after an update to DataSilos 2.5.0 on some instances. The duplicate must be deleted (Eventually the sys_id is db149778dbeee010ba15a3530596191a but it may also differ). Make sure that the remaining ACL requires the x_snc_datasilos.manager role.
  5. WARNING: The interface of the Maintenance() function changed. Introduced a new table maintenance mechanism based on the DevTools function RecordBulkProcessor(). A new field is introduced on all tables in scope: "Last Maintenance" which stores the point in time when the record has been "maintained" for the last time. New scheduled scripts have been introduced to maintain each table separately.
  6. The warning message on the ACL form is now displayed in red (instead of blue).
  7. Extended test "DataSilos - Test data sanity" to cover all supported tables.
  8. Extended test step configuration to check for a table specific test step in test "DataSilos - Test data sanity".
  9. Extended test step configuration "DataSilos - Check table implementation" to support table implementations in other application scopes. That is a first step to enable scoped apps to serve as DataSilos plugins.
  10. Extended test step configuration "DataSilos - Check table implementation" to check if there is a test step to check if ACLs have been properly configured for a supported table.
  11. Extended test step configuration "DataSilos - Check table implementation" to check if there is a test step to check the configuration of cross scope access for a supported table.
  12. Extended test step configuration "DataSilos - Check table implementation" to check for the new "Last Maintenance" field in each table.
  13. Added test step configuration "DataSilos - Log test data" which can be used to log all data of relevant test data records to the test step output. That can help to debug failing ATF tests.
  14. Added function DataSilosTestDataAPI.GetAllData() and DataSilosTestDataAPI.GetAllDataSilos() to gather all relevant test data records.
  15. RecordAddGroupToWhitelist() is now accessible from all application scopes.
  16. RecordGetVisibilityHash() is now accessible from all application scopes.
  17. GetAllSupportedTables() is now accessible from all application scopes.
  18. GetGroupFilter() is now accessible from all application scopes.
  19. Added get and set functions for all DataSilos specific fields. This is to support table fields provided by multiple application scopes.
  20. Added stories for user, group and silo configuration automation to the documentation.
  21. Improved installation guide: Added information on what can be done in case of failing ATFs.

2.4.0 - 2021-02-26

  1. Added support for Quebec.
  2. DevTools version 1.10.0 or higher is required.
  3. Added support for security clearance levels. In addition to the organizational dimension, access to records can now also be controlled based on security clearance levels. Each record and user has a security clearance level. A user can only access a record if their security clearance level is equal or higher than the record's security clearance level.
  4. The explicit check for Performance Analytics views has been replaced by a more generic check for queries starting with "sys_idIN". This will consider any query that comes with a defined list of records as not expensive.
  5. In the data silos to group mapping table the data silo, group and type fields are now mandatory to prevent the entry of incomplete mapping records.
  6. Renamed module "Import/Export" to "Groups to Silos". This is to clear the way for a potential upcoming feature to manage silo to silo relationships.
  7. Added experimental function AccessAnalysis() to check if and why users may see records.
  8. Removed unnecessary logging in DataSilosTestDataAPI.
  9. Fixed function GetSilosFromUser().
  10. Fixed function GetSilosFromUserbyTypes() - a crash might be caused by incomplete silo-to-group mappings.
  11. Fixed configuration of some DataSilos specific field ACLs.
  12. The test step configuration "DataSilos - Check Business Rule" has been renamed to "DataSilos - Check table implementation" and several checks have been added to make sure that the implementation for a given table is complete.
  13. Added more user stories to the documentation.
  14. Improved installation guide.
  15. Added a short description for the scoped application.

2.3.0 - 2021-01-22

  1. DevTools version 1.8.0 or higher is required.
  2. ExecuteACL can now handle the case that an incomplete record is presented to the ACL (Refer to KB0529493).
  3. If a request is triggered from a performance analytics view, we don't consider it expensive and will not apply the expensive query optimization.
  4. Added the user stories section to the documentation.
  5. Added the runbook section to the documentation with a first incident model on failing integrations.
  6. Extended the analysis section with insights from previous projects in the documentation.
  7. Using DevTools' DocumentationAPI to auto-generate parts of the documentation.

2.2.0 - 2020-11-27

  1. DevTools version 1.5.0 or higher is required.
  2. The Maintenance() function now supports a callback function to enable customized behavior when processing records.
  3. The Maintenance() function now supports to select a "window" of records to be processed to support multiple instances of scheduled jobs to run in parallel.
  4. The Maintenance() function no longer calls the RecordUpdate() function for records that have the ghost flag set to true. However, if a callback function is provided, the function is always called. Which means the callback function is responsible to check the ghost flag if needed.
  5. Improved log output of the Maintenance() function.
  6. Extended installation guide on how to troubleshoot failing tests.
  7. App sanity: All system properties have is_private set to false.
  8. App sanity: All ATF tests now have copied_from set to none.
  9. Added the ATF test "App sanity" to check for common pitfalls in scoped applications (like the two above).
  10. Added the ATF test "Test data sanity" to check if there is test data left from earlier test runs.

2.1.0 - 2020-11-16

  1. Added a warning message when editing ACLs. The warning message can be configured via system properties.
  2. ExecuteACL() did not check if DataSilos filtering is globally active or not. This has been fixed.

2.0.0 - 2020-11-11

  1. WARNING: The property x_snc_datasilos.incident.whitelist_trigger_fields has essentially be renamed to x_snc_datasilos.tables.incident.columns.whitelist_trigger.sys_user_group! Make sure to redo the configuration during upgrade!
  2. WARNING: The property x_snc_datasilos.active now controls the filter ONLY. It will no longer halt the population of the whitelists. As a result, it can now be more effectively used as an ON/OFF switch while preparing a go-live or during trouble shooting - without any negative impact on the whitelists of records that are changed in the meantime.
  3. WARNING: It is now required to configure ACLs not only for table_name.*, but also for table_name, . Updated the "ACLs" module and tests accordingly.
  4. WARNING: The business rules are now part of the scoped application. Any existing "DataSilos" business rules in the global scope should be removed!
  5. DevTools version 1.3.0 or higher is required.
  6. Introduced the supervisor datasilo whitelist field to enable silo-based whitelists for supervisors.
  7. Introduced the global visibility flag for all tables.
  8. Introduced the ghost flag for all tables. If set to true whitelists are not updated when a record is changed.
  9. Introduced configurable group filter. An encoded query can now be configured to control which groups should be considered for whitelist population.
  10. Introduced the option to avoid expensive queries. When the system property x_snc_datasilos.tables.table_name.avoid_expensive_queries is set to true, queries that do not contain a filter for active records are not altered. This results in users seeing a message that some records have been filtered from the result set and eventually empty result pages.
  11. The test "DataSilos - incident - cross-silo" now impersonates a user of the current assignment group before re-assigning the incident. This change was made to consider strict customer-specific customizations.
  12. Several script includes have been opened up for use from different scopes to enable implementation of customer specific Maintenance() functions and business rules.
  13. Extended the "Business Rules" module to show all before query business rules for the supported tables.
  14. Optimized ExecuteACL() so that it does not evaluate the access restrictions for a single record multiple times.
  15. Introduced the use of customized test step configurations.
  16. Extended the Developer Guide: The DataSilos API Reference has been added and "Scope Extension" chapter has been refined.
  17. Moved all contents from the "Requirement Analysis" chapter into the "Implementation Guide", added process relevant considerations.

1.4.0 - 2020-07-30

  1. DevTools version must be 1.2.1.
  2. Existence and version of DevTools is now checked in ATF "DataSilos - installation".
  3. Introduced the global visibility flag for the incident table. If set to true, everyone can see this particular record.
  4. Reactivated the "Log" module.
  5. The module "Maintenance" has been replaced by the "Scheduled Jobs" module and shows all scheduled script executions where the name contains "DataSilos".
  6. The module "Business rules" now shows all business rules that contain "DataSilos" in their name - this is to consider customer's naming conventions with prefixes.
  7. A bug in the Madrid release caused issues in some navigation modules preventing pre-defined filters to work properly. This has been fixed.

1.3.0 - 2020-07-16

  1. Introduced the configurable whitelist_trigger_field option. By setting the table specific option (e.g. x_snc_datasilos.incident.whitelist_trigger_field) the fields that should be used to populate the whitelist can be explicitly configured. If the option is empty the default for each table based on the task table is the assignment_group field. However, any other field that contains a reference to a group can be used. Dot-walking is NOT yet supported, though.
  2. Introduced the maintenance.log_frequency option to control how often the maintenance script should log the current progress.
  3. Deactivated the Log module for the time being.

1.2.2 - 2020-07-15

  1. The Maintenance script now explicitly checks the related tables, too. So *task records which are not associated to other whitelisted records are now whitelisted based on their assignment groups.
  2. Improved logging in the Maintenance script: Only the start and the end of the execution (with a consolidated summary) is logged.
  3. Extended the developer guide with more details.
  4. Added the "Rollback" section to the implementation guide.
  5. Added a navigation module to show all silo to group associations.

1.2.1 - 2020-07-08

  1. Users can now access request items where the associated request is opened on their behalf.
  2. Fixed issues on access rights to the admin and management navigation modules.

1.2.0 - 2020-07-07

  1. Added support for Service Catalog Requests.
  2. Restructured the installation guide. It is now an implementation guide covering analysis, installation, configuration, and the verification phase.

1.1.3 - 2020-07-06

  1. Related items (e.g. problem_task) did not apply the whitelist to their parents when created. This has been fixed.

1.1.2 - 2020-07-04

  1. Added modules to show all supported tables with the fields relevant for visibility - a useful treat for data silo managers and developers.

1.1.1 - 2020-07-04

  1. Extended "DataSilos - Installation" test to check if all business rules and ACLs have been set up correctly.
  2. Updated developer guide to update the "DataSilos - Installation" test.
  3. Updated installation guide on how to deal with erroneously created ACLs in the application scope.
  4. Test "DataSilos - Maintenance" is now Madrid compatible

1.1.0 - 2020-07-03

  1. Added support for problem and problem_task tables.
  2. Users that are referenced in a change_request in the requested_by field can see that change request.
  3. Updated documentation: What happens if a user does not have any roles?

1.0.0 - 2020-06-23

First baselined version with support for incidents and change requests.