DataSilos 2.9.0

Welcome to DataSilos - created and maintained by Sascha Wildgrube.

The DataSilos application helps to separate data between different organizational units - so called data silos.

In contrast to domain separation DataSilos allows the exchange of data (in form of individual records) between data silos and makes them visible to users on a need-to-know basis.

In addition to the organizational dimension, access to records can also be controlled based on security clearance levels. Each record and each user has a security clearance level. A user can only access a record if their security clearance level is equal or higher than the record's security clearance level.

Disclaimer

DataSilos is NOT an officially supported ServiceNow product.

DataSilos does NOT come with any kind of warranty. If you use it, you own it!

Use DataSilos if

  • Domain Separation is too strict
  • Domain Separation is already implemented for a different purpose and more detailed visibility rules must be enforced WITHIN a domain
  • There is a certain level of collaboration taking place between organizational units
  • Visibility of individual records can be defined by a user's assignment to groups

Do NOT use DataSilos if

  • The different organizational units shall NEVER exchange any records (in that case Domain Separation may be considered)
  • Visibility of data should be controlled by specific user roles

Supported processes

  1. Incident
  2. Change Request
  3. Problem
  4. Service Catalog Requests (not the Catalog itself)

How DataSilos controls access

Organizational dimension

In the organizational dimension user access to records is controlled on the basis of a user's group memberships and whether the user is directly associated to a record (e.g. because the user created the record).

Users are members of groups. Groups can be associated to data silos by a silo manager. Each association between a group and a data silo has a type. Whether an individual user can see a record of a supported process depends on the type of their group's association with a silo.

Member: Users in groups that are associated to a silo as type "Member" can only see those records that are or were directly assigned to that group.

Fulfiller: Users in groups that are associated to a silo as type "Fulfiller" can see all records that have been initially assigned to groups that are associated to the same silo. And they see all records that have ever been assigned directly assigned to their group. As a result, fulfillers have an overview of all work that originates from the same silo. However, fulfillers do not see any records that have been created in another silo and assigned to another team within the fulfiller's silo.

Supervisor: Users in groups that are associated to a silo as type "Supervisor" can see all records that have ever been assigned to any group of the same silo. As a result, supervisors have an overview of all work originating and/or that is conducted in their silos. This includes all records that have been created in other silos and that have been assigned to the supervisor's silo.

Security clearance dimension

Access to records can also be controlled based on individual user security clearance levels. The security clearance level is a numeric value. All users and all records have level 0 by default. If a record is set to a higher level, it can only be seen by users who have an equal or higher security clearance level.

DataSilos does not actively set the security clearance level when records or users are created. This should be done by a customer specific customization. But whenever two records are linked (like an incident task being associated to an incident) the highest security clearance level requirement of all related record is inherited. As a result, all related records will then have the highest security clearance level requirement of all related records.

Security clearance-based filtering and the organizational dimension are combined. If one of the two denies access, no access is granted. This may result in records not being visible even to the user who created it.

Why DataSilos is probably the better solution

Many solutions have been built over time by ServiceNow customers and ServiceNow consultants. Some of them at some point in time ran into either performance or maintenance issues. Here is why DataSilos is likely superior to many comparable solutions in the past:

  1. Many existing solutions reflected the complexity of the rules directly in the modified query with severe impact on performance. DataSilos separates the complexity of access control rules from the actual query. The rules are applied when records change, not when they are queried by the user.
  2. With an increasing number of records in a table, queries take longer - that can't be avoided. DataSilos always uses the same query pattern which remains unchanged even if the access rules change. As a result, indices and query hints can be applied on database level to improve performance.
  3. DataSilos also covers corner cases when field values from records are included in a request to another table (dot walking) or when records are presented to ACLs incompletely. DataSilos considers these cases.
  4. DataSilos also protects database views which contain tables that should be filtered. Most existing solutions did not cover that aspect and eventually left records unprotected when database views were in use.
  5. Business rules and ACLs must be used in combination to cover all scenarios. Many solutions miss one of the two.
  6. When ACLs are used to control access based on a general access rule, these rules must be checked in every ACL on a given table. If just one ACL does not also check the filtering rule, access might be granted to users which should not access the record. DataSilos uses ATF tests to verify that all ACLs of a table are configured correctly and can help to prevent that future changes open up security issues.

System requirements

Protection mechanisms

The following mechanism are applied to enforce the filtering:

Before query business rules

DataSilos has business rules that modify the query which is used to retrieve data for a table in scope from the database. Before query business rules are triggered whenever a user or background job requests data explicitly from a table. However, before query business rules are not triggered if data from a table is integrated into the response for a different table - e.g. if fields from a referenced record are used in a list view or form.

ACLs

Since before query business rules are not always triggered when retrieving data from a given table, ACLs are in place to perform a check on every single record that is delivered back from the database. This inlcudes cases where fields are joined from other tables via dot-walking or reference fields.

It's in the nature of ACLs to allow access to a table, record, or specific field if there is at least one ACL that grants access. This means that the DataSilos specific logic must be considered in every single ACL of a table in scope. And that implies that the protection mechanism is always in danger to be compromised by later changes to the platform. New or changed features might require new ACLs to be added or existing ones to be modified. A developer who is not aware of the role ACLs play for DataSilos filtering might easily remove (or not add) the required logic to the ACL script. To mitigate this, DataSilos contains ATF tests that check the configuration of all relevant ACLs. However, it is crucial that these tests are execute as part of the regular development and deployment process.

Database view protection

Database views need to be protected explicitly. The before query business rule of a table are not triggered if a database view is used to retrieve data from a table. ACLs are triggered when records are retrieved through a database view - but the records being presented to the ACL script are incomplete, so specific logic must be applied when checking records within database views.

In order to apply the filtering on records when using database views DataSilos uses a background job to configure business rules for all databse views automatically. In addition it adds the fields to the table view configuration of a databse view if required. As a result, no additional manual configuration is required to protect data delivered via database views.

Implementation guide

Analysis

The following aspects and topics should be considered when implementing the DataSilos application:

  1. What data (which specific tables) should be separated and what data should be shared across the organization?
  2. How should access be sliced and structured? What silos will exist?
  3. Who should have which access within a silo?
  4. HR, information security or other special stakeholder groups might have additional requirements. Have these been considered?
  5. Are security clearance levels of individual users to be considered to access records?
  6. How should external suppliers and vendors be affected? The specific rights and roles of users associated to external suppliers and vendors should be considered.
  7. Does the existing group structure support the later silo configuration? Are groups missing? Are there duplicates?
  8. Are users are assigned to multiple groups? Will that result in expected and intended behavior?
  9. Are there users without any assignment groups? Such users might lose access that they currently have.
  10. Who is responsible to define the data silos and can provide a mapping to groups? In large organizations this might be a challenging task - especially if there are multiple systems of record from which organization data is retrieved and imported into the ServiceNow instance.
  11. Who should be responsible to maintain the silo configuration? Which processes might be affected? What must happen if new groups are being created in the future?
  12. Should data silo configuration be automated - maybe as part of existing processes that integrate organizational data into the platform?
  13. Does a process exist to manage groups? Consider the process implications once a group must also be assigned to a silo.
  14. Do naming conventions exist? How should group silos be named?
  15. What should happen with records in supported tables that do not have an assignment group - and hence may be invisible to most users?
  16. Which impact will DataSilos have on the onboarding of parts of the organization that do not yet use ServiceNow yet (or not yet actively)? Can plans and arrangements be made already for that future scenario?
  17. Are there any before/query business rules that serve comparable purposes? Should they be replaced?
  18. Are there any customized ACLs in place?
  19. Which integrations exist and how will (or should) they be impacted? Keep in mind that technical users used for integrations will also be affected by access restrictions applied by DataSilos.
  20. How many records exist in the affected tables? If there are more than a million, expect a performance impact and validate carefully!
  21. Which database views exist that contain tables which are in scope? Should data segregation be applied?
  22. What reports are maintained and run and by whom? Consider the impact of data segregation on reports!
  23. Is Performance Analytics being used? What impact on Performance Analytics dashboards is expected?
  24. DataSilos introduces new roles (admin, manager etc.). What implication does it have to your license and user access management processes?
  25. Consider that before DataSilos is activated, existing records must be initialized - i.e. their whitelists must be populated so that visibility is not prevented for users who should have access. How exactly will this initialization take place? What should be considered?

User stories

The following list of user stories is a recommendation based on previous DataSilos implementations. It should be extended and modified based on the findings of the analysis.

  1. As the CISO I must ensure that incidents/problems/changes/... are only visible to specific groups of users.
  2. As the CISO I must ensure that specific rules are applied to control the initial visibility of records.
  3. As the CISO I want to allow users who raised an incident/problem/change/... to access it.
  4. As the CISO I must ensure that users can only access records according to their security clearance level.
  5. As the CHRO I demand that only entitled users can access records that contain sensitive HR-related data.
  6. As the platform owner I want to ensure that all relevant groups are set up and users are assigned to groups.
  7. As the platform owner I want to ensure that all relevant silos are set up and groups are assigned to silos.
  8. As the platform owner I want to have a comprehensive documentation on DataSilos, its dependencies and customizations.
  9. As the platform owner I want ServiceNow developers from my organization to be trained to maintain and support the solution.
  10. As the platform owner I want to ensure that future changes do not break the data segregation logic (e.g. by adding additional ACLs that bypass filtering).
  11. As the platform owner I want to reduce customizations. When upgrading to future releases we need checks if DataSilos can be replaced by OOTB features.
  12. As the platform owner I want to ensure that users have access from day one. This requires initializing all records before go-live.
  13. As the platform owner I want to ensure that no records become totally inaccessible to everyone. Maintenance jobs should always run.
  14. As the platform owner I want to ensure that scheduled jobs are run using a technical user of my choice.
  15. As the platform owner I want to ensure maintenance job performance. Database indices for the last_maintenance fields should be added to all tables.
  16. As the platform owner I want to reduce manual work. Where possible users should be assigned to groups automatically.
  17. As the platform owner I want to reduce manual work. Where possible groups should be assigned to silos automatically.
  18. As the platform owner I want to reduce manual work. User's security clearance levels should be set automatically.
  19. As the platform owner I want to ensure platform performance. One aspect is the consolidation of multiple ACLs into one.
  20. As the platform owner I want to ensure platform performance. One aspect is the optimization of expensive queries on database level.
  21. As the platform owner I want to ensure that existing and future database views do not bypass access restrictions.
  22. As the platform owner I want to ensure stability by running all related ATF tests as part of the development and deployment process.
  23. As the platform owner I want to ensure stability and consider running all related ATF tests to monitor production.
  24. As the platform owner I want to ensure that existing integrations continue to work as expected.
  25. As the QA manager I need existing tests (both manual and automated) are updated to reflect the new expected behavior.
  26. As the QA manager I need new (manual and/or automated) tests to verify the new access restrictions work as expected.
  27. As an operations user I want to be informed and trained properly so that I can operate the solution going forward.
  28. As an operations user I need documented incident models and operational procedures to deal with issues being raised during operations.
  29. As an operations user I need a process in place to manage user accounts that considers the new roles and eventually the need for new group memberships.
  30. As an operations user I need a process in place to manage groups and their association to silos on a day-to-day basis.
  31. As a user I need access to records even if I have not been part of any group in the past.
  32. As a user I want to be informed about the introduction of data segregation and to know what that means to me and my work.
  33. As the incident manager I want to make specific incidents (e.g. major incidents or incidents with the highest priority) visible to everyone in the organization.
  34. As the one responsible for reporting and performance analytics I want to make sure that reports and dashboards work as expected.

Preparing validation

The following spects should be considered for testing and validation:

  1. DataSilos modifies queries being sent to the database in order to implement the visibility filtering. These modifications come at a performance cost. Depending on how many teams (and silos) a user is a member of and depending on how many records are in the table, the performance impact might be visible to users. IT IS STRONGLY RECOMMENDED TO PERFORM A REALISTIC LOAD TEST! A simulation of actual user behavior in a production-like instance with production-like configuration can show if the performance impact is acceptable.
  2. It is recommended to identify pages and views that are most frequently used by users - both for automated and manual user acceptance testing.
  3. Functional test cases should proof that specific records are visible to specific user groups AND that other records are explicitly NOT visible to other user groups.
  4. Functional test cases should cover scenarios in which records are passed from one silo to another.
  5. Functional and performance tests for dashboards.
  6. Functional tests on existing reports and Performance Analytics based dashboards.
  7. Any existing test cases (that relate to the supported tables) should be executed as part of the QA. Behavior that was expected in the past may no longer be working with active filtering. In some cases, the silo configuration might have to be reviewed, in other cases, the test cases might have to be updated.
  8. It is strongly recommended to use specific test users and test groups for functional testing. Do not reuse users or groups from other tests!

Installation

  1. To install DevTools create an account on GitHub (if not done already).
  2. Create a personal access token for your GitHub account.
  3. Paris or later: Add credentials to access GitHub (Use "Basic Auth").
  4. Fork the repository https://github.com/saschawildgrube/servicenow-devtools.
  5. Go to Studio and import the DevTools application from source control.
  6. Follow the installation instruction contained in "DevTools/Start".
  7. Fork the repository https://github.com/saschawildgrube/servicenow-datasilos.
  8. Go to Studio and import the DataSilos application from source control.
  9. Elevate roles to security_admin
  10. Go to "DataSilos/Access Control (ACL)" and check if there are any ACLs in the "DataSilos" application scope that refer to OOTB tables (e.g. incident, incident_task, etc.) with a name like e.g. incident.* (This is "incident DOT asterix"). If you see some, they must be deleted. Do NOT delete any ACLs that refer to any x_snc_datasilos* fields (like e.g. x_snc_datasilos_group_whitelist). A platform bug eventually creates them during installation and associates them to the app scope. Note that they cannot be deleted directly, because the platform denies any change to an ACL that refers to a table in the global scope from within an application scope. Use the following trick: Change the table to "Data Silos" then "Save" and then "Delete" them.
  11. Switch to global scope
  12. Create a new update set in the global scope to capture all changes related to the DataSilos installation. In case some of the ACLs reside in different scopes, you might have to create additional update sets for these scopes to capture all changes.
  13. The next step is the configuration of existing ACLs. There are two options:
    1. Automated approach - WARNING: a script is going to make changes to existing scripts within ACLs. This approach is less work but it's only safe if you have sufficient test coverage in place to confirm regression safety.
    2. Manual approach - This is more work but you have full control over what is happening.
  14. Automated approach:
    1. Execute the following script in the scope x_snc_datasilos as a background script:
    2. AppInstall();
    3. Check out if there are any warnings in the script output.
    4. Apply the manual approach on those ACLs which were not changed automatically.
  15. Manual approach:
    1. For each of the following tables perform the steps listed below:
    2. incident
      incident_task
      change_request
      change_task
      problem
      problem_task
      sc_request
      sc_req_item
      sc_task
      1. At least two ACLs for the read operation must exist for each table: table_name and table_name.*. Ideally only ONE ACL exists for table_name and table_name.* (Note it is "table_name DOT asterix") for the read operation. You may have to create or change existing ACLs or consolidate multiple ACLs into one. Either way: make sure that all active ACLs contain the following script logic.
      2. Activate "Advanced" (if not active already) and add the following script:
      3. answer = x_snc_datasilos.ExecuteACL(current);
      4. If the ACL already contains a script, you will need to combine the DataSilos specific code with the existing script. You should use the following template:
      5. var bAnswer = false;
        
        /* Enter your (existing) code here! Make sure you set bAnswer (and not answer)! */
        bAnswer = your_code;	
        
        if (bAnswer == true)
        {
        	bAnswer = x_snc_datasilos.ExecuteACL(current);
        }
        answer = bAnswer;
      6. CAUTION: setting the response variable "answer" sometimes only works ONCE during a script. To avoid any issues, it is recommended to set the "answer" value only once at the end of the script.
  16. In case the default "System Administrator" user does no longer exist, is deactivated or should not be used to run scheduled jobs, identify a technical user that should be used to run scheduled jobs and configure the user for all DataSilos related jobs.
  17. Install any additional customer specific applications that are based on DataSilos and extend its functionality.
  18. If possible, activate the DataSilos app in the options and then go to "Test Suite" and execute all tests. Fix any issues reported by the tests. If needed deactivate the DataSilos app after running all tests. The following list may help you to troubleshoot failing tests:
    • If tests fail directly on the first step it means that DataSilos (or the specific table) has not been configured to be active.
    • Tests may fail because the scoped app "DevTools" has not been installed or not properly set up. Have all relevant tables been configured to allow cross-scope access from all scopes? Check out the DevTools installation guide for more details.
    • In rare occasions the ATF engine fails to remove test data that has been created during a test. This may result in test data being left over in several tables. The test "Test data sanity" can help to find such records. These records must be deleted manually.
    • Tests may fail because test data records cannot be created. One reason are customized fields which are configured to be required when creating a record. To resolve this, you can create business rules that populate these fields when new records are created. You may want to check if the name or short description starts with "_ATF_DATASILOS_" to determine records that are created during DataSilos specific tests.
    • Tests may also fail to to restrictions that result from other before query business rules. Try deactivating them to see what happens. Maybe additional configuration on users, groups or roles are needed.
    • Also consider restrictions resulting from Domain Separation. DataSilos can be used in combination with Domain Separation - however make sure that test data records are all created in the same domain.
  19. Set all update sets to complete.
  20. Execute the following background script in the x_snc_datasilos scope to trigger queries that may need database level optimization:
  21. Log(x_snc_devtools.RenderValue(GetOptimizationStatus()));
  22. If the queries appear to be slow, raise a case with ServiceNow support to implement optimizations on database level.

Configuration

  1. Assign the x_snc_datasilos.manager role to users who are tasked to configure the silos.
  2. As a data silos manager go to the "Data Silos" module and configure all data silos. Add the corresponding groups to each data silo. Groups associated to a data silo are either Member, Fulfiller or Supervisor groups. This can also be done automatically or via data import.
  3. Assign the x_snc_datasilos.sentinel role to technical users where applicable. They will not be affected by any filtering.
  4. Assign the x_snc_datasilos.sentinel role to users who are tasked to create reports and hence must access all data where applicable.
  5. Go to "Options" and configure the tables. Which tables should be affected by filtering? Which fields should be considered for instant access and whitelisting?
  6. Go to "Options" and choose if the maintenance jobs should be active or not. As a general rule of thumb, it's always a good idea to let them run UNLESS they cause a performance degradation. If there are already records in the tables in scope, the maintenance jobs need to run and process every record at least once to make sure all records are initialized (and hence visible). After initial processing of all existing records, the business rules on the affected tables should ensure that all records are visible to eligible users, however sometimes unexpected situations occur - like an import of data from an external system without triggering business rules etc. The maintenance jobs make sure that such events don't leave records invisible to users.
  7. Go to the "Status" page to verify the configuration settings.

Activation

  1. Go to "Options" and activate the maintenance jobs. Wait until all existing records have been initialized by checking the oldest date in the "Last Maintenance" field. The "Status" page provides an overview on the progress of the initial maintenance.
  2. Go to the "Options" module and activate the DataSilos application.
  3. Deactivate any existing conflicting business rules that may have been identified earlier.
  4. Go to "Test Suite" and execute all tests.

Rollback

In case anything goes terribly wrong after activation, follow these steps:

  1. Activate ATF testing and run all tests (in production!) to verify that the installation and functionality works from the perspective of the DataSilos application.
  2. Go to the "Options" module and DEACTIVATE the DataSilos application.
  3. If this does not resolve the issue, deactivate all "DataSilos" business rules and UNDO all changes done to ACLs.
  4. If this still does not resolve issues - the root cause is probably somewhere else. However, at this point one could remove the DataSilos application as a whole – BUT KEEP IN MIND THAT THIS WILL IRREVOCABLY DELETE ALL WHITELIST INFORMATION!

Roles

DataSilos supports the following roles:

  • x_snc_datasilos.admin A user that has access to all DataSilo administration and configuration modules. Includes the x_snc_datasilos.manager role.
  • x_snc_datasilos.manager A user that can configure data silos and assign groups to data silos.
  • x_snc_datasilos.sentinel A user that is NOT impacted by any filtering by DataSilos. This is suitable for users that should have global access (e.g. for reporting) or technical users (for integrations).
  • x_snc_datasilos.ghost A user that can create and modify records without impacting the whitelist.

Configuration options

Column fields

Each supported table is equipped with several new column fields:

  • x_snc_datasilos_global The global visibility flag controls if this record should be globally visible - i.e. it can be seen by all users without any exception. This flag can be useful to make very important incidents public to everyone.
  • x_snc_datasilos_supervisor_whitelist The supervisor whitelist field stores a list of data silos. The users in the supervisor groups within the data silos will be able to access the record.
  • x_snc_datasilos_group_whitelist The group whitelist field stores a list of groups which should have access to the record.
  • x_snc_datasilos_security_clearance The security clearance field defines the required security clearance level that a user must have to be allowed to see a record.
  • x_snc_datasilos_ghost The ghost flag controls if the record's whitelists should be updated. If set to false, whitelists are populated. If the ghost flag is true, whitelists remain unchanged. This allows custom implementations to control the whitelist population. If this flag is set to true, the record behaves in the same way as if it was changed by a user with the x_snc_datasilos.ghost role.
  • x_snc_datasilos_last_maintenance The last maintenance field stores the last point in time when the record has been "maintained" - that is when the corresponding scheduled maintenance job applied the visibility rules to the record.

FAQ

What happens if a user is not associated to any group?

The user will only see records that they created (opened_by) or which have been created on their behalf (caller_id). Those users will not see any other records unless that user has the admin role or the x_snc_datasilos.sentinel role.

What happens if a record is not associated to any group?

Only users who created that record or have the admin role will see this record.

How does it actually work?

DataSilos modifies all queries sent to the database of the tables in scope. Once the data comes back, the same filter logic is applied in ACLs. DataSilos adds several fields to all relevant tables that contain whitelists of groups and silos, the flag for global visibility, the security clearance level requirement and other relevant information. Whenever records are created or changed, these fields are updated according to the configured rules - if the ghost flag is not set for the record.

How do I apply my own visibility rules on records?

DataSilos implements a default behavior whenever a record is created or updated. Whitelists are extended and the security clearance level requirement is increased if a record is related to another record with a higher security clearance level. Some of that behavior can be configured using system properties. If you need a completely different behavior you can create a before business role that is triggered when a record is inserted or updated. In this business rule, set the ghost flag to true and populate the whitelists, security clearance level, and global flag as required. When the ghost flag is set to true, DataSilos will not make any changes to the record.

Runbook

Incident models

Instance not responding / Severe performance issue

Symptoms
  • The instance is completely unavailable.
  • Some requests take extremely long to be answered.
  • Large amount of expensive database queries.
Procedure
  1. Independently of whether DataSilos is the root cause, deactivating DataSilos completely may be considered as an emergency option. This should be aligned with stakeholders as this will (temporarily) expose the organization to legal or compliance risks.
  2. Deactivate DataSilos by setting the x_snc_datasilos.active property to "false".

Technical integration does not longer work as expected

Symptoms
  • A technical integration has suddenly stopped to work as expected.
  • Some records that should be exposed to a 3rd party system can no longer be accessed.
Procedure
  1. Identify the technical user(s) which are used as part of the integration.
  2. Check if the technical user has the required access.
  3. If not done already, consider to assign the x_snc_datasilos.sentinel role to such technical users. Keep in mind that this may provide access to the technical user which may not be intended. However, this may be a temporary workaround.

Individual users complain about not being able to see specific records

Symptoms
  • A user reports that they do not have access to a specific record.
  • After creating a new record, errors are being displayed.
Procedure
  1. Identify the specific user record.
  2. Check if the user has the x_snc_datasilos.sentinel role. If so, the user should not be affected by DataSilos filtering. The root cause is very likely somewhere else.
  3. Seek the record in the list view of the containing table using the DataSilos view.
  4. Check if the user has an equal or higher security clearance level as the record. If the user has a lower security clearance level than the record, the user is not allowed to see the record.
  5. If the record is set to be globally visible, the user should see the record.
  6. Check if the user is referenced in one of the user fields (e.g. caller, opened_by). Consider the specific configuration for the table as such fields may differ depending on table and configuration.
  7. Identify the groups of that user.
  8. Identify the silos that are associated with the user's group(s).
  9. Check the supervisor and group whitelist to see if the user's groups and associated silos match. Keep in mind that a user's group must be a "supervisor" group in a silo so that a user gets access via the silo whitelist.
  10. If the whitelisted silos or groups do not match expectations, the logic that populate the whitelists should be reviewed. Note that if a DataSilos extension sets the ghost field to true for a record, the default DataSilos whitelist population will not be applied.
  11. If at least one of the filtering factors (user reference fields, group whitelist, supervisor whitelist, global visibility) in combination with the security clearance level should allow the user to see the record, but the user cannot access the record, then it is very likely that other (non-DataSilos) related business rules or ACLs block the access to that record.

ATF test "DataSilos - Test data sanity" fails

Symptoms
  • ATF test "DataSilos - Test data sanity" fails.
Procedure
  1. Run this script in scope x_snc_datasilos:
  2. // Delete DataSilos Test Data
    var testdata = new x_snc_datasilos.DataSilosTestDataAPI('_ATF_DATASILOS_');
    x_snc_datasilos.Log('Before DeleteAllData(): '+x_snc_devtools.RenderValue(testdata.GetAllData()));
    testdata.DeleteAllData(testdata.GetAllData());
    x_snc_datasilos.Log('After DeleteAllData(): '+x_snc_devtools.RenderValue(testdata.GetAllData()));

Developer guide

Entry points

The DataSilos application is implemented in a number of script includes. Most of which contain a single function. The following script includes (i.e. functions) serve as entry points to the application. All other functions are then called from these entry points:

  1. ExecuteBusinessRule
  2. ExecuteACL
  3. Maintenance
  4. MaintenanceDatabaseViews

For a more detailed description refer to the DataSilos API Reference below.

DataSilos API reference

The following functions can be used to extend and modify out of the box behavior without making any changes to the core implementation. This allows for extensions which are compatible to future versions.

  • AppSetProperty

    Sets a DataSilos specific system property.

  • DataSilosTestDataAPI

    The class DataSilosTestDataAPI contains functions to create test data which is required for the many ATF tests that validate the DataSilos Application.

  • Debug

    Logs a debug output using DevTools' Debug function.

  • ExecuteACL

    ExecuteACL is the entry point function used within script blocks of ACLs.

  • ExecuteBusinessRule

    This function serves as the main entry point to the DataSilos application and should be called from the script block of a business rule which responds to insert, update and query operations.

  • GetAllSupportedTables

    Returns an array containing the names of all supported tables.

  • GetConfig

    This function returns the (cached) configuration object that contains all configuration settings of the DataSilos application. System properties may override default configuration. The function returns the configuration settings that are actually used by the application.

  • GetConfigBaseline

    Returns the baseline configuration for DataSilos.

  • GetConfigValue

    The function returns an individual value or sub-object from the configuration object provided by GetConfig().

  • GetGroupFilter

    Returns the configured encoded query string to filter for groups relevant for filtering. Use this to ignore e.g. permission/role groups.

  • GetGroupsFromUser

    Returns the groups of the given user considering the configured filter.

  • GetScopeFromRecord

    Returns the configured scope based on a record.

  • GetScopeFromTable

    Returns the configured scope based on a table.

  • GetSiloByName

    The function returns the sys_id of a silo based on a given silo name. It returns false if no silo can be found.

  • GetSilosFromUserByTypes

    Returns all silos that the user belongs to, filtered by the type of the group in the silo.

  • IsActive

    Returns if the DataSilos application is currently active. Active means that the filtering of records is applied.

  • IsRecordComplete

    Checks a record if all fields relevant for the access check are contained.

  • IsTableActive

    Returns true if the table is configured to be filtered, false if not.

  • IsTableSupported

    The function returns true if the given table is supported by the DataSilos application.

  • IsUserGhost

    The function returns true if the current user explicitly has the ghost role. It will return false for admin users who do not have the role explicitly.

  • IsUserSentinel

    The function returns true if the current user explicitly has the sentinel role. It will return false for admin users who do not have the role explicitly.

  • Log

    Writes a message to the log using DevTools' Log function.

  • Maintenance

    The Maintenance function is used to update records in supported tables based on the current values in fields. Essentially the Maintenance function iterates over records in the table specified by the given GlideRecord object and simulates a change to each record. It is used to initialize records after initial installation or when specific silo configuration changes were made. It is recommended to trigger the RecordMaintenance function on a regular basis. A GlideRecord object must be provided to choose a specific table and a predefined filter.

  • PrepareQuery

    The PrepareQuery function is the heart of the DataSilos application. It takes a GlideRecord object and adds all filter criteria that apply to the current user when trying to access records in the given table. It is used by all entry point functions. However, it does not check for the roles of the current user - which means it can also be used to check if a record would be visible to a user even if the user has the sentinel or admin role.

  • RecordAddGroupToWhitelist

    Adds a group to the whitelist of the given record.

  • RecordAddSiloToSupervisorWhitelist

    The function adds a silo (identified by a sys_id) to the supervisor whitelist field of a given record. If the silo is already whitelisted, no change is made.

  • RecordGetGhostValue

    Retrieves the value of the ghost field from a record.

  • RecordGetGlobalValue

    Retrieves the value of the global field from a record.

  • RecordGetGroupWhitelistValue

    Retrieves the value of the group whitelist field from a record.

  • RecordGetSecurityClearanceValue

    Retrieves the value of the security clearance field from a record.

  • RecordGetSupervisorWhitelistValue

    Retrieves the value of the supervisor whitelist field from a record.

  • RecordGetVisibilityHash

    Generates a hexadecimal hash based on a record's whitelist and other visibility rules.

  • RecordInitialize

    The function populates the whitelists of a given record and its related records. The function is used by ExecuteBusinessRule when a record is being inserted.

  • RecordMaintenance

    The RecordMaintenance function is used to update a record in a supported table during a maintenance operation. It will update the whitelists based on the current values in fields. It is used to initialize records after initial installation or when specific silo configuration changes were made.

  • RecordSetGhostValue

    Sets the value of the ghost field of a record.

  • RecordSetGlobalValue

    Sets the value of the global field of a record.

  • RecordSetGroupWhitelistValue

    Sets the value of the group whitelist field of a record.

  • RecordSetSecurityClearanceValue

    Sets the value of the security clearance field of a record.

  • RecordSetSupervisorWhitelistValue

    Sets the value of the supervisor whitelist field of a record.

  • RecordUpdate

    The function populates the whitelists of a given record and its related records. The function is used by ExecuteBusinessRule when a record is being updated.

Extension of scope

Extending the scope of the DataSilos application essentially means to add the filtering logic to more tables. This may have impact on the applications, views, reports, and other components that make use of these tables. So before actually extending the scope of DataSilos consider the following steps:

  1. Verify if there is a more recent version of DataSilos is available that may already cover the new requirements.
  2. Get in touch with the author of the DataSilos application and check if there is work in progress on a new version of DataSilos. Maybe development work can be synchronized, or knowledge can be shared. It is also possible that some of the analysis work has already been done.
  3. Identify the specific data that requires protection.
  4. Map out where the data is coming from: manual input from users or integrations?
  5. Identify the rules to be used to control access.

After a decision has been made to extend scope of the DataSilos application perform the following steps:

  1. Identify the affected tables
  2. Add new test steps in the ATF test "DataSilos - Implementation" for each of the tables. As soon as all following steps are completed, the test should pass.
  3. Add the global visibility field to all affected tables! Please note that you MUST adapt the column name after entering the column label!
  4. Type: True/False
    Column label: DataSilos Global Visibility
    Column name: x_snc_datasilos_global
    Default value: false
    Elevate role to security_admin and add read and write Access Controls to the new field for all affected tables:
    Name: DataSilos Global Visibility
    Requires role: x_snc_datasilos.manager
  5. Add the supervisor silo whitelist field to all affected tables! Please note that you MUST adapt the column name after entering the column label!
  6. Type: List
    Column label: DataSilos Supervisor Whitelist
    Column name: x_snc_datasilos_supervisor_whitelist
    Attributes: edge_encryption_enabled=true,no_sort=true,slushbucket_ref_no_expand=true
    Reference: DataSilo (search for "x_snc_datasilos")
    Reference cascade rule: None (not "--None--"!)
    Elevate role to security_admin and add read and write Access Controls to the new field for all affected tables:
    Name: DataSilos Supervisor Whitelist
    Requires role: x_snc_datasilos.manager
  7. Add the group whitelist field to all affected tables! Please note that you MUST adapt the column name after entering the column label!
  8. Type: List
    Column label: DataSilos Group Whitelist
    Column name: x_snc_datasilos_group_whitelist
    Attributes: edge_encryption_enabled=true,no_sort=true,slushbucket_ref_no_expand=true
    Reference: Group
    Reference cascade rule: None (not "--None--"!)
    Elevate role to security_admin and add read and write Access Controls to the table DataSilos whitelist columns for all affected tables:
    Name: DataSilos Group Whitelist
    Requires role: x_snc_datasilos.manager
  9. Add the security clearance field to all affected tables! Please note that you MUST adapt the column name after entering the column label!
  10. Type: Integer
    Column label: DataSilos Security Clearance
    Column name: x_snc_datasilos_security_clearance
    Default value: 0
    Elevate role to security_admin and add read and write Access Controls to the new field for all affected tables:
    Name: DataSilos Security Clearance
    Requires role: x_snc_datasilos.manager
  11. Add the ghost field to all affected tables! Please note that you MUST adapt the column name after entering the column label!
  12. Type: True/False
    Column label: DataSilos Ghost
    Column name: x_snc_datasilos_ghost
    Default value: false
    Elevate role to security_admin and add read and write Access Controls to the new field for all affected tables:
    Name: DataSilos Ghost
    Requires role: x_snc_datasilos.manager
  13. Add the last maintenance field to all affected tables! Please note that you MUST adapt the column name after entering the column label!
  14. Type: Date/Time
    Column label: DataSilos Last Maintenance
    Column name: x_snc_datasilos_last_maintenance
    Default value: false
    Elevate role to security_admin and add read and write Access Controls to the new field for all affected tables:
    Name: DataSilos Last Maintenance
    Requires role: x_snc_datasilos.manager
  15. Add new navigation modules to display the new tables
  16. Configure the DataSilos view for these new modules
  17. Extend the GetConfigBaseline script include to support all new tables. Use the existing table configurations as templates.
  18. Extend the GetRelatedRecords script include to support the identification of related records of a record (e.g. to get the incident_task records from an incident)
  19. Extend the filter condition in the "ACLs" module to cover the new tables.
  20. Extend the filter condition in the "Business Rules" module to cover the new tables.
  21. Add the properties for each table and add the properties to the DataSilos system property category. You can use the following properties as a template:
  22. x_snc_datasilos.tables.incident.active
    x_snc_datasilos.tables.incident.columns.instant_access.sys_user
    x_snc_datasilos.tables.incident.columns.whitelist_trigger.sys_user_group
  23. New business rules need to be added for each table. These business rules are "before query" business rules that will affect out-of-scope tables. To make that change, the following client scripts and the following business rule must be deactivated:
    1. Client Script: "No query for out-of-scope tables" (onChange)
    2. Client Script: "No query for out-of-scope tables" (onLoad)
    3. Business Rule: "Reject before-Q for out-of-scope tables"
  24. Add a new business rule for each of the new supported tables in the DataSilos application scope!
  25. Configure the business rule as follows (replace "table_name" by the corresponding table name):
  26. Name: "DataSilos - table_name"
    When: before
    Insert: true
    Update: true
    Query: true
    Add the following script:
    (function executeRule(current, previous /*null when async*/) {
    	x_snc_datasilos.ExecuteBusinessRule(current, previous);
    })(current, previous);
  27. Go to the "Business Rules" module and verify the new business rule is active and in the DataSilos application scope.
  28. Add a scheduled job for each table (replace "table_name" with the exact name of the table):
    Name: DataSilos - Maintenance - table_name
    Run: Periodically
    Repeat Intervall: 10 Minutes
    Run as: System Administrator
    Run this script (Again replace "table_name" by the table name):
    var grRecords = new GlideRecord('table_name')
    Maintenance(grRecords);
  29. Switch to global scope
  30. Elevate access to "security_admin".
  31. Add or modify corresponding ACLs for all affected tables. Refer to the implementation guide.
  32. Switch back to DataSilos scope
  33. Create ATF tests. The following tests can (and should) be used as templates:
  34. DataSilos - incident - installation
    DataSilos - incident - basic
    DataSilos - incident - opener, caller and sentinel
    DataSilos - incident - global
    DataSilos - incident - cross-silo
    DataSilos - incident_task - installation
    DataSilos - incident_task - basic
    DataSilos - incident_task - cross-silo
  35. Update THIS documentation
  36. Test the new feature against all known existing use cases related to the added tables.

Test Steps

DataSilos uses the following customized Test Step Configurations:

  • DataSilos - Check ACLs

    Checks all ACLs of a given table for compliance with DataSilos requirements.

  • DataSilos - Check table implementation

    Checks if DataSilos features are fully implemented for a given table. This includes business rules, fields, properties, ATF tests and more.

  • DataSilos - IsTableActive

    Checks if the given table is set active in the DataSilos configuration and if DataSilos is globally active. This test step should be used before any actual test steps are performed to verify that the test does not fail because of the configuration settings.

  • DataSilos - Log test data

    Displays all current test data records on the test step result log. This test step is intended to be used temporarily to analyze failing ATF tests.

License

Copyright 2020-2021 by Sascha Wildgrube

Licensed under the Apache License, Version 2.0 (the "License")

You may not use DevTools except in compliance with the License.

You may obtain a copy of the License at: https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Release notes

2.9.0 - 2021-06-30

  1. DevTools 1.19.0 is now required.
  2. AppInstall() can now add missing ACLs and existing modify one-line scripts. The installation guide has been updated accordingly.
  3. Added function CheckACLScript() to check if an ACL script complies with DataSilos requirements. The function is being used in the test step configuration and the evolving yet experimental AppInstall() function.
  4. Added chapter "Protection mechanisms" to the documentation.
  5. The test step configuration "DataSilos - Check table implementation" no longer checks for a test step that checks for a cross scope configuration on tables as this is no longer required.
  6. The security clearance level query condition was added as a float value. This has been fixed. It is an integer value now.

2.8.2 - 2021-06-28

  1. Fixed a bug in GetOptimizationStatus() leading to slightly incorrect queries to be checked for performance.
  2. Added incident model for the failing ATF test "DataSilos - Test data sanity".

2.8.1 - 2021-06-24

  1. DevTools 1.18.0 is now required.
  2. Database views that use left joins are now supported.
  3. Duplicate business rules were created for database views if the database view name exceeded 28 characters. This has been fixed.
  4. DataSilosTestDataAPI.GetAllData() now also returns records from sys_db_view.
  5. The test "DataSilos - incident - database_view" checks if the system property "com.snc.sla.engine.async". If it is set to true, some of the test case would fail and hence are ignored.

2.8.0 - 2021-06-21

  1. DevTools 1.17.0 is now required.
  2. Added generic protection of database views including automatic configuration of required business rules and database view table fields.
  3. Added system property "database_views.active" to control if database views should be protected or not.
  4. Added module "DB Views" to show all database views and tables in scope.
  5. When distributing visibility between related records, the ghost flag was not considered - resulting in modifications of ghost records. This has been fixed.
  6. The test "DataSilos - incident - database views" now sets all required system properties to run the test.
  7. The test step configuration "DataSilos - Check table implementation" no longer checks for the run_as user and if the scheduled job is active.
  8. Added experimental function GetOptimizationStatus() to trigger and measure typical queries when modified by DataSilos. This function should help database administrators to apply suitable optimizations (like adding indices and query hints).
  9. Added steps in the installation guide for database level optimizations.
  10. Added stories to update and create new tests (manual and automated).
  11. Added documentation on why DataSilos might be the better solution.
  12. AppInstall() can now be executed directly from the application scope and runs MaintenanceDatabaseViews() to prepare database view protection.

2.7.0 - 2021-05-05

  1. WARNING: New York is no longer actively supported.
  2. Added system property "avoid_expensive_queries" for all supported tables and extended the test step configuration "DataSilos - Check table implementation" to check for the property.
  3. Defined the sort order in module "Logs".
  4. Added view "DataSilos" for scheduled jobs to show the "Run as" field.
  5. The Status page now displays the backmarker correctly for empty tables.
  6. Added stories for group and silo setup to the documentation.
  7. Added stories and extended installation manual to consider technical users for scheduled jobs.
  8. Added public function AppSetProperty() to allow other scoped apps to configure DataSilos via scripts.
  9. Added internal function AppSetDefaults() to set all DataSilos system properties to their defaults to prepare a release.
  10. DataSilos tables are now configured to allow cross scope configuration to enable further extensions of functionality.

2.6.0 - 2021-04-16

  1. Added the "Status" page which provides an overview on the current configuration and state of the DataSilos application on a per table level.
  2. Added property x_snc_datasilos.maintenance.active to control if the maintenance jobs should run (i.e. do their job) or not.
  3. Renamed module "Table" to "ITSM" and added a new module "CORE" to contain core system tables like sys_user etc. - another step to allow for further extensions of DataSilos.
  4. Performance optimization in GetGroupsFromUser() - special thanks to Pat Casey.
  5. Performance improvement in RecordUpdateWhitelistsFromGroupByTypes() for the case that there may be thousands of supervisor groups in a silo.
  6. All business rules responsible for whitelist population now run at order 10000 to avoid side-effects with other business rules which set fields that could impact filter rules.
  7. Extended test step configuration "DataSilos - Check table implementation" to check if the business rule order is set to 10000.
  8. Using DevTools' GetGroupsFromUser() as the group filter is now part of DevTools.
  9. GetGroupsFromUser() is now public so that it can be used by other scoped apps.
  10. Improved the code in the maintenance jobs and extended the test step configuration "DataSilos - Check table implementation" to enforce the change in future implementations.
  11. Extended runbook section in the documentation with several incident models.
  12. Added more stories to the documentation.

2.5.1 - 2021-03-22

  1. Fixed a defect in the test step configuration "DataSilos - Check table implementation".
  2. Updated documentation on the last maintenance field.

2.5.0 - 2021-03-22

  1. Madrid is no longer supported.
  2. DevTools version 1.12.2 or higher is required.
  3. Due to a potential platform defect, some removed ACLs might still be present after an update to DataSilos 2.5.0. This will cause the "DataSilos - Implementation" test to fail. Run the following script in GLOBAL scope to remove the useless ACLs:
    var grSSA = new GlideRecord('sys_security_acl');
    grSSA.addQuery('sys_scope','=','dafbffebdbb78450ba15a353059619eb');
    grSSA.addQuery('name','CONTAINS','.');
    grSSA.query();
    while (grSSA.next())
    {
    	if (grSSA.operation == 'delete' || grSSA.operation == 'create')
    	{   
    		gs.info('Deleting ACL: '+grSSA.name+' / '+grSSA.operation+' / '+grSSA.sys_id);
    		grSSA.deleteRecord();
    	}
    }
  4. For unknown reasons a duplicate of ACL "change_request.x_snc_datasilos_security_clearance" for the "write" operation exist after an update to DataSilos 2.5.0 on some instances. The duplicate must be deleted (Eventually the sys_id is db149778dbeee010ba15a3530596191a but it may also differ). Make sure that the remaining ACL requires the x_snc_datasilos.manager role.
  5. WARNING: The interface of the Maintenance() function changed. Introduced a new table maintenance mechanism based on the DevTools function RecordBulkProcessor(). A new field is introduced on all tables in scope: "Last Maintenance" which stores the point in time when the record has been "maintained" for the last time. New scheduled scripts have been introduced to maintain each table separately.
  6. The warning message on the ACL form is now displayed in red (instead of blue).
  7. Extended test "DataSilos - Test data sanity" to cover all supported tables.
  8. Extended test step configuration to check for a table specific test step in test "DataSilos - Test data sanity".
  9. Extended test step configuration "DataSilos - Check table implementation" to support table implementations in other application scopes. That is a first step to enable scoped apps to serve as DataSilos plugins.
  10. Extended test step configuration "DataSilos - Check table implementation" to check if there is a test step to check if ACLs have been properly configured for a supported table.
  11. Extended test step configuration "DataSilos - Check table implementation" to check if there is a test step to check the configuration of cross scope access for a supported table.
  12. Extended test step configuration "DataSilos - Check table implementation" to check for the new "Last Maintenance" field in each table.
  13. Added test step configuration "DataSilos - Log test data" which can be used to log all data of relevant test data records to the test step output. That can help to debug failing ATF tests.
  14. Added function DataSilosTestDataAPI.GetAllData() and DataSilosTestDataAPI.GetAllDataSilos() to gather all relevant test data records.
  15. RecordAddGroupToWhitelist() is now accessible from all application scopes.
  16. RecordGetVisibilityHash() is now accessible from all application scopes.
  17. GetAllSupportedTables() is now accessible from all application scopes.
  18. GetGroupFilter() is now accessible from all application scopes.
  19. Added get and set functions for all DataSilos specific fields. This is to support table fields provided by multiple application scopes.
  20. Added stories for user, group and silo configuration automation to the documentation.
  21. Improved installation guide: Added information on what can be done in case of failing ATFs.

2.4.0 - 2021-02-26

  1. Added support for Quebec.
  2. DevTools version 1.10.0 or higher is required.
  3. Added support for security clearance levels. In addition to the organizational dimension, access to records can now also be controlled based on security clearance levels. Each record and user has a security clearance level. A user can only access a record if their security clearance level is equal or higher than the record's security clearance level.
  4. The explicit check for Performance Analytics views has been replaced by a more generic check for queries starting with "sys_idIN". This will consider any query that comes with a defined list of records as not expensive.
  5. In the data silos to group mapping table the data silo, group and type fields are now mandatory to prevent the entry of incomplete mapping records.
  6. Renamed module "Import/Export" to "Groups to Silos". This is to clear the way for a potential upcoming feature to manage silo to silo relationships.
  7. Added experimental function AccessAnalysis() to check if and why users may see records.
  8. Removed unnecessary logging in DataSilosTestDataAPI.
  9. Fixed function GetSilosFromUser().
  10. Fixed function GetSilosFromUserbyTypes() - a crash might be caused by incomplete silo-to-group mappings.
  11. Fixed configuration of some DataSilos specific field ACLs.
  12. The test step configuration "DataSilos - Check Business Rule" has been renamed to "DataSilos - Check table implementation" and several checks have been added to make sure that the implementation for a given table is complete.
  13. Added more user stories to the documentation.
  14. Improved installation guide.
  15. Added a short description for the scoped application.

2.3.0 - 2021-01-22

  1. DevTools version 1.8.0 or higher is required.
  2. ExecuteACL can now handle the case that an incomplete record is presented to the ACL (Refer to KB0529493).
  3. If a request is triggered from a performance analytics view, we don't consider it expensive and will not apply the expensive query optimization.
  4. Added the user stories section to the documentation.
  5. Added the runbook section to the documentation with a first incident model on failing integrations.
  6. Extended the analysis section with insights from previous projects in the documentation.
  7. Using DevTools' DocumentationAPI to auto-generate parts of the documentation.

2.2.0 - 2020-11-27

  1. DevTools version 1.5.0 or higher is required.
  2. The Maintenance() function now supports a callback function to enable customized behavior when processing records.
  3. The Maintenance() function now supports to select a "window" of records to be processed to support multiple instances of scheduled jobs to run in parallel.
  4. The Maintenance() function no longer calls the RecordUpdate() function for records that have the ghost flag set to true. However, if a callback function is provided, the function is always called. Which means the callback function is responsible to check the ghost flag if needed.
  5. Improved log output of the Maintenance() function.
  6. Extended installation guide on how to troubleshoot failing tests.
  7. App sanity: All system properties have is_private set to false.
  8. App sanity: All ATF tests now have copied_from set to none.
  9. Added the ATF test "App sanity" to check for common pitfalls in scoped applications (like the two above).
  10. Added the ATF test "Test data sanity" to check if there is test data left from earlier test runs.

2.1.0 - 2020-11-16

  1. Added a warning message when editing ACLs. The warning message can be configured via system properties.
  2. ExecuteACL() did not check if DataSilos filtering is globally active or not. This has been fixed.

2.0.0 - 2020-11-11

  1. WARNING: The property x_snc_datasilos.incident.whitelist_trigger_fields has essentially be renamed to x_snc_datasilos.tables.incident.columns.whitelist_trigger.sys_user_group! Make sure to redo the configuration during upgrade!
  2. WARNING: The property x_snc_datasilos.active now controls the filter ONLY. It will no longer halt the population of the whitelists. As a result, it can now be more effectively used as an ON/OFF switch while preparing a go-live or during trouble shooting - without any negative impact on the whitelists of records that are changed in the meantime.
  3. WARNING: It is now required to configure ACLs not only for table_name.*, but also for table_name, . Updated the "ACLs" module and tests accordingly.
  4. WARNING: The business rules are now part of the scoped application. Any existing "DataSilos" business rules in the global scope should be removed!
  5. DevTools version 1.3.0 or higher is required.
  6. Introduced the supervisor datasilo whitelist field to enable silo-based whitelists for supervisors.
  7. Introduced the global visibility flag for all tables.
  8. Introduced the ghost flag for all tables. If set to true whitelists are not updated when a record is changed.
  9. Introduced configurable group filter. An encoded query can now be configured to control which groups should be considered for whitelist population.
  10. Introduced the option to avoid expensive queries. When the system property x_snc_datasilos.tables.table_name.avoid_expensive_queries is set to true, queries that do not contain a filter for active records are not altered. This results in users seeing a message that some records have been filtered from the result set and eventually empty result pages.
  11. The test "DataSilos - incident - cross-silo" now impersonates a user of the current assignment group before re-assigning the incident. This change was made to consider strict customer-specific customizations.
  12. Several script includes have been opened up for use from different scopes to enable implementation of customer specific Maintenance() functions and business rules.
  13. Extended the "Business Rules" module to show all before query business rules for the supported tables.
  14. Optimized ExecuteACL() so that it does not evaluate the access restrictions for a single record multiple times.
  15. Introduced the use of customized test step configurations.
  16. Extended the Developer Guide: The DataSilos API Reference has been added and "Scope Extension" chapter has been refined.
  17. Moved all contents from the "Requirement Analysis" chapter into the "Implementation Guide", added process relevant considerations.

1.4.0 - 2020-07-30

  1. DevTools version must be 1.2.1.
  2. Existence and version of DevTools is now checked in ATF "DataSilos - installation".
  3. Introduced the global visibility flag for the incident table. If set to true, everyone can see this particular record.
  4. Reactivated the "Log" module.
  5. The module "Maintenance" has been replaced by the "Scheduled Jobs" module and shows all scheduled script executions where the name contains "DataSilos".
  6. The module "Business rules" now shows all business rules that contain "DataSilos" in their name - this is to consider customer's naming conventions with prefixes.
  7. A bug in the Madrid release caused issues in some navigation modules preventing pre-defined filters to work properly. This has been fixed.

1.3.0 - 2020-07-16

  1. Introduced the configurable whitelist_trigger_field option. By setting the table specific option (e.g. x_snc_datasilos.incident.whitelist_trigger_field) the fields that should be used to populate the whitelist can be explicitly configured. If the option is empty the default for each table based on the task table is the assignment_group field. However, any other field that contains a reference to a group can be used. Dot-walking is NOT yet supported, though.
  2. Introduced the maintenance.log_frequency option to control how often the maintenance script should log the current progress.
  3. Deactivated the Log module for the time being.

1.2.2 - 2020-07-15

  1. The Maintenance script now explicitly checks the related tables, too. So *task records which are not associated to other whitelisted records are now whitelisted based on their assignment groups.
  2. Improved logging in the Maintenance script: Only the start and the end of the execution (with a consolidated summary) is logged.
  3. Extended the developer guide with more details.
  4. Added the "Rollback" section to the implementation guide.
  5. Added a navigation module to show all silo to group associations.

1.2.1 - 2020-07-08

  1. Users can now access request items where the associated request is opened on their behalf.
  2. Fixed issues on access rights to the admin and management navigation modules.

1.2.0 - 2020-07-07

  1. Added support for Service Catalog Requests.
  2. Restructured the installation guide. It is now an implementation guide covering analysis, installation, configuration, and the verification phase.

1.1.3 - 2020-07-06

  1. Related items (e.g. problem_task) did not apply the whitelist to their parents when created. This has been fixed.

1.1.2 - 2020-07-04

  1. Added modules to show all supported tables with the fields relevant for visibility - a useful treat for data silo managers and developers.

1.1.1 - 2020-07-04

  1. Extended "DataSilos - Installation" test to check if all business rules and ACLs have been set up correctly.
  2. Updated developer guide to update the "DataSilos - Installation" test.
  3. Updated installation guide on how to deal with erroneously created ACLs in the application scope.
  4. Test "DataSilos - Maintenance" is now Madrid compatible

1.1.0 - 2020-07-03

  1. Added support for problem and problem_task tables.
  2. Users that are referenced in a change_request in the requested_by field can see that change request.
  3. Updated documentation: What happens if a user does not have any roles?

1.0.0 - 2020-06-23

First baselined version with support for incidents and change requests.