Open Data API Cache and the Data Control API

By default, the Open Data API cache uses the structure of the table as the basis for determining when to return results from the cache. A table with the same explicit structure will return the same results (as long as the user requesting the table has permission to access all of the elements in the table, such as the fields and field items).

If you are using Data Control API modules that change the table results based on the requesting user or group, then the default caching behaviour will incorrectly result in all requests for a specific table returning the same results as the first user who requested that table.

To resolve this, you need to add and configure the cacheDeterminism property on the Data Control module. This has the following available properties:

full

none

group:<key_to_id_map>

user:<key_to_id_map>

Specifying the Key to ID Mappings for Groups or Users

The <key_to_id_map> allows you to specify one or more groups or individual users who will share the same cache results. It takes the following format:

<key1>=<id1>|<id2>|<id3>,<key2>=<id4>|<id5>...

Where:

• The keys (in this example <key1> and <key2>) are arbitrary names for each collection of groups or users that share a set of cache results for a given table.
• The IDs are the IDs of the groups or users.

Use the | separator between each group or user ID in the list, and use , between each set of key/ID pairs.

IDs of users and groups will be compared ignoring case, but the key values are case sensitive.

If you need to specify group or user IDs that contain either the | or , characters, then use the \ character to escape these characters and force the following character to be treated as part of the ID.

Example of Group Keys

In this example any users who are either in the group Guest or the group UserReg will share a set of cache results for this dataset. A separate cache result will be returned to users who are not in either of those two groups.

group:Restricted=Guest|UserReg

Example of User Keys

The user version works in the same way, except using user IDs rather than group IDs. For example:

user:Restricted=jbrown|rwood|rstarr,Internal=mpalin|eidle|tjones

Multiple Group Matches

If you specify multiple keys with groups, then it is possible that a user can match more than one key if they are in multiple groups. Each unique combination of matches defines a unique cache entry for the same table.

For example:

group:Restricted=Guest|UserReg,Internal=analysts,Admin=administrators

In this scenario, there will be unique cache entries for all the possible combinations of matches.

So, for example, separate unique caches will be maintained for:

• Users who are only in the group UserReg
• Users who are in both UserReg  and analysts
• Users who are in all three of UserReg, analysts  and administrators

And so on for all possible combinations.

Example Configuration in SuperADMIN

To configure the cache settings, use SuperADMIN to add the cacheDeterminism property to your Data Control module like any other property. You can either add this as a common property to the method in SuperADMIN, or as a property on an individual Data Control plugin on the method.

The following example adds group-based caching as a common property on a Data Control method called MyDataControlMethod. 

> method MyDataControlMethod common addproperty cacheDeterminism "group:Restricted=Guest|UserReg"

Manually Clear the Cache if Group Memberships Change

If you make changes to group memberships, then you must manually clear the cache. Follow the steps in Configure the Cache. You must use an API key associated with an administrator account to clear the cache. 

Use Logging to Verify Configuration

If you want to verify that your configuration is correct, you can configure logging so that you can see the effects of your configuration in the SuperWEB2 log files.

You may wish to turn off the logging once you have verified the configuration to reduce the amount of information recorded in the log files, as the cache configuration will otherwise be logged for every table request to the Open Data API. 

Configure Logging

To enable logging:

1. Open <tomcat_home>\webapps\webapi#rest#v1\WEB-INF\classes\log4j2.xml in a text editor.
2. Locate the <Loggers> ... </Loggers> section.

3. Add one of the following inside this section:

   CODE

   <Logger name="au.com.str.webapi.webcontent.database.catalogue.CatalogueMethodInfo" level="DEBUG"/>

   Or:

   CODE

   <Logger name="au.com.str.webapi.webcontent.database.catalogue.CatalogueMethodInfo" level="INFO"/>

   The logging levels are as follows:

   • DEBUG will record all messages related to parsing of the configuration as well as the details of the final cache selection. This is recommended only for debugging, if something in the results seems incorrect.
   • INFO records only details of the final cache selection. 
4. Save your changes and restart Tomcat or the SuperWEB2 service.

For example, with the following configuration:

> method MyDataControlMethod common addproperty cacheDeterminism "group:Restricted=Guest|UserReg"

A message similar to the following would be recorded at the INFO level:

INFO : 2022-10-20T15:53:00,843--[main]-- - Created for method 'MyDataControlMethod': Keyed[Method='MyDataControlMethod', Module='null', 'restricted'={group in 'guest', 'userreg'}] --CatalogueMethodInfo

If there are no details recorded in this section of the log output then that indicates either that the cacheDeterminism property has not been configured, or it has been configured incorrectly.