Unicode - SuperSTAR

From version 9.0 onwards, SuperSTAR includes full Unicode support. This means that SuperSERVER and the SuperSTAR clients fully support all characters and languages, and SXV4 databases are no longer restricted to the character set supported by the user's system codepage or locale.

Upgrading from Previous Releases

If you are upgrading from an earlier version of SuperSTAR, then there are some things you need to be aware of if you plan to use files created with an earlier version.

SXV4s

SXV4s created with versions 8.0 and above will work automatically regardless of what character set or language they use.

SXV4s created prior to version 8.0 may not work automatically if they contain non English/ASCII characters. To use these SXV4s you either need to make sure the system locale matches the character set of the SXV4 or set an appropriate codepage for the relevant SuperSTAR component.

In many cases, no changes will be necessary. If you were using the same SXV4s with a previous version of SuperSTAR then it is likely that you have already implemented the appropriate locale or codepage setting on your system (in previous versions, this was required for all SXV4s; now it is only required to support older SXV4s). You will only need to make changes if you now want to use older SXV4s that do not match the system locale or codepage.

SuperSERVER	To add the SXV4 to SuperSERVER, you need to do one of the following: Set the system locale of the machine running SuperSERVER to one that supports all the characters used in the SXV4; or Use the -c command line option when starting SuperSERVER to set the SuperSERVER codepage to match the SXV4. See Start SuperADMIN and SuperSERVER for more details.
SuperCROSS local mode	To use the SXV4 in SuperCROSS local mode (open the SXV4 from the Local Access section of the database catalogue), you need to do one of the following: Set the system locale of the machine running SuperCROSS to one that supports all the characters used in the SXV4; or Change the SuperCROSS legacy codepage setting to match the SXV4. You can access and edit this option in SuperCROSS as follows: Select Edit > Options. Select the Configuration tab. Select Legacy from the drop-down list. Select the Codepage for local old sxv4 databases variable and enter your chosen codepage in the Value field. Click OK. These steps only apply when using local mode. You do not need to change any SuperCROSS settings to access old SXV4s via SuperSERVER. You just need to make sure that SuperSERVER is using the appropriate system locale or codepage.
Production System local mode	To use the SXV4 with Production System local mode, you need to do one of the following: Set the system locale of the machine running Production System to one that supports all the characters used in the SXV4; or Configure the LocalLegacySXV4Codepage setting in the Production System configuration file to a codepage that matches the SXV4. If you are using remote mode, you do not need to change any Production System settings to use the SXV4. You just need to make sure that SuperSERVER is using the appropriate system locale or codepage.
SuperCHANNEL	If you are using the SXV4 as a source in SuperCHANNEL, or you are updating the SXV4, then you will need to set the system locale of the machine running SuperCHANNEL to one that supports all the characters used in the SXV4.

TXDs

You can continue to use old Textual Table Definition (TXD) files with this release. In most cases it is likely that these will work automatically:

If your TXDs contain characters not supported by the current system locale/codepage (SuperCROSS) or the configured codepage (SuperWEB2), then you will need to convert them.

If you were using the same TXDs with a previous version of SuperSTAR then it is very likely that they only contain characters supported by the current system locale/codepage, as this was a requirement to use these files in previous releases. If this is the case then they will work automatically without conversion (if you are using SuperWEB2 you should check that you have set an appropriate value for the TXDEncodingType in web.xml; see Add Datasets and Tables - SuperWEB2 for more information).

If you need to convert your TXDs, you will need to change them to UTF-8 encoding with a Byte Order Mark (BOM).

One way to do this is to use a text editor such as Notepad++. Open the TXD file in Notepad++ and select Encoding > Convert to UTF-8.

Macro Files

You can continue to use your existing SuperADMIN macro files with this release.

SuperADMIN will attempt to automatically detect the encoding of the macro file. However, in some cases this may not be possible. If this is the case then SuperADMIN displays the following message:

Encoding with BOM can be used to ensure correct encoding detection if needed.

If you see this message, then you are recommended to convert the encoding of your macro files to UTF-8 with a Byte Order Mark (BOM). One way to do this is to use a text editor such as Notepad++. Open the macro file in Notepad++ and select Encoding > Convert to UTF-8.

User Accounts

There is a potential issue with user accounts that may prevent some users from logging in following an upgrade to version 9. This issue arises if users were created in a previous version of SuperSTAR and had non ASCII characters in their passwords.

This is caused by a change in the way SuperADMIN hashes (encrypts) the passwords: in version 8, SuperADMIN used the configured character encoding to hash user passwords (by default this was Windows 1252 encoding). In version 9, SuperADMIN uses UTF-8 encoding to hash passwords. For this reason, if an existing user has a password that contains non ASCII characters, that user will not be able to log in following the upgrade to version 9 (because when SuperADMIN hashes the password supplied by the user, it will not match the saved hash of the password).

If you encounter this issue, you have two options:

• Individually reset the passwords of any affected users, using the command account <user_id> setpassword in SuperADMIN (see the documentation for the account command for more information); or
• Modify the server.bat script that starts SuperADMIN server, forcing it to use the same encoding as you were using with the previous release:
  1. Open server.bat in a text editor (by default it is located at C:\ProgramData\STR\SuperADMIN\server\server.bat).

  2.  Add the argument -Dstr.superadmin.passwordHash.encoding=windows-1252 to the Java command that launches SuperADMIN. You will need to replace windows-1252 with the encoding you were using in your existing deployment.

     For example, update:

     CODE

     ...
         "%JAVA_CMD%" ^
          "-Dstr.config.server.jasdb.location=%JASDB_HOME%" ^
          "-Dpidfile.path=%SA_PROGRAM_DATA%\data\play\play.pid" ^
     ...

     To:

     CODE

     ...
         "%JAVA_CMD%" ^
          "-Dstr.superadmin.passwordHash.encoding=windows-1252" ^
          "-Dstr.config.server.jasdb.location=%JASDB_HOME%" ^
          "-Dpidfile.path=%SA_PROGRAM_DATA%\data\play\play.pid" ^
     ...

  3. Save your changes and restart the SuperADMIN server.

Where possible, the first option (resetting the password) is preferable, otherwise you will need to reapply the change to the SuperADMIN encoding every time you upgrade SuperADMIN in future.

This issue only affects users with non ASCII characters in their passwords. User accounts with passwords that only contain English alphanumeric characters are not affected by the issue and can continue logging in using their existing credentials following the upgrade.

Saving Files in Version 9.0

SuperCROSS

When saving output, SuperCROSS now uses UTF-8 (Unicode) by default.

For some of the text based output formats SuperCROSS gives you the option to select the encoding of the output file. This is designed to allow you to save output in the same encoding as previously, and to help you if you need to use the output file with another application that does not support Unicode.

See Output Formats - SuperCROSS for more information.

From this release onwards, SRD (SuperCUBE) files support Unicode. However, if you save an SRD that uses any non ASCII characters then it will not be possible for users to load that SRD into earlier versions of SuperCROSS or SuperTABLE. To ensure full backwards compatibility of your SRD files, make sure they contain ASCII characters only.

Production System

When saving output, Production System uses UTF-8 (Unicode) by default.

For CSV output and Record View you can choose to set a specific codepage if necessary. Set the value of the CodePage property in the Production System configuration file. If this is left blank then Production System defaults to UTF-8.

SuperWEB2

If you are likely to have databases and tables with non ASCII characters (non English language characters) in their names and IDs, then you may need to make a configuration change to the Tomcat server.

This is because some SuperWEB2 features (including the system direct URLs feature) use the database and table IDs in the page URL. If your deployment uses non ASCII characters then you will need to ensure that Tomcat is configured to use UTF-8 encoding for URLs. See Use Direct URLs to Open a Specific Dataset or Table - SuperWEB2 for more information.

Unicode and SuperCHANNEL

SuperCHANNEL supports Unicode channelling when using general JDBC drivers, as well as the DDI and SXV4 drivers. That means your source data can contain any characters or languages and you will be able to convert it to SXV4 format for use with SuperSERVER.

The following drivers do not support Unicode:

• The supplied Komodo driver (this is provided with SuperCHANNEL for converting the older, deprecated, SXV2 and SXV3 formats into SXV4).
• The jdbc:odbc bridge (used for connecting to sources such as Excel or Access).

There may be other JDBC drivers available for connecting to Excel or Access that do support Unicode, although these have not been tested by Space-Time Research. We strongly recommend using a relational database to store your source data rather than Excel or Access.

Unicode and the TDD Driver

The supplied TDD driver (used for connecting to textual database formats) supports Unicode channelling. By default, this driver will use the system codepage (e.g., Windows 1252). To channel Unicode data, you must must edit the following setting in the SuperCHANNEL config.txt configuration file:

# str.sctextdriver.encoding=UTF-8

Remove the comment character at the start of the line, and save your changes:

str.sctextdriver.encoding=UTF-8

If you do not enable UTF-8 encoding in the SuperCHANNEL configuration, then you must ensure that your TDD files use the same encoding as the system codepage.

For Unicode using UTF-8, UTF BOM characters are not supported. You must ensure that the files in your TDD do not contain a Byte Order Mark.