Job Expiry and Removal

By default, jobs remain in the database indefinitely and will continue to be available to download from the Manage Tables page.

Job Queue Manager offers two methods for periodically cleaning up the database to remove old jobs:

Expired job removal: jobs will automatically be removed a (configurable) amount of time after they have been downloaded.
Dead job removal: jobs will be removed if they have been in the same state for longer than a (configurable) period of time. You can use this feature to clean up jobs in any state (whereas the expired job removal feature only applies to jobs that have been downloaded).

If you are running multiple instances of Job Queue Manager against a single Job Queue Manager database, configure the scheduled tasks for job removal on one Job Queue Manager instance only.

If you configure the scheduled tasks across multiple Job Queue Manager instances, this will cause unnecessary concurrent database access and error messages in the log files.

Configure Expired Job Removal

To configure automatic removal of expired jobs, you first need to configure a global expiry time, and then activate the expiredJobRemover scheduled task.

The configured expiry time is saved to the database at the time the job is first downloaded (i.e., when the job STATUS in the database changes to COLLECTED).

If there are any existing jobs in the database that have already been downloaded at the time you set the job expiry configuration, then these jobs will not have an expiry time set and will not be removed when the expired job removal process runs. You will either need to manually clean up these jobs as required, or use the dead job removal feature described below.

Step 1 - Configure Job Expiry Time

This step sets the period of time after a job has been downloaded when it will be considered expired. The expiry time is shown to end users in the Expiry Time column on the Manage Tables page:

The expiry time is configured in SuperWEB2, rather than Job Queue Manager. If you are running multiple instances of SuperWEB2, this configuration needs to be set on every instance.

Go to the Tomcat instance that is running SuperWEB2 and open <tomcat_home>\webapps\webapi\WEB-INF\config\data-access-config-jqm.xml in a text editor.

Make a backup copy of this file before making any changes.

Locate the following line:

XML

<!-- <property name="globalJobExpiryPeriod" value="7 days" /> -->

Remove the comments:

XML

<property name="globalJobExpiryPeriod" value="7 days" />

Change the value to set the expiry time. The expiry time must be set in days and must include the word day or days. The following are some examples of valid expiry periods: 5 days, 1 day, 0.2 day, 7.3 days.

Jobs are not removed exactly on their expiry time: once a job has expired, it will be removed the next time the expiredJobRemover scheduled task runs. For example, if you set the expiry time to 7 days and configure the expiredJobRemover task to run once a day, then a job will be removed at some point between 7 and 8 days after it was downloaded. You will configure how frequently the scheduled task runs in the next step.
Save your changes to data-access-config-jqm.xml.
Restart the instance of Tomcat where you have made changes.
If you are running multiple SuperWEB2 instances, repeat the above steps to set the expiry time on each one.

Step 2 - Configure the Expired Job Removal Scheduled Task

Once you have configured an expiry period, you need to configure and activate the scheduled task to automatically remove expired jobs from the database.

Go to the Tomcat instance that is running Job Queue Manager and open <tomcat_home>\webapps\queuedjobservice\WEB-INF\config\common\service-config.xml in a text editor.

Make a backup copy of this file before making any changes.

Locate the following section:

XML

    <!-- Scheduled task to remove expired jobs in the COLLECTED state. The expiry time is saved in the database when
         queued. -->
    <!-- Example task to run every hour.
    <bean name="expiredJobRemover" class="au.com.str.jobqueue.server.scheduling.ExpiredJobRemoval.Factory">
        <property name="runFrequency" value="1h" />
        <property name="entityManagerFactory" ref="entityManagerFactory" />
        <property name="scheduler" ref="jobQueueManager" />
    </bean>
    -->

Remove the comments from around the <bean> ... </bean> element:

XML

    <!-- Scheduled task to remove expired jobs in the COLLECTED state. The expiry time is saved in the database when
         queued. -->

    <bean name="expiredJobRemover" class="au.com.str.jobqueue.server.scheduling.ExpiredJobRemoval.Factory">
        <property name="runFrequency" value="1h" />
        <property name="entityManagerFactory" ref="entityManagerFactory" />
        <property name="scheduler" ref="jobQueueManager" />
    </bean>

Edit the following line to specify how frequently you want the job removal task to run. Specify the frequency as a number followed by the first letter of days, hours, minutes or seconds. For example 1h sets it to run every hour, and 1d sets it to run every day.
XML
```
        <property name="runFrequency" value="1h" />
```
Save your changes to service-config.xml.
Restart the instance of Tomcat where you have made changes.

Configure Dead Job Removal

This feature allows you to configure Job Queue Manager to automatically remove jobs that have been in the same status for longer than a specific period of time.

Go to the Tomcat instance that is running Job Queue Manager and open <tomcat_home>\webapps\queuedjobservice\WEB-INF\config\common\service-config.xml in a text editor.

Make a backup copy of this file before making any changes.

Locate the following section:

XML

    <!-- Example task to delete any job that has been in the same status for over 30 days, running every hour.
    <bean name="deadJobRemover" class="au.com.str.jobqueue.server.scheduling.DeadJobRemoval.Factory">
        <property name="time" value="30d" />
        <property name="runFrequency" value="1h" />
        <property name="entityManagerFactory" ref="entityManagerFactory" />
        <property name="scheduler" ref="jobQueueManager" />
    </bean>
    -->

Remove the comments around the <bean> ... </bean> element:

XML

    <bean name="deadJobRemover" class="au.com.str.jobqueue.server.scheduling.DeadJobRemoval.Factory">
        <property name="time" value="30d" />
        <property name="runFrequency" value="1h" />
        <property name="entityManagerFactory" ref="entityManagerFactory" />
        <property name="scheduler" ref="jobQueueManager" />
    </bean>

Configure the values for time and runFrequency (specify the values as a number followed by the first letter of days, hours, minutes or seconds; for example 30d means every 30 days, and 1d means every day):
- time is the age of the job you want to clean up (for example in the above code block, 30d means any jobs that have been in the same status for 30 days).
- runFrequency is how frequently you want to scheduled task to run (for example in the above code block, 1h means that the task will run every hour).
The above example will apply to all jobs in any status. If you only want to clean up jobs with specific statuses, then add a statuses property with a comma separated list of status values that you want the rule to apply to. Possible status values are:
- SUBMITTED
- SCHEDULED
- RUNNING
- COMPLETE
- COLLECTED
- RECOVERABLE
- ERROR
For example, the following configuration will clean up jobs that have been in either the COLLECTED or ERROR statuses for 7 days, with the task running every hour:
XML
```
    <bean name="deadJobRemover" class="au.com.str.jobqueue.server.scheduling.DeadJobRemoval.Factory">
        <property name="statuses" value="COLLECTED, ERROR" />
        <property name="time" value="7d" />
        <property name="runFrequency" value="1h" />
        <property name="entityManagerFactory" ref="entityManagerFactory" />
        <property name="scheduler" ref="jobQueueManager" />
    </bean>
```

(Optionally): if you want to configure different rules for different statuses, you can include multiple instances of the dead job remover scheduled task: simply copy the <bean> ... </bean> element and give each instance a unique name. For example, the following configuration ensures that jobs that have been collected will be removed after 7 days, with all other jobs being removed after 30 days (both scheduled tasks run every hour):

XML

    <bean name="deadJobRemover" class="au.com.str.jobqueue.server.scheduling.DeadJobRemoval.Factory">
        <property name="time" value="30d" />
        <property name="runFrequency" value="1h" />
        <property name="entityManagerFactory" ref="entityManagerFactory" />
        <property name="scheduler" ref="jobQueueManager" />
    </bean>
	
    <bean name="collectedJobRemover" class="au.com.str.jobqueue.server.scheduling.DeadJobRemoval.Factory">
        <property name="statuses" value="COLLECTED" />
        <property name="time" value="7d" />
        <property name="runFrequency" value="1h" />
        <property name="entityManagerFactory" ref="entityManagerFactory" />
        <property name="scheduler" ref="jobQueueManager" />
    </bean>

Save your changes to service-config.xml.
Restart the instance of Tomcat where you have made changes.