Batch Container

In addition to the OLTP scheduling framework delivered with KFS, a mechanism for managing a separate batch container and triggering jobs from an external scheduler is provided.

Reference

Used By	Classes	Description
Batch Container	kfs.BatchStepRunner	Starts a SpringContext, executes a Step, shuts down the SpringContext. This is the class the KFS uses to run Steps in batch. However, we don't want the batch processing to start and stop KFS for every step in a job. So the IU process will use BatchStepRunner, but we give it a step (BatchContainerStep) which continues to run until it's told to shut down. The BatchContainerStep handles executing Steps received from the scripts/scheduler. This is called the 'Batch Container'.
Batch Container	BatchContainerStep	looks for .run files to process. When one is found it deletes the file and creates a new thread (BatchStepExecutor) which executes the Step indicated by the .run file. It continues looking for .run files until it finds a stopBatchContainerStep.run file at which point it exits. When BatchContainerStep begins it looks for a .runlock file and if one is found it will throw an exception. If one is not found it will write a .runlock file to indicate that the batch container is running. The BatchContainerStep will log the results of the Steps' executions when it exits.
Batch Container	BatchStepExecutor	executes a Step in its own Thread. It writes either a .success or .error file after execution. BatchStepExecutor notifies the BatchContainerStep when a step is about to start and after it completes.
Batch Container	ContainerStepListener	implementors of this class are notified when a step starts and completes.

Batch Container Client	BatchContainerDirectory	knows the path to the directory for the BatchContainerStep and BatchStepTrigger semaphore files. It also handles much of the logic for writing, reading, and removing those files. The Batch Container and the Client each have a separate instance of the BatchContainerDirectory. For the Batch Container the path to the directory is set using the Spring property, `staging.directory.sys.batchContainer`, in configuration.properties. For the Client the path to the directory is also set using the `staging.directory.sys.batchContainer` property.
Batch Container Client	BatchStepFileDescriptor	contains the properties of a batch step file (name of the job, name of the step, and the file type extension). The index of the step, the start date, and the completion date can also be stored here. These values are set by the BatchStepExecutor when it notifies the Batch Container.
Batch Container Client	BatchLogger	provides a method for adding a Console Appender to a class' Logger. This will allow any logging statments for that class to write to the ConsoleAppender. In the test environments the log4j configuration doesn't include the StdOut (ConsoleAppender) appender. The configuration for OLTP and Batch are controlled by the same log4j.properties file. We don't want logging to standard out in OLTP (which would go to tomcat's catalina.out log), but in batch it is useful to have log statements appear in the batch logs.

Client	BatchStepTrigger	writes .run files containing a job name and step name for BatchContainerStep to read. It loops and sleeps until either a .success or an .error file is written for the Step. Once a result file is found it logs the results and exits. When the BatchStepTrigger begins it checks for BatchContainerStep's .runlock file. If it doesn't find one (indicating the batch container is not running) it exits. It also checks for the presence of a .runlock file after it returns from sleeping. Note that this class runs without starting the SpringContext. KFS Services and Beans are not available for use.
Client	BatchStepTriggerParameters	is a simple class to parse and validate the arguments passed to the BatchStepTrigger.

Upload	SemaphoreInputFileType	allows the batch semaphore files to be uploaded to the batch container directory. Currently configured to only upload .error files.
Developers	BatchStepLauncher	allows developers in the development environment to start the batch container and execute steps in one step. Use the StepRunner.launch file to execute the main method in this class.

Execution

java -classpath ./kfs-web/target/kfs-web/WEB-INF/lib/:/usr/local/apache-tomcat-7.0.63/lib/ -Dadditional.kfs.config.locations=~/home/config/financials/kfs-mysql-dev.properties org.kuali.kfs.sys.context.BatchStepRunner batchContainerStep

Process Flow

Batch Container	Client	Action	Batch Container Directory Contents
kfs_batch_container.sh calls BatchStepRunner passing the step 'batchContainerStep'
BatchStepRunner.main('batchContainerStep') calls BatchContainerStep.execute()
BatchContainerStep.execute()		Exists?	*.runlock
BatchContainerStep		Write	kfs_batch_container.sh~batchContainerStep.runlock
BatchContainerStep		Polling	*.run

	kfs_sample_job.sh calls BatchStepTrigger.main passing in jobname, stepname, stepindex, path to the batch container, and the semaphore processing interval
	BatchStepTrigger.main(jobname,stepname,stepindex,path,interval)	Exists?	*.runlock
	BatchStepTrigger	Write	jobname~stepname.run
	BatchStepTrigger	Polling	jobname~stepname.success/.error

BatchContainerStep		Read	jobname~stepname.run
BatchContainerStep		Delete	jobname~stepname.run
BatchContainerStep spawns BatchStepExecutor(jobname,stepname,stepindex) in its own thread
BatchContainerStep		Polling	*.run

BatchStepExecutor.execute(jobname,stepname,stepindex) executes the step
BatchStepExecutor		Write	jobname~stepname.success/.error
BatchStepExecutor exits

	BatchStepTrigger	Read	jobname~stepname.success/.error
	BatchStepTrigger checks for success or failure, logs results, exits the system with status of 0, 4, or 8
	BatchStepTrigger	Delete	jobname~stepname.success/.error
	BatchStepTrigger exits
	kfs_sample_job.sh succeeds or fails depending on exit status

	kfs_stop_batch_container.sh calls BatchStepTrigger.main ('stopBatchContainerStep')
	BatchStepTrigger.main('stopBatchContainerStep')	Exists?	*.runlock
	BatchStepTrigger	Write	kfs_stop_batch_container.sh~stopBatchContainerStep.run
	BatchStepTrigger	Polling	kfs_stop_batch_container.sh~stopBatchContainerStep.success/.error

BatchContainerStep		Read	kfs_stop_batch_container.sh~stopBatchContainerStep.run
BatchContainerStep		Delete	kfs_stop_batch_container.sh~stopBatchContainerStep.run
BatchContainerStep does not execute stopBatchContainerStep
BatchContainerStep		Write	kfs_stop_batch_container.sh~stopBatchContainerStep.success
BatchContainerStep		Delete	kfs_batch_container.sh~batchContainerStep.runlock
BatchContainerStep exits
BatchStepRunner exits
kfs_batch_container.sh succeeds
	BatchStepTrigger	Read	kfs_stop_batch_container.sh~stopBatchContainerStep.success
	BatchStepTrigger	Delete	kfs_stop_batch_container.sh~stopBatchContainerStep.success
	BatchStepTrigger exits
	kfs_stop_batch_container.sh succeeds

Semaphore Files Types found in the Batch Container Directory - Naming Format: JobName~StepName.FileTypeExtension

.runlock	This file is written by the Batch Container and read by both the Batch Container and the Client. It is used to indicate to the Client that the Batch Container is running. The absence of this file will cause the Client to fail. An instance of the Batch Container will not start if it finds an existing .runlock file (i.e. two Batch Containers pointing to the same directory cannot run at the same time.) The Batch Container removes this file when it shuts down.
.run	This file is written by the Client and read by the Batch Container. It is used to indicate to the Batch Container to execute the specified Step. The first line in this file contains the index of the Step in the Job.
.success	This file is written by the Batch Container and read by the Client. It is used to indicate to the Client that the execution of the Step was successful.
.error	This file is written by the Batch Container and read by the Client. It is used to indicate to the Client that the execution of the Step was unsuccessful. If the file is empty this indicates that the Step returned a false value. If the file is not empty this indicates that the execution of the Step threw an Exception. The Client will write the exception to the log and indicate a failure occurred.

Parameters

Name	Description	Value	Used By
SEMAPHORE_PROCESSING_INTERVAL	The amount of time (in milliseconds) which the BatchContainerStep should sleep per loop while waiting for run files	30,000	BatchContainerStep

Spring Configuration Properties

Property	Description	Value	Used By
staging.directory.sys.batchContainer	The full path to the batch container directory.	${staging.directory}/sys/batchContainer	Injected into the BatchContainerStep and SemaphoreInputFileType beans when the KFS' SpringContext begins.
step.status.check.interval	The amount of time (in seconds) which the BatchStepTrigger should sleep per loop while waiting for a result file.	10	Used by run-step-trigger target

Batch Semaphore File Upload

Secured by the KFS-SYS Technical Administrator role
Currently configured to only upload error files, but can easily be configured to upload the other file types
To upload an error file for a Step create a new text file and name it whatever you like. The first line of the file should contain the name of the job and the the name of the Step (the bean id) in the format jobName~stepName. This line and all subsequent lines in the file will be considered error messages and will be logged by BatchStepTrigger.
The File Identifier is required by the upload page, but is not used so enter anything.
Once the file is saved to the batch container directory its name will be jobName~stepName.error.
A new Permission type has been created to handle securing access for downloading and deleting the semaphore files in the batch container directory. Permission has been given to the KFS-SYS Technical Administrator role.

Kuali documentation is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 License.

Kuali software is licensed for use pursuant to the Affero General Public License, version 3.

Copyright © 2014 Kuali, Inc. All rights reserved.

Portions of Kuali are copyrighted by other parties as described in the Acknowledgments screen.

Kuali ® is a registered trademark of the Trustees of Indiana University.