Page Comparison

Batch Container

kfs.BatchStepRunner

Starts a SpringContext, executes a Step, shuts down the SpringContext. This is the class the KFS uses to run Steps in batch. However, we don't want the batch processing

to start and stop KFS for every step in a

job. So the IU process will use BatchStepRunner, but we give it a step (BatchContainerStep) which continues to run until it's told to shut down. The BatchContainerStep handles executing Steps received from the

scripts/scheduler. This is called the 'Batch Container'.

Batch Container

BatchContainerStep

looks for .run files to process. When one is found it deletes the file and creates a new thread (BatchStepExecutor) which executes the Step indicated by the .run file. It continues looking for .run files until it finds a stopBatchContainerStep.run file at which point it exits. When BatchContainerStep begins it looks for a .runlock file and if one is found it will throw an exception. If one is not found it will write a .runlock file to indicate that the batch container is running. The BatchContainerStep will log the results of the Steps' executions when it exits.

Batch Container

BatchStepExecutor

executes a Step in its own Thread. It writes either a .success or .error file after execution. BatchStepExecutor notifies the BatchContainerStep when a step is about to start and after it completes.

Batch Container

ContainerStepListener

implementors of this class are notified when a step starts and completes.

Batch Container
Client

BatchContainerDirectory

knows the path to the directory for the BatchContainerStep and BatchStepTrigger semaphore files. It also handles much of the logic for writing, reading, and removing those files. The Batch Container and the Client each have a separate instance of the BatchContainerDirectory. For the Batch Container the path to the directory is set using the Spring property, staging.directory.sys.batchContainer, in configuration.properties. For the Client the path to the directory is also set using the staging.directory.sys.batchContainer property.

Batch Container
Client

BatchStepFileDescriptor

contains the properties of a batch step file (name of the job, name of the step, and the file type extension). The index of the step, the start date, and the completion date can also be stored here. These values are set by the BatchStepExecutor when it notifies the Batch Container.

Batch Container
Client

BatchLogger

provides a method for adding a Console Appender to a class' Logger. This will allow any logging statments for that class to write to the ConsoleAppender

. In the test environments the log4j configuration doesn't include the StdOut (ConsoleAppender) appender. The configuration for OLTP and Batch are controlled by the same log4j.properties file. We don't want logging to standard out in OLTP (which would go to tomcat's catalina.out log), but in batch it is useful to have log statements appear in the

batch logs.

Client

BatchStepTrigger

writes .run files containing a job name and step name for BatchContainerStep to read. It loops and sleeps until either a .success or an .error file is written for the Step. Once a result file is found it logs the results and exits. When the BatchStepTrigger begins it checks for BatchContainerStep's .runlock file. If it doesn't find one (indicating the batch container is not running) it exits. It also checks for the presence of a .runlock file after it returns from sleeping. Note that this class runs without starting the SpringContext. KFS Services and Beans are not available for use.

Client

BatchStepTriggerParameters

is a simple class to parse and validate the arguments passed to the BatchStepTrigger.

Upload

SemaphoreInputFileType

allows the batch semaphore files to be uploaded to the batch container directory. Currently configured to only upload .error files.

Developers

BatchStepLauncher

allows developers in the development environment to start the batch container and execute steps in one step. Use the StepRunner.launch file to execute the main method in this class.

kfs_batch_container.sh

• calls BatchStepRunner passing the step 'batchContainerStep'

BatchStepRunner.main('batchContainerStep')

• calls BatchContainerStep.execute()

BatchContainerStep.execute()

Exists?

*.runlock

BatchContainerStep

Write

kfs_batch_container.sh~batchContainerStep.runlock

BatchContainerStep

Polling

*.run

kfs_sample_job.sh

• calls BatchStepTrigger.main passing in jobname, stepname, stepindex, path to the batch container, and the semaphore processing interval

BatchStepTrigger.main(jobname,stepname,stepindex,path,interval)

Exists?

*.runlock

BatchStepTrigger

Write

jobname~stepname.run

BatchStepTrigger

Polling

jobname~stepname.success/.error

BatchContainerStep

Read

jobname~stepname.run

BatchContainerStep

Delete

jobname~stepname.run

BatchContainerStep

• spawns BatchStepExecutor(jobname,stepname,stepindex) in its own thread

BatchContainerStep

Polling

*.run

BatchStepExecutor.execute(jobname,stepname,stepindex)

• executes the step

BatchStepExecutor

Write

jobname~stepname.success/.error

BatchStepExecutor exits

BatchStepTrigger

Read

jobname~stepname.success/.error

BatchStepTrigger

• checks for success or failure, logs results, exits the system with status of 0, 4, or 8

BatchStepTrigger

Delete

jobname~stepname.success/.error

BatchStepTrigger exits

kfs_sample_job.sh succeeds or fails depending on exit status

kfs_stop_batch_container.sh

• calls BatchStepTrigger.main ('stopBatchContainerStep')

BatchStepTrigger.main('stopBatchContainerStep')

Exists?

*.runlock

BatchStepTrigger

Write

kfs_stop_batch_container.sh~stopBatchContainerStep.run

BatchStepTrigger

Polling

kfs_stop_batch_container.sh~stopBatchContainerStep.success/.error

BatchContainerStep

Read

kfs_stop_batch_container.sh~stopBatchContainerStep.run

BatchContainerStep

Delete

kfs_stop_batch_container.sh~stopBatchContainerStep.run

BatchContainerStep

• does not execute stopBatchContainerStep

BatchContainerStep

Write

kfs_stop_batch_container.sh~stopBatchContainerStep.success

BatchContainerStep

Delete

kfs_batch_container.sh~batchContainerStep.runlock

BatchContainerStep exits

BatchStepRunner exits

kfs_batch_container.sh succeeds

BatchStepTrigger

Read

kfs_stop_batch_container.sh~stopBatchContainerStep.success

BatchStepTrigger

Delete

kfs_stop_batch_container.sh~stopBatchContainerStep.success

BatchStepTrigger exits

kfs_stop_batch_container.sh succeeds

.runlock

This file is written by the Batch Container and read by both the Batch Container and the Client. It is used to indicate to the Client that the Batch Container is running. The absence of this file will cause the Client to fail. An instance of the Batch Container will not start if it finds an existing .runlock file (i.e. two Batch Containers pointing to the same directory cannot run at the same time.) The Batch Container removes this file when it shuts down.

.run

This file is written by the Client and read by the Batch Container. It is used to indicate to the Batch Container to execute the specified Step. The first line in this file contains the index of the Step in the Job.

.success

This file is written by the Batch Container and read by the Client. It is used to indicate to the Client that the execution of the Step was successful.

.error

This file is written by the Batch Container and read by the Client. It is used to indicate to the Client that the execution of the Step was unsuccessful. If the file is empty this indicates that the Step returned a false value. If the file is not empty this indicates that the execution of the Step threw an Exception. The Client will write the exception to the log and indicate a failure occurred.

SEMAPHORE_PROCESSING_INTERVAL

The amount of time (in milliseconds) which the BatchContainerStep should sleep per loop while waiting for run files

30,000

BatchContainerStep