Multiple jobs writing to same gdg. Best practice?

jasorn · Active User Joined: 12 Jul 2006 Posts: 109

I work in a shop which has several gdg bases that many jobs write to on a daily basis. A nightly batch job copies the gdg base, processes the copy, and deletes the generations at the end of the job.

This works fine except for the following scenario, which results in lost transactions.

1. Nightly batch job, JOBB, copies the gdg base, processes the copy, and deletes the gdg base at the end of the job, abends after the copy step.

2. While JOBB is down, JOBX, which creates a new generation that JOBB should process, runs and creates a new generation.

3. The abend for JOBB is resolved and JOBB is restarted from the step that abended. In other words doesn't process the new generation JOBX created while JOBB was down.

4. JOBB runs to eoj and thus deletes the unprocessed generation in it's delete step at the end of the job.

So far it's up to us to watch the abends and catch these situations and reprocess the missing transactions.

But I think there's got to be a better way. My idea is:

1. Change the copy step to a utility that will cause an abend if something goes wrong, unlike idcams which will just give a bad return code.

2. Code DISP=(OLD,DELETE,KEEP) on the copy step.

I think this should work as if JOBB abends and JOBX makes a new generation while JOBB is down, the new generation won't be processed by the restart of JOBB but it won't be deleted either. That way the next run of JOBB will pick of the generation JOBX made while JOBB was down.

And if JOBB is reran from the top instead of restarted, the new generation will be picked from the RESTART of JOBB.

This seems like such an obvious solution to me I'm guessing there must be something I'm missing or they would have coded it this way to begin with. So I'm asking here to see what you guys think.

What's the right way to handle this situation?

jasorn · Active User Joined: 12 Jul 2006 Posts: 109

I did search for this before posting. But sometimes my search skills aren't the best. If this is covered somewhere else, kindly point me in the right direction. Better yet, post where you searched and the keywords you used.

bijumon

Hi,

How is the GDG base specified in the JOB, JOBB , is it the current base of the GDG created by JOBX, or the entire GDG generation, please provide with more details...

Thanks & Regards,
----------------------
Biju

IQofaGerbil

Isn't this a scheduling problem?

William Thompson · Posted: Thu Aug 23, 2007 7:28 pm Post subject:

jasorn · Active User Joined: 12 Jul 2006 Posts: 109

jasorn · Active User Joined: 12 Jul 2006 Posts: 109

jasorn · Active User Joined: 12 Jul 2006 Posts: 109

dick scherrer · Posted: Fri Aug 24, 2007 7:34 am Post subject:

Hello,

In a similar situation, we ran a job that copied all of the generations to a "processing" dataset and deleted all of the cataloged versions before the process job ever started. It also cataloged a "new" empty generation.

The nightly process job was started and was able to run independent of the sales/distribution runs that created new generations as the gdg was never mentioned in the process job. We needed this because there were something like 422 sales/distribution centers and we could not control when they would complete their daily processing and upload the sales and shipping info.

Please let me know if i've not been clear.

jasorn · Active User Joined: 12 Jul 2006 Posts: 109

jasorn · Active User Joined: 12 Jul 2006 Posts: 109

[quote="dick scherrer"]Hello,

In a similar situation, we ran a job that copied all of the generations to a "processing" dataset and deleted all of the cataloged versions before the process job ever started. It also cataloged a "new" empty generation.
[quote]
But the same question applies, just for the job that does the copy, or no?

Errr, I forgot an important point

There are hundreds of jobs that write to the gdg bases in question. And, while a goal is to prevent generations from being overwritten as I've stated. The main goal is to avoid coding conflcts in ca-7. The reason is that we created and delete jobs rapidly and updating the scheduler with the conflicts is too burdensome. So, I guess the question of 'Is this a scheduling issue?' has merit.

Since throughput manager handles the dataset conflicts, I figured that coding disp=old,delete,keep would handle the restart situation and we'd not have to code the conflicts.

dick scherrer · Posted: Fri Aug 24, 2007 8:20 am Post subject:

Hello,

If the scheduling system is told to manage that particular dataset, it surely does look like a "scheduling issue".

What if that dataset was removed from being "scheduled"? Big-OZ will not let anything "bad" happen if the DISPs are proper for the job that initially copies all of the accumulated generations, deletes them, and catalogs the new "empty" starter generation. The worst thing we had happen was that one or more of the processes that needed to catalog the +1's had to wait until the pre-processor completed. We only let the jobs that created +1's run single thread anyway - they all had the same jobname. They were often a spin-off of a more involved process and existed only to copy the needed data to a new +1 of the common gdg.

jasorn · Active User Joined: 12 Jul 2006 Posts: 109

MtClimber

jasorn;
Try the "switch dataset" technique. Here's how it works.
In your JOBB that currently has 3 steps: copies, processes, and deletes the GDG files, add a step ahead of those that deletes a dummy file. Call it file A. After the copy, process and delete steps add a new last step to catalog that dummy file, file A. So JOBB will now have 5 steps.

Now add a new first step to JOBX that uses file A. The way this works is when JOBB starts, the first thing that happens is file A is deleted. If a JOBX starts, it will try to use file A and immediately abend with a dataset not found. JOBX will only run after the 5th step in JOBB runs to re-create file A. You can take all the time in the world to resolve problems with JOBB because JOBX will not run until that 5th step runs to re-create file A. All JOBX abends can easily be restarted from the top after JOBB finishes.

To get this all working, you will, one time, have to create a file A. You can catalog the dummy file A with IEBGENER with SYSUT1 dummied. For your delete step, I would recommend using IDCAMS with a DELETE A sysin card.

I hope this helps.
MtClimber

jasorn · Active User Joined: 12 Jul 2006 Posts: 109

I'm not sure how to read the fact that nobody addressed whether there is an issue with using the disp=old,delete,keep in the copy step and forgoing the delete step altogether. Not sure if that means everyone thinks it's problematic or if nobody's considered it?

All of the suggestions given here are good, appreciated, and ones we're considering but are more 'invloved' than changing the disp on the copy step and eliminating the delete step. This seems to work in the testing I've done so far but I wanted to see if anyone here knew if there were issues with that approach.