|
View previous topic :: View next topic
|
| Author |
Message |
Amb
New User
Joined: 12 Mar 2010
Posts: 64
Location: India
|
|
|
|
I am trying to execute a REXX program which processes a huge file.
I am getting an error saying
"EXEC IO error. Unable to obtain storage. Machine storage exhausted."
This is a critical REXX program. Is there any way I can increase the storage capacity for this? |
|
| Back to top |
|
 |
prino
Senior Member
Joined: 07 Feb 2009
Posts: 1306
Location: Vilnius, Lithuania
|
|
|
|
Answer to last question: NO
General remark: You do not process huge files with REXX, ever! |
|
| Back to top |
|
|
Stefan
Active User
Joined: 12 Jan 2006
Posts: 110
Location: Germany
|
|
|
|
If it's really necessary to read a huge dataset with REXX you could try to read it "in sections". Put your EXECIO in a loop and specify a fixed number of records to be read per cycle, for example "EXECIO 100 DISKR ... ". Then process these 100 records and continue with reading the next 100 records into the same variables.
But in general I agree with prino that huge files should be processed by compiled programs.
Maybe you could pre-process your dataset with some kind of utility such as SORT or IDCAMS to extract only the data which is needed. So it might be possible to reduce the amount of data to be processed by your REXX program.
Give us an example of what your business needs are. |
|
| Back to top |
|
|
DB2 Guy
New User
Joined: 28 Oct 2008
Posts: 98
Location: Cubicle
|
|
|
|
| Quote: |
| But in general I agree with prino that huge files should be processed by compiled programs. |
I do not speak REXX much but does the above sentence mean if the REXX code is complied (using Change Man or any other version control tool), it is good to use REXX with huge file opposed to when it is used as interpreted-language? |
|
| Back to top |
|
|
enrico-sorichetti
Superior Member
Joined: 14 Mar 2007
Posts: 10873
Location: italy
|
|
|
|
the issue is about the improper use of EXECIO
the format
EXECIO * READ <yak yak yak>
will read into the stack or into a stem which means in storage
ALL the records of the file
irrelevant of the interpreted compiled mode
so if the file is huge an insufficient storage abend is most likely to occur |
|
| Back to top |
|
|
PeterHolland
Global Moderator
Joined: 27 Oct 2009
Posts: 2481
Location: Netherlands, Amstelveen
|
|
|
|
| Compiled programs means in this case anything BUT rexx. Like Cobol, PL/I, Assembler etc. Compiled Rexx means (a part of the) interpreting is compiled, storage handling will not be changed. |
|
| Back to top |
|
|
prino
Senior Member
Joined: 07 Feb 2009
Posts: 1306
Location: Vilnius, Lithuania
|
|
|
|
| DB2 Guy wrote: |
| I do not speak REXX much but does the above sentence mean if the REXX code is complied (using Change Man or any other version control tool), it is good to use REXX with huge file opposed to when it is used as interpreted-language? |
Neither REXX (nor any other language) is ever compiled by ChangeMan (or any other version control tool)! |
|
| Back to top |
|
|
PeterHolland
Global Moderator
Joined: 27 Oct 2009
Posts: 2481
Location: Netherlands, Amstelveen
|
|
|
|
| prino wrote: |
| DB2 Guy wrote: |
| I do not speak REXX much but does the above sentence mean if the REXX code is complied (using Change Man or any other version control tool), it is good to use REXX with huge file opposed to when it is used as interpreted-language? |
Neither REXX (nor any other language) is ever compiled by ChangeMan (or any other version control tool)! |
Give the guy a break, he means probably a changeman invoked compile process. Like invoking compiles/assemblings in ispf. |
|
| Back to top |
|
|
prino
Senior Member
Joined: 07 Feb 2009
Posts: 1306
Location: Vilnius, Lithuania
|
|
|
|
| PeterHolland wrote: |
| prino wrote: |
| DB2 Guy wrote: |
| I do not speak REXX much but does the above sentence mean if the REXX code is complied (using Change Man or any other version control tool), it is good to use REXX with huge file opposed to when it is used as interpreted-language? |
Neither REXX (nor any other language) is ever compiled by ChangeMan (or any other version control tool)! |
Give the guy a break, he means probably a changeman invoked compile process. Like invoking compiles/assemblings in ispf. |
Then he should say so. It's the same with people talking about REXX Panels. |
|
| Back to top |
|
|
Pedro
Global Moderator
Joined: 01 Sep 2006
Posts: 2547
Location: Silicon Valley
|
|
|
|
note: I agree that rexx is not a good choice for huge files.
| Quote: |
| the issue is about the improper use of EXECIO |
You can use EXECIO to read the file in chunks so that you possibly avoid exhausting the storage.
| Quote: |
| Is there any way I can increase the storage capacity for this? |
You can increase the region size when you logon or in the batch job. 'Huge' is a relative term. You might be able to get by just by increasing the region size to a huge value. |
|
| Back to top |
|
|
DB2 Guy
New User
Joined: 28 Oct 2008
Posts: 98
Location: Cubicle
|
|
|
|
Thank you Enrico and Peter.
Robert - Terminology is critical in IT, I know and understand that. Most of us (at least me ) are born when market was taken over by the tools such as Change-Man and/or endeavor. In my career, I've never executed, IGYCRCTL, to COMPILE components and then IEWL, to LINK-EDIT Component and possibly will never write my own JCL to do that, it's 2011. It's pretty common to say, "compiled using ChangeMan", around many shops I had been to.
If there is some compilation error in a COBOL-program compile, I won't talk about the under-the-cover-asssembler-code in error. I'll talk about one-level-up -- I'll talk about COBOL error. I'm compiling my programs using Change-man, I won't say it's copiled by IGYCRCTL+IEWL, I'll talk one level up - it's compiled using Change-Man.
We young musketeers are anyways, dealing with bad managers/leads, don't sound like them, please... |
|
| Back to top |
|
|
Kjeld
Active User
Joined: 15 Dec 2009
Posts: 365
Location: Denmark
|
|
|
|
| DB2 Guy wrote: |
| Quote: |
| But in general I agree with prino that huge files should be processed by compiled programs. |
I do not speak REXX much but does the above sentence mean if the REXX code is complied (using Change Man or any other version control tool), it is good to use REXX with huge file opposed to when it is used as interpreted-language? |
If you want to process large files by interpreted code I will suggest SAS, if it is available at the site. It will usually handle records one by one, and intermediate data will be written to external DASD storage. |
|
| Back to top |
|
|
valyk
Active User
Joined: 16 Apr 2008
Posts: 104
Location: South Carolina
|
|
|
|
| DB2 Guy wrote: |
Thank you Enrico and Peter.
Robert - Terminology is critical in IT, I know and understand that. Most of us (at least me ) are born when market was taken over by the tools such as Change-Man and/or endeavor. In my career, I've never executed, IGYCRCTL, to COMPILE components and then IEWL, to LINK-EDIT Component and possibly will never write my own JCL to do that, it's 2011. It's pretty common to say, "compiled using ChangeMan", around many shops I had been to.
If there is some compilation error in a COBOL-program compile, I won't talk about the under-the-cover-asssembler-code in error. I'll talk about one-level-up -- I'll talk about COBOL error. I'm compiling my programs using Change-man, I won't say it's copiled by IGYCRCTL+IEWL, I'll talk one level up - it's compiled using Change-Man.
We young musketeers are anyways, dealing with bad managers/leads, don't sound like them, please... |
I can totally sympathize with you. As a 'young muskateer' myself, I started programming on the mainframe several years ago. I started in the application area of my shop and quickly realized that I did not want to spend my career as an application programmer. Then I hopped at an opportunity to be a systems programmer.
Companies now-a-days do not care about training application programmers. They expect you to have a super high level understanding of what goes on, and are happy at leaving it at that.
If you want a good understanding of how things work, switch to a systems programmer job. You will not be disappointed. Granted, you will probably work your butt off, but you will feel like you are actually accomplishing something. |
|
| Back to top |
|
|
Victor Niu
New User
Joined: 11 Mar 2010
Posts: 59
Location: China,Shanghai
|
|
|
|
rexx works below 2G address space.
You need not deal with too huge data by rexx,I believe you can divide the data of the huge file into parts and handle one part each time，then one by one.
I have experienced the similar problem,if your pgm used over 1g memory and still increasing,it will be very dangerous.you check the pgm's overhead by RMF. |
|
| Back to top |
|
|
gcicchet
Senior Member
Joined: 28 Jul 2006
Posts: 1702
Location: Australia
|
|
|
|
Hi,
unless there is a requirement, don't use STEM
Read a record
Process the record
Write the record
and you won't have any storage issues.
I used the following code to process 11415828 records, FB and LRECL of 133.
| Code: |
DO WHILE (RETURN_CODE ¬= EOFFLAG)
"EXECIO 1 DISKR OLD"
RETURN_CODE = RC
IF RETURN_CODE = 0 THEN
DO
IN_CNTR = IN_CNTR + 1
PARSE PULL RECORD
OUT_CNTR = OUT_CNTR + 1;
QUEUE RECORD;
"EXECIO * DISKW NEW";
END;
END;
|
Now apart from this test I ran, I would never use REXX for large files, it took 26 mins CPU to process the above file.
I copied the same file using DFSORT in less than a minute.
Gerry |
|
| Back to top |
|
|
dbzTHEdinosauer
Global Moderator
Joined: 20 Oct 2006
Posts: 6966
Location: porcelain throne
|
|
|
|
Gerry,
do you run your test in foreground or background?  |
|
| Back to top |
|
|
gcicchet
Senior Member
Joined: 28 Jul 2006
Posts: 1702
Location: Australia
|
|
|
|
Hi Dick,
my test is in batch, I wouldn't lock up my session for that amount of time.
Gerry |
|
| Back to top |
|
|
|
|
