|
|
| Author |
Message |
dbzTHEdinosauer
Senior Member
Joined: 20 Oct 2006 Posts: 1673 Location: germany
|
|
|
|
billiions of records is considered needed info when talking about rexx. rexx is not a high end i/o utility. you could compile your rexx.
you could build sort control statements
or
write a cobol/pl1/assembler module to process this data. |
|
| Back to top |
|
 |
References
|
|
 |
vidyaa
New User
Joined: 02 May 2008 Posts: 46 Location: chennai
|
|
|
|
| this rexx consists of panels and rexx codes, where the user input some values i need tovalidate each fields and throw message after validation take the values entered by the user and search if the combination of values entered by the user is present in a file which is going to have billion of records if the combination exists writes it to a file if not pops up an error message as 'Combination not found' and all these will be done online REXX validtions will be the backend so its not possible to put this in a batch mode.Searching of the combinations of field values entered by the user in the file containing billion of records takes me time. |
|
| Back to top |
|
 |
gcicchet
Senior Member
Joined: 28 Jul 2006 Posts: 691
|
|
|
|
Hi,
I can't believe anyone could come up with such a design, how often are these requests run ? how many users can request this ? what is the current time it takes to satisfy one of these requests ?
Surely the values entered by the user can be validated and saved into a file which can then be used in batch.
I would try to capture all requests into one file and run a batch job on a daily basis.
Gerry |
|
| Back to top |
|
 |
expat
Global Moderator
Joined: 14 Mar 2007 Posts: 3731 Location: Brussels once more ...
|
|
|
|
Gerry, stop it will you, coming up with sensible suggestions
What do you think this is, a help forum or something  |
|
| Back to top |
|
 |
dick scherrer
Global Moderator
Joined: 23 Nov 2006 Posts: 9191 Location: 221 B Baker St
|
|
|
|
Hello,
If the concept has been proven, it would be best to do as DBZ suggests and re-code the process in a better suited language.
REXX may be the worst choice for processing high volumes of data. . . |
|
| Back to top |
|
 |
Pedro
Senior Member
Joined: 01 Sep 2006 Posts: 632 Location: work
|
|
|
|
| Quote: |
| search if the combination of values entered by the user is present in a file which is going to have billion of records if the combination exists |
If this is searched regularly, I think the data is stored in the wrong format. There is not really a good way to search billions of records stored in a sequential format. It should be in a database, where it might take the reading of a few index blocks before finding (or not) the record. |
|
| Back to top |
|
 |
Craq Giegerich
Senior Member
Joined: 19 May 2007 Posts: 1027 Location: Virginia, USA
|
|
|
|
| Billions of records, Millions of records I think some of these record estimates are a little off. |
|
| Back to top |
|
 |
Pedro
Senior Member
Joined: 01 Sep 2006 Posts: 632 Location: work
|
|
|
|
| Quote: |
| my input has billons od records its very slow on doing this is there any way to make it better and faster. |
It is hard to believe that you got to billions of records without first realizing that there were performance problems. When you say 'billions', I am just assuming that you really meant "large number of records".
The way to speed it up is to not use a sequential file. Use VSAM with keyed records. I think you can get tools to process VSAM from rexx. (or as I said earlier, use a real database). |
|
| Back to top |
|
 |
vidyaa
New User
Joined: 02 May 2008 Posts: 46 Location: chennai
|
|
|
|
| yes i got your words intially it was thought to be few record and now it has come to billions. This is need to process online to get the user input and validate and throw him error messages. this cannot be changed to a program i guess. the seraching part kills time is there any way to incorporate this searching part in languale like cobol and combine with rexx as this needs to be done online. |
|
| Back to top |
|
 |
gcicchet
Senior Member
Joined: 28 Jul 2006 Posts: 691
|
|
|
|
Hi,
I want to be the user entering the data and waiting for the results.
I'll have to find ways of amusing myself.
Gerry |
|
| Back to top |
|
 |
dick scherrer
Global Moderator
Joined: 23 Nov 2006 Posts: 9191 Location: 221 B Baker St
|
|
|
|
Hello,
| Quote: |
| as this needs to be done online |
Sorry, but i very much disagree.
I believe what is needed is a smooth process for the user to get the needed answers in a timely fashion. That does not make the use of online resources a requirement. |
|
| Back to top |
|
 |
expat
Global Moderator
Joined: 14 Mar 2007 Posts: 3731 Location: Brussels once more ...
|
|
|
|
If it must be done in a wasteful and inappropriate manner then you will need to look at some of the ways to maybe waste a little less resource.
I guess that your shop does not bill each department on its usage of resource, because if they do there will probably be some very unhappy bean counters when the resource bills start rolling in.
If you can use a unique key, then using a VSAM KSDS accessed by IDCAMS within REXX should work. |
|
| Back to top |
|
 |
vidyaa
New User
Joined: 02 May 2008 Posts: 46 Location: chennai
|
|
|
|
| What i meant to say is the user only sees the front screen designed by panels he doesnot know what is happening in the backend rexx he corrects the input if he gets the error message, all the validations are done in rexx and throws the message |
|
| Back to top |
|
 |
expat
Global Moderator
Joined: 14 Mar 2007 Posts: 3731 Location: Brussels once more ...
|
|
|
|
What happens behind the panels are usually of no interest to the user, just the results.
So how you design / implement a package that delivers the results is of no consequence to the user, which gives you a free hand to search the best and most effective methods. |
|
| Back to top |
|
 |
Pedro
Senior Member
Joined: 01 Sep 2006 Posts: 632 Location: work
|
|
|
|
| Quote: |
| is there any way to incorporate this searching part in languale like cobol and combine with rexx as this needs to be done online. |
Yes, you can write the searching part in a compiled language. But a billion records is still a billion records! With a sequential file, I do not think you will ever have a satisfactory process for an online environment. It is the reading of the file that takes a long time and that does not change much regardless of rexx or compiled language. I stand by my recommendation to use VSAM or a database so that less I/O is performed.
A REXX program can call a compiled program to do part of the work. The compiled program can use ISPF VGET / VPUT services to pass information to REXX. |
|
| Back to top |
|
 |
|
|