I'm interested to hear how other users are using Quantitative and Control Resources for job definitions. Some of the users at our company use simple numbers to indicate how many jobs can run against a machine (or an instance or a schema) but this doesn't take into account how much impact the job will actually have. What I'm saying is that, all jobs are not created equal. So I may be able to run 10 "small" jobs against the database or 2 "big" jobs but I'm not sure how to quantify this. Here is what we've been trying to do (it has not been entirely successful)....
for Java programs, we use a resource "memory@" that represents the amount of memory the JVM will request (in gigabytes). We also use a "cpu@DATABASE SERVER" resource to indicate how much of the database we're going to use. This is in terms of percentages of a CPU core. We usually set single-threaded processes to 50 or 100 (meaning 1/2 or 1 whole CPU).
The memory stuff is very easy to quantify and it's easy for users to define these. The cpu utilization is much harder to grasp. We also have issues where the usage footprint of a job changes. Some of our jobs run for 5, 10 or more hours. One job type in particular will read the database for 2 hours, then do batch side processing for 6 hours. It would be great if the QR usage of the job could change dynamically, but I think I'm just dreaming. In that case, the job would require X amount of resources from the database server for 2 hours, then Y amount from the batch server for 6 hours.
Is any of this possible? What techniques do you use to make quantitative resources effective for you?
How are you using Quantitative (and Control) Resources?
-
superwesman
- Nouveau
- Posts: 6
- Joined: 24 Jul 2009 12:00
-
superwesman
- Nouveau
- Posts: 6
- Joined: 24 Jul 2009 12:00
yes, I understand that Quantitative Resources can be used in this primitive way in order to limit the number of jobs executing. However, the question I'm seeking to ask is: are there more powerful ways that these can be used? Given that each job may have a vastly different physical resource consumption profile, limiting to X jobs seems basic and minimally effective.
-
superwesman
- Nouveau
- Posts: 6
- Joined: 24 Jul 2009 12:00
kaaaahn!
OK - I think your replies are reinforcing what I was thinking and that is that no one has dealt with managing real physical resources in a productive way. All the solutions I hear (even at BMC user world last year) are primitive - "define your job's resource consumption in terms of arbitrary name/value pairs that have no bearing on the server load and are difficult for your users to understand and define." - that's just not working for me.
In a response, specifically to ejtoedtli, what I'm really looking to do is develop a standard that all users in our company can use to define PHYSICAL resource consumption in meaningful ways.
When I tell a user that he needs to define the physical resource consumption of job X in terms of resource Y, how does he know how much to use? Does his job use 3 Y? or 12? As I said in the original post, for managing memory on the batch server for Java applications, it's quite easy. Set a maximum memory@BATCHSERVER that leaves a little room for the OS and have each job consume as much of the memory@ resource as the JVM's maximum heap size. Unfortunately, many of our jobs hit against one or more database servers and quantifying the impact to those servers based on job type is very difficult.
Yes, the control-m facility is QUANTITATIVE resources which, by definition, do not map directly to any physical resources. I'm trying to come up with a way that makes sense for users and also does something meaningful in terms of resource balancing. Defining a job's physical resource consumption (which is, from a load-balancing standpoint, what really matters) in terms of arbitrary name/value pairs is difficult.
OK - I think your replies are reinforcing what I was thinking and that is that no one has dealt with managing real physical resources in a productive way. All the solutions I hear (even at BMC user world last year) are primitive - "define your job's resource consumption in terms of arbitrary name/value pairs that have no bearing on the server load and are difficult for your users to understand and define." - that's just not working for me.
In a response, specifically to ejtoedtli, what I'm really looking to do is develop a standard that all users in our company can use to define PHYSICAL resource consumption in meaningful ways.
When I tell a user that he needs to define the physical resource consumption of job X in terms of resource Y, how does he know how much to use? Does his job use 3 Y? or 12? As I said in the original post, for managing memory on the batch server for Java applications, it's quite easy. Set a maximum memory@BATCHSERVER that leaves a little room for the OS and have each job consume as much of the memory@ resource as the JVM's maximum heap size. Unfortunately, many of our jobs hit against one or more database servers and quantifying the impact to those servers based on job type is very difficult.
Yes, the control-m facility is QUANTITATIVE resources which, by definition, do not map directly to any physical resources. I'm trying to come up with a way that makes sense for users and also does something meaningful in terms of resource balancing. Defining a job's physical resource consumption (which is, from a load-balancing standpoint, what really matters) in terms of arbitrary name/value pairs is difficult.
I am with you on this superwesman.
Yes it is not the right use the resources until or unless we know the exact resource utilization by the particular job. I would think that having a sys admin rerun the process in a test environment to capture an approx value and then defining the quatitative resource. But then, I also understand this is far from being a practically viable option.
There are tools out there in the market that tells us exactly the usage of a process. Creating a job in control-m to run it for a week, analysing the data and then working on modifying the resources seems a better and less painful process. That said, I shouls also admit I haven't actually implemented this.
Just my 2 cents
Yes it is not the right use the resources until or unless we know the exact resource utilization by the particular job. I would think that having a sys admin rerun the process in a test environment to capture an approx value and then defining the quatitative resource. But then, I also understand this is far from being a practically viable option.
There are tools out there in the market that tells us exactly the usage of a process. Creating a job in control-m to run it for a week, analysing the data and then working on modifying the resources seems a better and less painful process. That said, I shouls also admit I haven't actually implemented this.
Just my 2 cents