[cf-dev] Understanding hard CPU limits

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[cf-dev] Understanding hard CPU limits

Grifalconi, Michael
Hi,

We were trying out the hard CPU limit as per docs
https://github.com/cloudfoundry/garden-runc-release/releases?after=v1.9.2

according to formula for single core machine,

APP_MEM_IN_MB * 100us / 1000 = MILLISECONDS_PER_100_MILLISECOND_PERIOD

In our tests, to get 6.4% CPU usage on a 64MB application and ~100% for a 1GB application we had to set the 'cpu_quota_per_share_in_us' to 3200. (Cell has 4 core and 16GB or ram, overcommit factor of 2).

That changes the formula to:
APP_MEM_IN_MB * 100us / 32000 = MILLISECONDS_PER_100_MILLISECOND_PERIOD

Can you help us to understand from where this 'times 32' comes from? Is the total available RAM of the cell (16GB * overcommit of 2) and number of cpu cores does not matter?

Thanks and regards,
Michael


-=-=-=-=-=-=-=-=-=-=-=-
Links:

You receive all messages sent to this group.

View/Reply Online (#7890): https://lists.cloudfoundry.org/g/cf-dev/message/7890
View All Messages In Topic (1): https://lists.cloudfoundry.org/g/cf-dev/topic/16273332
Mute This Topic: https://lists.cloudfoundry.org/mt/16273332/474226
New Topic: https://lists.cloudfoundry.org/g/cf-dev/post

Change Your Subscription: https://lists.cloudfoundry.org/g/cf-dev/editsub/474226
Group Home: https://lists.cloudfoundry.org/g/cf-dev
Contact Group Owner: [hidden email]
Terms of Service: https://lists.cloudfoundry.org/static/tos
Unsubscribe: https://lists.cloudfoundry.org/g/cf-dev/leave/920759/1741049355/xyzzy
-=-=-=-=-=-=-=-=-=-=-=-

Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Understanding hard CPU limits

Marco Voelz


/cc Eric and Julz: Could you maybe help us understand this? Thanks!


From: [hidden email] <[hidden email]> on behalf of Grifalconi, Michael <[hidden email]>
Sent: Monday, March 26, 2018 10:54:32 AM
To: [hidden email]
Subject: [CAUTION] [cf-dev] Understanding hard CPU limits
 
Hi,

We were trying out the hard CPU limit as per docs
https://github.com/cloudfoundry/garden-runc-release/releases?after=v1.9.2



according to formula for single core machine,

APP_MEM_IN_MB * 100us / 1000 = MILLISECONDS_PER_100_MILLISECOND_PERIOD

In our tests, to get 6.4% CPU usage on a 64MB application and ~100% for a 1GB application we had to set the 'cpu_quota_per_share_in_us' to 3200. (Cell has 4 core and 16GB or ram, overcommit factor of 2).

That changes the formula to:
APP_MEM_IN_MB * 100us / 32000 = MILLISECONDS_PER_100_MILLISECOND_PERIOD

Can you help us to understand from where this 'times 32' comes from? Is the total available RAM of the cell (16GB * overcommit of 2) and number of cpu cores does not matter?

Thanks and regards,
Michael




_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#7894) | [hidden email] | [hidden email] | Mute This Topic | New Topic

Change Your Subscription
Group Home
[hidden email]
Terms Of Service
Unsubscribe From This Group

_._,_._,_
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Understanding hard CPU limits

Dieu Cao
I believe Julz is on vacation this week.
Adding Ed King, the anchor on the Garden team.

Dieu

On Tue, Apr 3, 2018, 3:08 AM Marco Voelz <[hidden email]> wrote:


/cc Eric and Julz: Could you maybe help us understand this? Thanks!


From: [hidden email] <[hidden email]> on behalf of Grifalconi, Michael <[hidden email]>
Sent: Monday, March 26, 2018 10:54:32 AM
To: [hidden email]
Subject: [CAUTION] [cf-dev] Understanding hard CPU limits
 
Hi,

We were trying out the hard CPU limit as per docs
https://github.com/cloudfoundry/garden-runc-release/releases?after=v1.9.2
GitHub is where people build software. More than 27 million people use GitHub to discover, fork, and contribute to over 80 million projects.



according to formula for single core machine,

APP_MEM_IN_MB * 100us / 1000 = MILLISECONDS_PER_100_MILLISECOND_PERIOD

In our tests, to get 6.4% CPU usage on a 64MB application and ~100% for a 1GB application we had to set the 'cpu_quota_per_share_in_us' to 3200. (Cell has 4 core and 16GB or ram, overcommit factor of 2).

That changes the formula to:
APP_MEM_IN_MB * 100us / 32000 = MILLISECONDS_PER_100_MILLISECOND_PERIOD

Can you help us to understand from where this 'times 32' comes from? Is the total available RAM of the cell (16GB * overcommit of 2) and number of cpu cores does not matter?

Thanks and regards,
Michael




_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#7904) | [hidden email] | [hidden email] | Mute This Topic | New Topic

Change Your Subscription
Group Home
[hidden email]
Terms Of Service
Unsubscribe From This Group

_._,_._,_
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Understanding hard CPU limits

Eric Malm
Hey, Michael and Marco,

Sorry for the delay in getting a chance to respond to this thread. In general, CF apps receive an allocation of CPU shares proportional to their allocated memory, but with some high and low cutoffs and some limited granularity:

- The minimum number of CPU shares that an app instance (or task) will receive is 10.
- The maximum is 1024.
- The granularity is roughly every 10 shares (10.24, to be precise).


In my personal experiments when setting the garden.cpu_quota_per_share_in_us property, the number of cores does not factor into the per-instance limit, and the quota is enforced as CPU time across all the cores. To constrain a 64 MB-memory app instance to at most 6.4% CPU usage, I had to set garden.cpu_quota_per_share_in_us to 640. A 1 GB-memory app instance, which has 122 CPU shares, can then use up to 78.1% of a CPU core.

Best,
Eric

On Fri, Apr 6, 2018 at 11:02 AM, Dieu Cao <[hidden email]> wrote:
I believe Julz is on vacation this week.
Adding Ed King, the anchor on the Garden team.

Dieu

On Tue, Apr 3, 2018, 3:08 AM Marco Voelz <[hidden email]> wrote:


/cc Eric and Julz: Could you maybe help us understand this? Thanks!


From: [hidden email] <[hidden email]> on behalf of Grifalconi, Michael <[hidden email]>
Sent: Monday, March 26, 2018 10:54:32 AM
To: [hidden email]
Subject: [CAUTION] [cf-dev] Understanding hard CPU limits
 
Hi,

We were trying out the hard CPU limit as per docs
https://github.com/cloudfoundry/garden-runc-release/releases?after=v1.9.2
GitHub is where people build software. More than 27 million people use GitHub to discover, fork, and contribute to over 80 million projects.



according to formula for single core machine,

APP_MEM_IN_MB * 100us / 1000 = MILLISECONDS_PER_100_MILLISECOND_PERIOD

In our tests, to get 6.4% CPU usage on a 64MB application and ~100% for a 1GB application we had to set the 'cpu_quota_per_share_in_us' to 3200. (Cell has 4 core and 16GB or ram, overcommit factor of 2).

That changes the formula to:
APP_MEM_IN_MB * 100us / 32000 = MILLISECONDS_PER_100_MILLISECOND_PERIOD

Can you help us to understand from where this 'times 32' comes from? Is the total available RAM of the cell (16GB * overcommit of 2) and number of cpu cores does not matter?

Thanks and regards,
Michael





_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#7905) | [hidden email] | [hidden email] | Mute This Topic | New Topic

Change Your Subscription
Group Home
[hidden email]
Terms Of Service
Unsubscribe From This Group

_._,_._,_
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Understanding hard CPU limits

Eric Malm
Oh, I omitted one detail from the CC logic: the 1024 CPU-share maximum corresponds to 8192 MB (8 GB) of allocated memory, and this controls the proportionality constant between memory and CPU shares.

Thanks,
Eric

On Tue, Apr 10, 2018 at 10:42 PM, Eric Malm <[hidden email]> wrote:
Hey, Michael and Marco,

Sorry for the delay in getting a chance to respond to this thread. In general, CF apps receive an allocation of CPU shares proportional to their allocated memory, but with some high and low cutoffs and some limited granularity:

- The minimum number of CPU shares that an app instance (or task) will receive is 10.
- The maximum is 1024.
- The granularity is roughly every 10 shares (10.24, to be precise).


In my personal experiments when setting the garden.cpu_quota_per_share_in_us property, the number of cores does not factor into the per-instance limit, and the quota is enforced as CPU time across all the cores. To constrain a 64 MB-memory app instance to at most 6.4% CPU usage, I had to set garden.cpu_quota_per_share_in_us to 640. A 1 GB-memory app instance, which has 122 CPU shares, can then use up to 78.1% of a CPU core.

Best,
Eric

On Fri, Apr 6, 2018 at 11:02 AM, Dieu Cao <[hidden email]> wrote:
I believe Julz is on vacation this week.
Adding Ed King, the anchor on the Garden team.

Dieu

On Tue, Apr 3, 2018, 3:08 AM Marco Voelz <[hidden email]> wrote:


/cc Eric and Julz: Could you maybe help us understand this? Thanks!


From: [hidden email] <[hidden email]> on behalf of Grifalconi, Michael <[hidden email]>
Sent: Monday, March 26, 2018 10:54:32 AM
To: [hidden email]
Subject: [CAUTION] [cf-dev] Understanding hard CPU limits
 
Hi,

We were trying out the hard CPU limit as per docs
https://github.com/cloudfoundry/garden-runc-release/releases?after=v1.9.2
GitHub is where people build software. More than 27 million people use GitHub to discover, fork, and contribute to over 80 million projects.



according to formula for single core machine,

APP_MEM_IN_MB * 100us / 1000 = MILLISECONDS_PER_100_MILLISECOND_PERIOD

In our tests, to get 6.4% CPU usage on a 64MB application and ~100% for a 1GB application we had to set the 'cpu_quota_per_share_in_us' to 3200. (Cell has 4 core and 16GB or ram, overcommit factor of 2).

That changes the formula to:
APP_MEM_IN_MB * 100us / 32000 = MILLISECONDS_PER_100_MILLISECOND_PERIOD

Can you help us to understand from where this 'times 32' comes from? Is the total available RAM of the cell (16GB * overcommit of 2) and number of cpu cores does not matter?

Thanks and regards,
Michael






_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#7906) | [hidden email] | [hidden email] | Mute This Topic | New Topic

Change Your Subscription
Group Home
[hidden email]
Terms Of Service
Unsubscribe From This Group

_._,_._,_
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Understanding hard CPU limits

Grifalconi, Michael

Hello Eric,

 

Many thanks for the detailed explanation!

 

Best regards,

Michael

 

From: <[hidden email]> on behalf of Eric Malm <[hidden email]>
Reply-To: "[hidden email]
g" <[hidden email]>
Date: Wednesday, 11. April 2018 at 17:56
To: cf-dev <[hidden email]>
Subject: Re: [cf-dev] Understanding hard CPU limits

 

 

On Tue, Apr 10, 2018 at 10:42 PM, Eric Malm <[hidden email]> wrote:

Hey, Michael and Marco,

 

Sorry for the delay in getting a chance to respond to this thread. In general, CF apps receive an allocation of CPU shares proportional to their allocated memory, but with some high and low cutoffs and some limited granularity:

 

- The minimum number of CPU shares that an app instance (or task) will receive is 10.

- The maximum is 1024.

- The granularity is roughly every 10 shares (10.24, to be precise).

 

 

In my personal experiments when setting the garden.cpu_quota_per_share_in_us property, the number of cores does not factor into the per-instance limit, and the quota is enforced as CPU time across all the cores. To constrain a 64 MB-memory app instance to at most 6.4% CPU usage, I had to set garden.cpu_quota_per_share_in_us to 640. A 1 GB-memory app instance, which has 122 CPU shares, can then use up to 78.1% of a CPU core.

 

Best,

Eric

 

On Fri, Apr 6, 2018 at 11:02 AM, Dieu Cao <[hidden email]> wrote:

I believe Julz is on vacation this week.

Adding Ed King, the anchor on the Garden team.

 

Dieu

 

On Tue, Apr 3, 2018, 3:08 AM Marco Voelz <[hidden email]> wrote:

 

/cc Eric and Julz: Could you maybe help us understand this? Thanks!


From: [hidden email] <[hidden email]> on behalf of Grifalconi, Michael <[hidden email]>
Sent: Monday, March 26, 2018 10:54:32 AM
To:
[hidden email]
Subject: [CAUTION] [cf-dev] Understanding hard CPU limits

 

Hi,

We were trying out the hard CPU limit as per docs
https://github.com/cloudfoundry/garden-runc-release/releases?after=v1.9.2

GitHub is where people build software. More than 27 million people use GitHub to discover, fork, and contribute to over 80 million projects.




according to formula for single core machine,

APP_MEM_IN_MB * 100us / 1000 = MILLISECONDS_PER_100_MILLISECOND_PERIOD

In our tests, to get 6.4% CPU usage on a 64MB application and ~100% for a 1GB application we had to set the 'cpu_quota_per_share_in_us' to 3200. (Cell has 4 core and 16GB or ram, overcommit factor of 2).

That changes the formula to:
APP_MEM_IN_MB * 100us / 32000 = MILLISECONDS_PER_100_MILLISECOND_PERIOD

Can you help us to understand from where this 'times 32' comes from? Is the total available RAM of the cell (16GB * overcommit of 2) and number of cpu cores does not matter?

Thanks and regards,
Michael



 

 

_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#7909) | [hidden email] | [hidden email] | Mute This Topic | New Topic

Change Your Subscription
Group Home
[hidden email]
Terms Of Service
Unsubscribe From This Group

_._,_._,_
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Understanding hard CPU limits

Marco Voelz
In reply to this post by Eric Malm

Thanks for the thorough explanation, Eric!

 

Warm regards

Marco

 

From: <[hidden email]> on behalf of Eric Malm <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Wednesday, 11. April 2018 at 11:32
To: cf-dev <[hidden email]>
Subject: Re: [cf-dev] Understanding hard CPU limits

 

 

On Tue, Apr 10, 2018 at 10:42 PM, Eric Malm <[hidden email]> wrote:

Hey, Michael and Marco,

 

Sorry for the delay in getting a chance to respond to this thread. In general, CF apps receive an allocation of CPU shares proportional to their allocated memory, but with some high and low cutoffs and some limited granularity:

 

- The minimum number of CPU shares that an app instance (or task) will receive is 10.

- The maximum is 1024.

- The granularity is roughly every 10 shares (10.24, to be precise).

 

 

In my personal experiments when setting the garden.cpu_quota_per_share_in_us property, the number of cores does not factor into the per-instance limit, and the quota is enforced as CPU time across all the cores. To constrain a 64 MB-memory app instance to at most 6.4% CPU usage, I had to set garden.cpu_quota_per_share_in_us to 640. A 1 GB-memory app instance, which has 122 CPU shares, can then use up to 78.1% of a CPU core.

 

Best,

Eric

 

On Fri, Apr 6, 2018 at 11:02 AM, Dieu Cao <[hidden email]> wrote:

I believe Julz is on vacation this week.

Adding Ed King, the anchor on the Garden team.

 

Dieu

 

On Tue, Apr 3, 2018, 3:08 AM Marco Voelz <[hidden email]> wrote:

 

/cc Eric and Julz: Could you maybe help us understand this? Thanks!


From: [hidden email] <[hidden email]> on behalf of Grifalconi, Michael <[hidden email]>
Sent: Monday, March 26, 2018 10:54:32 AM
To:
[hidden email]
Subject: [CAUTION] [cf-dev] Understanding hard CPU limits

 

Hi,

We were trying out the hard CPU limit as per docs
https://github.com/cloudfoundry/garden-runc-release/releases?after=v1.9.2

GitHub is where people build software. More than 27 million people use GitHub to discover, fork, and contribute to over 80 million projects.




according to formula for single core machine,

APP_MEM_IN_MB * 100us / 1000 = MILLISECONDS_PER_100_MILLISECOND_PERIOD

In our tests, to get 6.4% CPU usage on a 64MB application and ~100% for a 1GB application we had to set the 'cpu_quota_per_share_in_us' to 3200. (Cell has 4 core and 16GB or ram, overcommit factor of 2).

That changes the formula to:
APP_MEM_IN_MB * 100us / 32000 = MILLISECONDS_PER_100_MILLISECOND_PERIOD

Can you help us to understand from where this 'times 32' comes from? Is the total available RAM of the cell (16GB * overcommit of 2) and number of cpu cores does not matter?

Thanks and regards,
Michael



 

 

_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#7917) | [hidden email] | [hidden email] | Mute This Topic | New Topic

Change Your Subscription
Group Home
[hidden email]
Terms Of Service
Unsubscribe From This Group

_._,_._,_
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Understanding hard CPU limits

German-Evtushenko, Stanislav | Stas | OPS
In reply to this post by Eric Malm
Hi Eric,

The current logic looks a little bit too complicated. Wouldn't it be simpler if it was not using CPU shares in calculation and just using memory share (container memory / cell memory) for this? In that case we could just specify CPU FACTOR which is easy to understand.

For example:
# Maximum CPU time is proportional to Memory share
# In this example scaling factor is 2
CPU_MAX_FACTOR   =  2

CELL_CPU         =  8
CELL_MEMORY      = 16 * 1024^3  # 16 GiB
CONTAINER_MEMORY =  4 * 1024^3  #  4 GiB
PERIOD           =  100000      # 100 ms (default)

QUOTA_RATIO = ( CONTAINER_MEMORY / CELL_MEMORY ) * CELL_CPU * CPU_MAX_FACTOR = 1/4 * 8 * 2 = 4   # i.e. 4 CPUs time at max
QUOTA = QUOTA_RATIO * PERIOD = 400000   # in ms

PERIOD could also be made configurable so that we could adjust this to our latency requirements if needed.

Best regards,
Stanislav
_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#7958) | [hidden email] | [hidden email] | Mute This Topic | New Topic

Change Your Subscription
Group Home
[hidden email]
Terms Of Service
Unsubscribe From This Group

_._,_._,_