[cf-dev] Discrepancy between `cf apps` memory usage and cgroup's memory.usage_in_bytes in CF 2.13.0

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[cf-dev] Discrepancy between `cf apps` memory usage and cgroup's memory.usage_in_bytes in CF 2.13.0

Daniel Jones
Hi all,

Whilst investigating the Java Buildpack out-of-memory issues David Head-Rapson mailed about the other day, we discovered a discrepancy between the memory usage stat provided by `cf app` and the value stored in the corresponding cgroup's `memory.usage_in_bytes` file. The latter seems to be bumping right along the maximum allowed.

  • We did a `cf app`, and got a memory stat of 847.6MiB of 896MiB.
  • We got the appId from CF_TRACE, `bosh ssh`'d onto the right DEA
  • We then did `cat tmp/warden/cgroup/memory/instance-id/memory.usage_in_bytes` and got 939,515,904, which equates to 895.99ish MiB.
Does anyone know why the latter is so high, and why it would differ from what the DEA reports back to the Cloud Controller? There's clearly a gap in our understanding somewhere, so any help would be much appreciated.

Many thanks,

Daniel Jones
EngineerBetter.com

_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Discrepancy between `cf apps` memory usage and cgroup's memory.usage_in_bytes in CF 2.13.0

Matthew Sykes

On Thu, May 7, 2015 at 8:28 AM, Daniel Jones <[hidden email]> wrote:
Hi all,

Whilst investigating the Java Buildpack out-of-memory issues David Head-Rapson mailed about the other day, we discovered a discrepancy between the memory usage stat provided by `cf app` and the value stored in the corresponding cgroup's `memory.usage_in_bytes` file. The latter seems to be bumping right along the maximum allowed.

  • We did a `cf app`, and got a memory stat of 847.6MiB of 896MiB.
  • We got the appId from CF_TRACE, `bosh ssh`'d onto the right DEA
  • We then did `cat tmp/warden/cgroup/memory/instance-id/memory.usage_in_bytes` and got 939,515,904, which equates to 895.99ish MiB.
Does anyone know why the latter is so high, and why it would differ from what the DEA reports back to the Cloud Controller? There's clearly a gap in our understanding somewhere, so any help would be much appreciated.

Many thanks,

Daniel Jones
EngineerBetter.com

_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




--
Matthew Sykes
[hidden email]

_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Discrepancy between `cf apps` memory usage and cgroup's memory.usage_in_bytes in CF 2.13.0

Daniel Jones
Thanks for the reply Matthew!

We saw that bit of code when we were following it through. When we go as far as the call from the DEA's Warden client to the Warden Server, we struggled to find where those stats (total_rss, total_cache, total_inactive_file) came from. DO you happen to know what the source of truth is for those data?

Thanks for your help.

On Thu, May 7, 2015 at 1:58 PM, Matthew Sykes <[hidden email]> wrote:

On Thu, May 7, 2015 at 8:28 AM, Daniel Jones <[hidden email]> wrote:
Hi all,

Whilst investigating the Java Buildpack out-of-memory issues David Head-Rapson mailed about the other day, we discovered a discrepancy between the memory usage stat provided by `cf app` and the value stored in the corresponding cgroup's `memory.usage_in_bytes` file. The latter seems to be bumping right along the maximum allowed.

  • We did a `cf app`, and got a memory stat of 847.6MiB of 896MiB.
  • We got the appId from CF_TRACE, `bosh ssh`'d onto the right DEA
  • We then did `cat tmp/warden/cgroup/memory/instance-id/memory.usage_in_bytes` and got 939,515,904, which equates to 895.99ish MiB.
Does anyone know why the latter is so high, and why it would differ from what the DEA reports back to the Cloud Controller? There's clearly a gap in our understanding somewhere, so any help would be much appreciated.

Many thanks,

Daniel Jones
EngineerBetter.com

_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




--
Matthew Sykes
[hidden email]

_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




--
Regards,

Daniel Jones
EngineerBetter.com

_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Discrepancy between `cf apps` memory usage and cgroup's memory.usage_in_bytes in CF 2.13.0

Daniel Jones
Aha - just found it, and it does indeed tally up. Thanks again!

On Thu, May 7, 2015 at 3:06 PM, Daniel Jones <[hidden email]> wrote:
Thanks for the reply Matthew!

We saw that bit of code when we were following it through. When we go as far as the call from the DEA's Warden client to the Warden Server, we struggled to find where those stats (total_rss, total_cache, total_inactive_file) came from. DO you happen to know what the source of truth is for those data?

Thanks for your help.

On Thu, May 7, 2015 at 1:58 PM, Matthew Sykes <[hidden email]> wrote:

On Thu, May 7, 2015 at 8:28 AM, Daniel Jones <[hidden email]> wrote:
Hi all,

Whilst investigating the Java Buildpack out-of-memory issues David Head-Rapson mailed about the other day, we discovered a discrepancy between the memory usage stat provided by `cf app` and the value stored in the corresponding cgroup's `memory.usage_in_bytes` file. The latter seems to be bumping right along the maximum allowed.

  • We did a `cf app`, and got a memory stat of 847.6MiB of 896MiB.
  • We got the appId from CF_TRACE, `bosh ssh`'d onto the right DEA
  • We then did `cat tmp/warden/cgroup/memory/instance-id/memory.usage_in_bytes` and got 939,515,904, which equates to 895.99ish MiB.
Does anyone know why the latter is so high, and why it would differ from what the DEA reports back to the Cloud Controller? There's clearly a gap in our understanding somewhere, so any help would be much appreciated.

Many thanks,

Daniel Jones
EngineerBetter.com

_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




--
Matthew Sykes
[hidden email]

_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




--
Regards,

Daniel Jones
EngineerBetter.com



--
Regards,

Daniel Jones
EngineerBetter.com

_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev