[cf-dev] Buildpack Cache

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[cf-dev] Buildpack Cache

Stephan Merker

Hi,

 

is there documentation available that describes how the buildpack cache works?

 

I found the following info:

https://docs.cloudfoundry.org/concepts/how-applications-are-staged.html

“The Task downloads buildpacks and the app’s buildpack cache, if present.”

 

https://github.com/cloudfoundry/java-buildpack/blob/master/docs/extending-caches.md

 

I wonder how often the buildpacks and artifacts like JVM, Tomcat etc. are downloaded and how they are cached? Is this cache per app, per diego cell, CF global?

I’m talking about the default online buildpacks.

 

Thanks,

Stephan

 

_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#8061) | [hidden email] | [hidden email] | Mute This Topic | New Topic

Your Subscription | [hidden email] | Unsubscribe [[hidden email]]

_._,_._,_
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Buildpack Cache

Stephen Levine
Hi Stephan,

The buildpack cache (aka "app cache," or "build artifacts cache") is a per-app cache that is stored as a tarball in the blob store and recovered when an existing app is restaged (regardless of app code or buildpack changes).

Different buildpacks use this cache for different things. Ben Hale (CCd) may have more details about how the Java buildpack handles caching. In general, the purpose of the cache is to speed up build time and/or reduce data transfer during staging.


-Stephen


On Thu, Jun 14, 2018 at 10:31 AM Stephan Merker <[hidden email]> wrote:

Hi,

 

is there documentation available that describes how the buildpack cache works?

 

I found the following info:

https://docs.cloudfoundry.org/concepts/how-applications-are-staged.html

“The Task downloads buildpacks and the app’s buildpack cache, if present.”

 

https://github.com/cloudfoundry/java-buildpack/blob/master/docs/extending-caches.md

 

I wonder how often the buildpacks and artifacts like JVM, Tomcat etc. are downloaded and how they are cached? Is this cache per app, per diego cell, CF global?

I’m talking about the default online buildpacks.

 

Thanks,

Stephan

 

_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#8065) | [hidden email] | [hidden email] | Mute This Topic | New Topic

Your Subscription | [hidden email] | Unsubscribe [[hidden email]]

_._,_._,_
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Buildpack Cache

Stephan Merker

Thanks for the explanation.

 

I have one more question: Is there any caching for the buildpack itself?

The CC coding looks like the buildpack is downloaded for each staging operation (either as zip file or via git clone).

Is there a difference between system buildpacks and custom buildpacks when it comes to download and caching?

 

Best regards,

Stephan

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Stephen Levine
Sent: Donnerstag, 14. Juni 2018 22:04
To: Discussions about Cloud Foundry projects and the system overall. <[hidden email]>; Ben Hale <[hidden email]>
Subject: Re: [cf-dev] Buildpack Cache

 

Hi Stephan,

 

The buildpack cache (aka "app cache," or "build artifacts cache") is a per-app cache that is stored as a tarball in the blob store and recovered when an existing app is restaged (regardless of app code or buildpack changes).

 

Different buildpacks use this cache for different things. Ben Hale (CCd) may have more details about how the Java buildpack handles caching. In general, the purpose of the cache is to speed up build time and/or reduce data transfer during staging.

 

 

-Stephen

 

 

On Thu, Jun 14, 2018 at 10:31 AM Stephan Merker <[hidden email]> wrote:

Hi,

 

is there documentation available that describes how the buildpack cache works?

 

I found the following info:

https://docs.cloudfoundry.org/concepts/how-applications-are-staged.html

“The Task downloads buildpacks and the app’s buildpack cache, if present.”

 

https://github.com/cloudfoundry/java-buildpack/blob/master/docs/extending-caches.md

 

I wonder how often the buildpacks and artifacts like JVM, Tomcat etc. are downloaded and how they are cached? Is this cache per app, per diego cell, CF global?

I’m talking about the default online buildpacks.

 

Thanks,

Stephan

 

_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#8072) | [hidden email] | [hidden email] | Mute This Topic | New Topic

Your Subscription | [hidden email] | Unsubscribe [[hidden email]]

_._,_._,_
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Buildpack Cache

Stephen Levine
The system buildpacks are cached on the cell and bind mounted read-only into each staging container. Buildpack URLs (to a git repo or a zip file) are downloaded every time staging occurs.

On Mon, Jun 18, 2018 at 12:50 PM Stephan Merker <[hidden email]> wrote:

Thanks for the explanation.

 

I have one more question: Is there any caching for the buildpack itself?

The CC coding looks like the buildpack is downloaded for each staging operation (either as zip file or via git clone).

Is there a difference between system buildpacks and custom buildpacks when it comes to download and caching?

 

Best regards,

Stephan

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Stephen Levine
Sent: Donnerstag, 14. Juni 2018 22:04
To: Discussions about Cloud Foundry projects and the system overall. <[hidden email]>; Ben Hale <[hidden email]>
Subject: Re: [cf-dev] Buildpack Cache

 

Hi Stephan,

 

The buildpack cache (aka "app cache," or "build artifacts cache") is a per-app cache that is stored as a tarball in the blob store and recovered when an existing app is restaged (regardless of app code or buildpack changes).

 

Different buildpacks use this cache for different things. Ben Hale (CCd) may have more details about how the Java buildpack handles caching. In general, the purpose of the cache is to speed up build time and/or reduce data transfer during staging.

 

 

-Stephen

 

 

On Thu, Jun 14, 2018 at 10:31 AM Stephan Merker <[hidden email]> wrote:

Hi,

 

is there documentation available that describes how the buildpack cache works?

 

I found the following info:

https://docs.cloudfoundry.org/concepts/how-applications-are-staged.html

“The Task downloads buildpacks and the app’s buildpack cache, if present.”

 

https://github.com/cloudfoundry/java-buildpack/blob/master/docs/extending-caches.md

 

I wonder how often the buildpacks and artifacts like JVM, Tomcat etc. are downloaded and how they are cached? Is this cache per app, per diego cell, CF global?

I’m talking about the default online buildpacks.

 

Thanks,

Stephan

 

_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#8073) | [hidden email] | [hidden email] | Mute This Topic | New Topic

Your Subscription | [hidden email] | Unsubscribe [[hidden email]]

_._,_._,_
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Buildpack Cache

Ben Hale
In reply to this post by Stephen Levine
> Different buildpacks use this cache for different things. Ben Hale (CCd) may have more details about how the Java buildpack handles caching. In general, the purpose of the cache is to speed up build time and/or reduce data transfer during staging.

The Java Buildpack treats the Buildpack cache as an opaque directory.  This means that there are larger system policies about whether it's cached per-cell, per-foundation, etc. but from the perspective of the Java Buildpack, there is only a directory that may or may not contain cached artifacts.

All artifacts that the Java Buildpack might download are cached upon retrieval.  So if you are using an online buildpack and download a JVM, we'll only download once per buildpack-cache (again, the system-wide policies are what really govern how often this happens) and store it for future staging.  In addition to the artifact itself, we also store the `Last-Modified` timestamp and the `ETag` if either of these is returned when the artifact is downloaded.  For all subsequent uses of an artifact, the buildpack first attempts a download with `If-Modified-Since` and `If-None-Match`.  If all goes well (i.e. the artifact hasn't changed on the server), a `304 Not Modified` response will be returned and we know that our cached copies are safe for use and they'll be served up to the application without and bytes downloaded.

Cache hits are called out as part of staging output:

```
-----> Java Buildpack v4.12 (offline) | https://github.com/cloudfoundry/java-buildpack.git#5dca820
-----> Downloading Jvmkill Agent 1.12.0_RELEASE from https://java-buildpack.cloudfoundry.org/jvmkill/trusty/x86_64/jvmkill-1.12.0_RELEASE.so (found in cache)
-----> Downloading Open Jdk JRE 1.8.0_172 from https://java-buildpack.cloudfoundry.org/openjdk/trusty/x86_64/openjdk-1.8.0_172.tar.gz (found in cache)
       Expanding Open Jdk JRE to .java-buildpack/open_jdk_jre (1.1s)
       JVM DNS caching disabled in lieu of BOSH DNS caching
-----> Downloading Open JDK Like Memory Calculator 3.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/memory-calculator/trusty/x86_64/memory-calculator-3.13.0_RELEASE.tar.gz (found in cache)
       Loaded Classes: 16818, Threads: 250
-----> Downloading Client Certificate Mapper 1.6.0_RELEASE from https://java-buildpack.cloudfoundry.org/client-certificate-mapper/client-certificate-mapper-1.6.0_RELEASE.jar (found in cache)
-----> Downloading Container Security Provider 1.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/container-security-provider/container-security-provider-1.13.0_RELEASE.jar (found in cache)
-----> Downloading Spring Auto Reconfiguration 2.4.0_RELEASE from https://java-buildpack.cloudfoundry.org/auto-reconfiguration/auto-reconfiguration-2.4.0_RELEASE.jar (found in cache)
```

versus:

```
-----> Java Buildpack 222ce62 | https://github.com/cloudfoundry/java-buildpack.git#222ce62
-----> Downloading Jvmkill Agent 1.14.0_RELEASE from https://java-buildpack.cloudfoundry.org/jvmkill/trusty/x86_64/jvmkill-1.14.0_RELEASE.so (0.0s)
-----> Downloading Open Jdk JRE 1.8.0_172 from https://java-buildpack.cloudfoundry.org/openjdk/trusty/x86_64/openjdk-1.8.0_172.tar.gz (0.9s)
       Expanding Open Jdk JRE to .java-buildpack/open_jdk_jre (1.4s)
       JVM DNS caching disabled in lieu of BOSH DNS caching
-----> Downloading Open JDK Like Memory Calculator 3.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/memory-calculator/trusty/x86_64/memory-calculator-3.13.0_RELEASE.tar.gz (0.0s)
       Loaded Classes: 16818, Threads: 250
-----> Downloading Client Certificate Mapper 1.6.0_RELEASE from https://java-buildpack.cloudfoundry.org/client-certificate-mapper/client-certificate-mapper-1.6.0_RELEASE.jar (0.0s)
-----> Downloading Container Security Provider 1.14.0_RELEASE from https://java-buildpack.cloudfoundry.org/container-security-provider/container-security-provider-1.14.0_RELEASE.jar (0.0s)
-----> Downloading Spring Auto Reconfiguration 2.4.0_RELEASE from https://java-buildpack.cloudfoundry.org/auto-reconfiguration/auto-reconfiguration-2.4.0_RELEASE.jar (0.0s)
```


-Ben Hale
Cloud Foundry Java Experience


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#8074): https://lists.cloudfoundry.org/g/cf-dev/message/8074
Mute This Topic: https://lists.cloudfoundry.org/mt/22187254/474226
Group Owner: [hidden email]
Unsubscribe: https://lists.cloudfoundry.org/g/cf-dev/leave/920759/1741049355/xyzzy  [[hidden email]]
-=-=-=-=-=-=-=-=-=-=-=-

Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Buildpack Cache

Stephan Merker
> This means that there are larger system policies about whether it's cached per-cell, per-foundation, etc.

Do you mean with "larger system policies" the usage of online vs. offline buildpacks? Or are there configuration parameters in e.g. the CC  or Diego to configure the caching policy?
In other words: what is the global buildpack artifact caching policy for a standard https://github.com/cloudfoundry/cf-deployment/releases/tag/v1.40.0 

Best regards,
Stephan

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Ben Hale
Sent: Montag, 18. Juni 2018 20:23
To: Discussions about Cloud Foundry projects and the system overall. <[hidden email]>
Subject: Re: [cf-dev] Buildpack Cache

> Different buildpacks use this cache for different things. Ben Hale (CCd) may have more details about how the Java buildpack handles caching. In general, the purpose of the cache is to speed up build time and/or reduce data transfer during staging.

The Java Buildpack treats the Buildpack cache as an opaque directory.  This means that there are larger system policies about whether it's cached per-cell, per-foundation, etc. but from the perspective of the Java Buildpack, there is only a directory that may or may not contain cached artifacts.

All artifacts that the Java Buildpack might download are cached upon retrieval.  So if you are using an online buildpack and download a JVM, we'll only download once per buildpack-cache (again, the system-wide policies are what really govern how often this happens) and store it for future staging.  In addition to the artifact itself, we also store the `Last-Modified` timestamp and the `ETag` if either of these is returned when the artifact is downloaded.  For all subsequent uses of an artifact, the buildpack first attempts a download with `If-Modified-Since` and `If-None-Match`.  If all goes well (i.e. the artifact hasn't changed on the server), a `304 Not Modified` response will be returned and we know that our cached copies are safe for use and they'll be served up to the application without and bytes downloaded.

Cache hits are called out as part of staging output:

```
-----> Java Buildpack v4.12 (offline) | https://github.com/cloudfoundry/java-buildpack.git#5dca820
-----> Downloading Jvmkill Agent 1.12.0_RELEASE from https://java-buildpack.cloudfoundry.org/jvmkill/trusty/x86_64/jvmkill-1.12.0_RELEASE.so (found in cache)
-----> Downloading Open Jdk JRE 1.8.0_172 from https://java-buildpack.cloudfoundry.org/openjdk/trusty/x86_64/openjdk-1.8.0_172.tar.gz (found in cache)
       Expanding Open Jdk JRE to .java-buildpack/open_jdk_jre (1.1s)
       JVM DNS caching disabled in lieu of BOSH DNS caching
-----> Downloading Open JDK Like Memory Calculator 3.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/memory-calculator/trusty/x86_64/memory-calculator-3.13.0_RELEASE.tar.gz (found in cache)
       Loaded Classes: 16818, Threads: 250
-----> Downloading Client Certificate Mapper 1.6.0_RELEASE from https://java-buildpack.cloudfoundry.org/client-certificate-mapper/client-certificate-mapper-1.6.0_RELEASE.jar (found in cache)
-----> Downloading Container Security Provider 1.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/container-security-provider/container-security-provider-1.13.0_RELEASE.jar (found in cache)
-----> Downloading Spring Auto Reconfiguration 2.4.0_RELEASE from https://java-buildpack.cloudfoundry.org/auto-reconfiguration/auto-reconfiguration-2.4.0_RELEASE.jar (found in cache)
```

versus:

```
-----> Java Buildpack 222ce62 | https://github.com/cloudfoundry/java-buildpack.git#222ce62
-----> Downloading Jvmkill Agent 1.14.0_RELEASE from https://java-buildpack.cloudfoundry.org/jvmkill/trusty/x86_64/jvmkill-1.14.0_RELEASE.so (0.0s)
-----> Downloading Open Jdk JRE 1.8.0_172 from https://java-buildpack.cloudfoundry.org/openjdk/trusty/x86_64/openjdk-1.8.0_172.tar.gz (0.9s)
       Expanding Open Jdk JRE to .java-buildpack/open_jdk_jre (1.4s)
       JVM DNS caching disabled in lieu of BOSH DNS caching
-----> Downloading Open JDK Like Memory Calculator 3.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/memory-calculator/trusty/x86_64/memory-calculator-3.13.0_RELEASE.tar.gz (0.0s)
       Loaded Classes: 16818, Threads: 250
-----> Downloading Client Certificate Mapper 1.6.0_RELEASE from https://java-buildpack.cloudfoundry.org/client-certificate-mapper/client-certificate-mapper-1.6.0_RELEASE.jar (0.0s)
-----> Downloading Container Security Provider 1.14.0_RELEASE from https://java-buildpack.cloudfoundry.org/container-security-provider/container-security-provider-1.14.0_RELEASE.jar (0.0s)
-----> Downloading Spring Auto Reconfiguration 2.4.0_RELEASE from https://java-buildpack.cloudfoundry.org/auto-reconfiguration/auto-reconfiguration-2.4.0_RELEASE.jar (0.0s)
```


-Ben Hale
Cloud Foundry Java Experience





-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#8081): https://lists.cloudfoundry.org/g/cf-dev/message/8081
Mute This Topic: https://lists.cloudfoundry.org/mt/22187254/474226
Group Owner: [hidden email]
Unsubscribe: https://lists.cloudfoundry.org/g/cf-dev/leave/920759/1741049355/xyzzy  [[hidden email]]
-=-=-=-=-=-=-=-=-=-=-=-

Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Buildpack Cache

Stephen Levine
The buildpack cache in CF is always per-app and foundation-wide. It can only be erased by deleting the app or dropping all cache artifacts on the platform.

The system buildpacks are cached on the cell, and if you use an offline buildpack with dependencies included, the buildpack will generally not need to download those dependencies. Therefore, they won't get cached in the app's buildpack cache (but other things might still be).

On Tue, Jun 19, 2018 at 11:04 AM Stephan Merker <[hidden email]> wrote:
> This means that there are larger system policies about whether it's cached per-cell, per-foundation, etc.

Do you mean with "larger system policies" the usage of online vs. offline buildpacks? Or are there configuration parameters in e.g. the CC  or Diego to configure the caching policy?
In other words: what is the global buildpack artifact caching policy for a standard https://github.com/cloudfoundry/cf-deployment/releases/tag/v1.40.0

Best regards,
Stephan

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Ben Hale
Sent: Montag, 18. Juni 2018 20:23
To: Discussions about Cloud Foundry projects and the system overall. <[hidden email]>
Subject: Re: [cf-dev] Buildpack Cache

> Different buildpacks use this cache for different things. Ben Hale (CCd) may have more details about how the Java buildpack handles caching. In general, the purpose of the cache is to speed up build time and/or reduce data transfer during staging.

The Java Buildpack treats the Buildpack cache as an opaque directory.  This means that there are larger system policies about whether it's cached per-cell, per-foundation, etc. but from the perspective of the Java Buildpack, there is only a directory that may or may not contain cached artifacts.

All artifacts that the Java Buildpack might download are cached upon retrieval.  So if you are using an online buildpack and download a JVM, we'll only download once per buildpack-cache (again, the system-wide policies are what really govern how often this happens) and store it for future staging.  In addition to the artifact itself, we also store the `Last-Modified` timestamp and the `ETag` if either of these is returned when the artifact is downloaded.  For all subsequent uses of an artifact, the buildpack first attempts a download with `If-Modified-Since` and `If-None-Match`.  If all goes well (i.e. the artifact hasn't changed on the server), a `304 Not Modified` response will be returned and we know that our cached copies are safe for use and they'll be served up to the application without and bytes downloaded.

Cache hits are called out as part of staging output:

```
-----> Java Buildpack v4.12 (offline) | https://github.com/cloudfoundry/java-buildpack.git#5dca820
-----> Downloading Jvmkill Agent 1.12.0_RELEASE from https://java-buildpack.cloudfoundry.org/jvmkill/trusty/x86_64/jvmkill-1.12.0_RELEASE.so (found in cache)
-----> Downloading Open Jdk JRE 1.8.0_172 from https://java-buildpack.cloudfoundry.org/openjdk/trusty/x86_64/openjdk-1.8.0_172.tar.gz (found in cache)
       Expanding Open Jdk JRE to .java-buildpack/open_jdk_jre (1.1s)
       JVM DNS caching disabled in lieu of BOSH DNS caching
-----> Downloading Open JDK Like Memory Calculator 3.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/memory-calculator/trusty/x86_64/memory-calculator-3.13.0_RELEASE.tar.gz (found in cache)
       Loaded Classes: 16818, Threads: 250
-----> Downloading Client Certificate Mapper 1.6.0_RELEASE from https://java-buildpack.cloudfoundry.org/client-certificate-mapper/client-certificate-mapper-1.6.0_RELEASE.jar (found in cache)
-----> Downloading Container Security Provider 1.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/container-security-provider/container-security-provider-1.13.0_RELEASE.jar (found in cache)
-----> Downloading Spring Auto Reconfiguration 2.4.0_RELEASE from https://java-buildpack.cloudfoundry.org/auto-reconfiguration/auto-reconfiguration-2.4.0_RELEASE.jar (found in cache)
```

versus:

```
-----> Java Buildpack 222ce62 | https://github.com/cloudfoundry/java-buildpack.git#222ce62
-----> Downloading Jvmkill Agent 1.14.0_RELEASE from https://java-buildpack.cloudfoundry.org/jvmkill/trusty/x86_64/jvmkill-1.14.0_RELEASE.so (0.0s)
-----> Downloading Open Jdk JRE 1.8.0_172 from https://java-buildpack.cloudfoundry.org/openjdk/trusty/x86_64/openjdk-1.8.0_172.tar.gz (0.9s)
       Expanding Open Jdk JRE to .java-buildpack/open_jdk_jre (1.4s)
       JVM DNS caching disabled in lieu of BOSH DNS caching
-----> Downloading Open JDK Like Memory Calculator 3.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/memory-calculator/trusty/x86_64/memory-calculator-3.13.0_RELEASE.tar.gz (0.0s)
       Loaded Classes: 16818, Threads: 250
-----> Downloading Client Certificate Mapper 1.6.0_RELEASE from https://java-buildpack.cloudfoundry.org/client-certificate-mapper/client-certificate-mapper-1.6.0_RELEASE.jar (0.0s)
-----> Downloading Container Security Provider 1.14.0_RELEASE from https://java-buildpack.cloudfoundry.org/container-security-provider/container-security-provider-1.14.0_RELEASE.jar (0.0s)
-----> Downloading Spring Auto Reconfiguration 2.4.0_RELEASE from https://java-buildpack.cloudfoundry.org/auto-reconfiguration/auto-reconfiguration-2.4.0_RELEASE.jar (0.0s)
```


-Ben Hale
Cloud Foundry Java Experience







_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#8083) | [hidden email] | [hidden email] | Mute This Topic | New Topic

Your Subscription | [hidden email] | Unsubscribe [[hidden email]]

_._,_._,_