[cf-dev] Buildpack deep dive questions...

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[cf-dev] Buildpack deep dive questions...

Cade Thacker
Long time reader... first time poster :D 

I'm really digging and deconstruction a few different buildpacks trying to understand how they really work.  supply, compile, detect, etc.  Especially the multi buildpack POV. 

So I think i'm starting to get the details straight but I have some basic architecture questions. 

1) For example, if the core functionality of the buildpack is a compiled app (like go or java) and not a scripted language but the github repo only contains the source code then somewhere in this process it must be compiled.  If I want to use the buildpack as an online buildpack that means that the buildpack has to be "built" after it is cloned down during staging.  Is this the correct path?  Or should it be built and the binary pushed back with the code into github? This seems wrong. Somebody hit me with the clue stick. 

2) Second, somewhat tangent question. I know that CloudFoundry is doing tons of work around envoy/istio, but we have a more pressing need right now.  My company has a need to connect many apps and CloudFoundry is just one small part of our runtime. Would it be possible to write a final build pack for envoy and have it basically bind to PORT and then pass the calls to whatever was the 2nd to last buildpack? java, ruby, go, etc.  Forgive me if this is a silly question, but trying to find the edge of this new multi build pack world. Need to inject some custom ingress/egress rules. 

-- 
--cade
_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#8547) | [hidden email] | [hidden email] | Mute This Topic | New Topic

Your Subscription | [hidden email] | Unsubscribe [[hidden email]]

_._,_._,_
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Buildpack deep dive questions...

Daniel Mikusa
On Mon, Mar 25, 2019 at 10:45 AM Cade Thacker <[hidden email]> wrote:
Long time reader... first time poster :D 

I'm really digging and deconstruction a few different buildpacks trying to understand how they really work.  supply, compile, detect, etc.  Especially the multi buildpack POV. 

So I think i'm starting to get the details straight but I have some basic architecture questions. 

1) For example, if the core functionality of the buildpack is a compiled app (like go or java) and not a scripted language but the github repo only contains the source code then somewhere in this process it must be compiled.  If I want to use the buildpack as an online buildpack that means that the buildpack has to be "built" after it is cloned down during staging.  Is this the correct path?  Or should it be built and the binary pushed back with the code into github? This seems wrong. Somebody hit me with the clue stick. 

If your buildpack itself needs to be compiled, you have two choices:

1.) Compile locally, package a buildpack and install with `cf create-buildpack`.
2.) Use a shim script to build then run your buildpack. If you want to support `cf push -b <a href="https://git-url/my-buildpack`">https://git-url/my-buildpack` then you have to do this.


and the script it runs to install Golang.

 

2) Second, somewhat tangent question. I know that CloudFoundry is doing tons of work around envoy/istio, but we have a more pressing need right now.  My company has a need to connect many apps and CloudFoundry is just one small part of our runtime. Would it be possible to write a final build pack for envoy and have it basically bind to PORT and then pass the calls to whatever was the 2nd to last buildpack? java, ruby, go, etc.  Forgive me if this is a silly question, but trying to find the edge of this new multi build pack world. Need to inject some custom ingress/egress rules. 

The final buildpack can basically do what ever it wants, since it gets to dictate the start command. If you want it to start Envoy + some other stuff you can do that. A couple things to keep in mind. First, if you're starting multiple processes in the app container you need to manage them. If one fails, you need to make it so that they all fail, or you'll have unexpected behavior. For example, if Envoy is running but your second process, presumably your actual app, crashes then you need to make sure they all crash so that the platform can detect the crash and restart your app. Second, multiple processes in the container makes it harder to reason about available memory, be especially careful if you're running a Java app. Lastly, only the supply script will run for non-final buildpacks. If you need finalize to run for a non-final buildpack, like to get the proper Java start command, then that's not going to be easy.

Hope that helps!

Dan


_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#8556) | [hidden email] | [hidden email] | Mute This Topic | New Topic

Your Subscription | [hidden email] | Unsubscribe [[hidden email]]

_._,_._,_
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Buildpack deep dive questions...

Tim Downey
Hi Cade,

This won't necessarily help for your immediate needs, but wanted to share that the Cloud Controller (CAPI) team is actively working on experimental support for including user-specified sidecar processes within the app container.

For the MVP form of this you'll still need to find a way to include the sidecar executable with your app (either via a buildpack or including it with the app source), but it should provide an easier way to specify/view information about the sidecar processes via `/v3/sidecars`.


As I said, this is still under active development and very experimental, but we'd be interested in learning more about your use cases.

Stay tuned!

Tim Downey, CAPI Team

On Tue, Apr 2, 2019, 6:53 AM Daniel Mikusa <[hidden email]> wrote:
On Mon, Mar 25, 2019 at 10:45 AM Cade Thacker <[hidden email]> wrote:
Long time reader... first time poster :D 

I'm really digging and deconstruction a few different buildpacks trying to understand how they really work.  supply, compile, detect, etc.  Especially the multi buildpack POV. 

So I think i'm starting to get the details straight but I have some basic architecture questions. 

1) For example, if the core functionality of the buildpack is a compiled app (like go or java) and not a scripted language but the github repo only contains the source code then somewhere in this process it must be compiled.  If I want to use the buildpack as an online buildpack that means that the buildpack has to be "built" after it is cloned down during staging.  Is this the correct path?  Or should it be built and the binary pushed back with the code into github? This seems wrong. Somebody hit me with the clue stick. 

If your buildpack itself needs to be compiled, you have two choices:

1.) Compile locally, package a buildpack and install with `cf create-buildpack`.
2.) Use a shim script to build then run your buildpack. If you want to support `cf push -b https://git-url/my-buildpack` then you have to do this.


and the script it runs to install Golang.

 

2) Second, somewhat tangent question. I know that CloudFoundry is doing tons of work around envoy/istio, but we have a more pressing need right now.  My company has a need to connect many apps and CloudFoundry is just one small part of our runtime. Would it be possible to write a final build pack for envoy and have it basically bind to PORT and then pass the calls to whatever was the 2nd to last buildpack? java, ruby, go, etc.  Forgive me if this is a silly question, but trying to find the edge of this new multi build pack world. Need to inject some custom ingress/egress rules. 

The final buildpack can basically do what ever it wants, since it gets to dictate the start command. If you want it to start Envoy + some other stuff you can do that. A couple things to keep in mind. First, if you're starting multiple processes in the app container you need to manage them. If one fails, you need to make it so that they all fail, or you'll have unexpected behavior. For example, if Envoy is running but your second process, presumably your actual app, crashes then you need to make sure they all crash so that the platform can detect the crash and restart your app. Second, multiple processes in the container makes it harder to reason about available memory, be especially careful if you're running a Java app. Lastly, only the supply script will run for non-final buildpacks. If you need finalize to run for a non-final buildpack, like to get the proper Java start command, then that's not going to be easy.

Hope that helps!

Dan


_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#8557) | [hidden email] | [hidden email] | Mute This Topic | New Topic

Your Subscription | [hidden email] | Unsubscribe [[hidden email]]

_._,_._,_
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Buildpack deep dive questions...

Cade Thacker-2
This is outstanding.  How can I get involved? I’d love to jump into that conversation with both feet and help!! 

We are looking to better control egress and offload app security on ingress.  

Specifically envoy.  

Cade


On Apr 2, 2019, at 12:38, Tim Downey <[hidden email]> wrote:

Hi Cade,

This won't necessarily help for your immediate needs, but wanted to share that the Cloud Controller (CAPI) team is actively working on experimental support for including user-specified sidecar processes within the app container.

For the MVP form of this you'll still need to find a way to include the sidecar executable with your app (either via a buildpack or including it with the app source), but it should provide an easier way to specify/view information about the sidecar processes via `/v3/sidecars`.


As I said, this is still under active development and very experimental, but we'd be interested in learning more about your use cases.

Stay tuned!

Tim Downey, CAPI Team

On Tue, Apr 2, 2019, 6:53 AM Daniel Mikusa <[hidden email]> wrote:
On Mon, Mar 25, 2019 at 10:45 AM Cade Thacker <[hidden email]> wrote:
Long time reader... first time poster :D 

I'm really digging and deconstruction a few different buildpacks trying to understand how they really work.  supply, compile, detect, etc.  Especially the multi buildpack POV. 

So I think i'm starting to get the details straight but I have some basic architecture questions. 

1) For example, if the core functionality of the buildpack is a compiled app (like go or java) and not a scripted language but the github repo only contains the source code then somewhere in this process it must be compiled.  If I want to use the buildpack as an online buildpack that means that the buildpack has to be "built" after it is cloned down during staging.  Is this the correct path?  Or should it be built and the binary pushed back with the code into github? This seems wrong. Somebody hit me with the clue stick. 

If your buildpack itself needs to be compiled, you have two choices:

1.) Compile locally, package a buildpack and install with `cf create-buildpack`.
2.) Use a shim script to build then run your buildpack. If you want to support `cf push -b https://git-url/my-buildpack` then you have to do this.


and the script it runs to install Golang.

 

2) Second, somewhat tangent question. I know that CloudFoundry is doing tons of work around envoy/istio, but we have a more pressing need right now.  My company has a need to connect many apps and CloudFoundry is just one small part of our runtime. Would it be possible to write a final build pack for envoy and have it basically bind to PORT and then pass the calls to whatever was the 2nd to last buildpack? java, ruby, go, etc.  Forgive me if this is a silly question, but trying to find the edge of this new multi build pack world. Need to inject some custom ingress/egress rules. 

The final buildpack can basically do what ever it wants, since it gets to dictate the start command. If you want it to start Envoy + some other stuff you can do that. A couple things to keep in mind. First, if you're starting multiple processes in the app container you need to manage them. If one fails, you need to make it so that they all fail, or you'll have unexpected behavior. For example, if Envoy is running but your second process, presumably your actual app, crashes then you need to make sure they all crash so that the platform can detect the crash and restart your app. Second, multiple processes in the container makes it harder to reason about available memory, be especially careful if you're running a Java app. Lastly, only the supply script will run for non-final buildpacks. If you need finalize to run for a non-final buildpack, like to get the proper Java start command, then that's not going to be easy.

Hope that helps!

Dan


_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#8558) | [hidden email] | [hidden email] | Mute This Topic | New Topic

Your Subscription | [hidden email] | Unsubscribe [[hidden email]]

_._,_._,_
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Buildpack deep dive questions...

Tim Downey
Feel free to reach out in the #capi channel on the Cloud Foundry Slack and folks from our team would be happy to chat!

As I mentioned earlier, we're actively working on the MVP so it's very experimental and there's not much yet in the way of docs for it, but we're working on it! Don't mean to derail your immediate plans though, so I would recommend looking into Dan's suggestions for the near-term.

- Tim

On Tue, Apr 2, 2019 at 1:22 PM Cade Thacker <[hidden email]> wrote:
This is outstanding.  How can I get involved? I’d love to jump into that conversation with both feet and help!! 

We are looking to better control egress and offload app security on ingress.  

Specifically envoy.  

Cade


On Apr 2, 2019, at 12:38, Tim Downey <[hidden email]> wrote:

Hi Cade,

This won't necessarily help for your immediate needs, but wanted to share that the Cloud Controller (CAPI) team is actively working on experimental support for including user-specified sidecar processes within the app container.

For the MVP form of this you'll still need to find a way to include the sidecar executable with your app (either via a buildpack or including it with the app source), but it should provide an easier way to specify/view information about the sidecar processes via `/v3/sidecars`.


As I said, this is still under active development and very experimental, but we'd be interested in learning more about your use cases.

Stay tuned!

Tim Downey, CAPI Team

On Tue, Apr 2, 2019, 6:53 AM Daniel Mikusa <[hidden email]> wrote:
On Mon, Mar 25, 2019 at 10:45 AM Cade Thacker <[hidden email]> wrote:
Long time reader... first time poster :D 

I'm really digging and deconstruction a few different buildpacks trying to understand how they really work.  supply, compile, detect, etc.  Especially the multi buildpack POV. 

So I think i'm starting to get the details straight but I have some basic architecture questions. 

1) For example, if the core functionality of the buildpack is a compiled app (like go or java) and not a scripted language but the github repo only contains the source code then somewhere in this process it must be compiled.  If I want to use the buildpack as an online buildpack that means that the buildpack has to be "built" after it is cloned down during staging.  Is this the correct path?  Or should it be built and the binary pushed back with the code into github? This seems wrong. Somebody hit me with the clue stick. 

If your buildpack itself needs to be compiled, you have two choices:

1.) Compile locally, package a buildpack and install with `cf create-buildpack`.
2.) Use a shim script to build then run your buildpack. If you want to support `cf push -b https://git-url/my-buildpack` then you have to do this.


and the script it runs to install Golang.

 

2) Second, somewhat tangent question. I know that CloudFoundry is doing tons of work around envoy/istio, but we have a more pressing need right now.  My company has a need to connect many apps and CloudFoundry is just one small part of our runtime. Would it be possible to write a final build pack for envoy and have it basically bind to PORT and then pass the calls to whatever was the 2nd to last buildpack? java, ruby, go, etc.  Forgive me if this is a silly question, but trying to find the edge of this new multi build pack world. Need to inject some custom ingress/egress rules. 

The final buildpack can basically do what ever it wants, since it gets to dictate the start command. If you want it to start Envoy + some other stuff you can do that. A couple things to keep in mind. First, if you're starting multiple processes in the app container you need to manage them. If one fails, you need to make it so that they all fail, or you'll have unexpected behavior. For example, if Envoy is running but your second process, presumably your actual app, crashes then you need to make sure they all crash so that the platform can detect the crash and restart your app. Second, multiple processes in the container makes it harder to reason about available memory, be especially careful if you're running a Java app. Lastly, only the supply script will run for non-final buildpacks. If you need finalize to run for a non-final buildpack, like to get the proper Java start command, then that's not going to be easy.

Hope that helps!

Dan


_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#8559) | [hidden email] | [hidden email] | Mute This Topic | New Topic

Your Subscription | [hidden email] | Unsubscribe [[hidden email]]

_._,_._,_
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Buildpack deep dive questions...

José Riguera López
In reply to this post by Cade Thacker
I have a kind of same issue as you trying to understand how a buildpack works and creating a new one. The golang library used in the official buildpacks has no documentation, so it makes really difficult starting from zero, specially if we are talking of something which should be a "helper".
I made a grafana buildpack with bash scripts (https://github.com/SpringerPE/cf-grafana-buildpack) which think it makes easy to understand what each step (the scripts in `bin` folder) are responsible for, and the argument parameters (folders) they get. Is specially important the correct usage of the caching folder to avoid downloading again and again the binaries (resources), because there are other examples on github I think they do not properly manage that part (ofc, the buildpack works but it downloads again and again the resources).

Jose,

El lun., 25 mar. 2019 a las 15:45, Cade Thacker (<[hidden email]>) escribió:
Long time reader... first time poster :D 

I'm really digging and deconstruction a few different buildpacks trying to understand how they really work.  supply, compile, detect, etc.  Especially the multi buildpack POV. 

So I think i'm starting to get the details straight but I have some basic architecture questions. 

1) For example, if the core functionality of the buildpack is a compiled app (like go or java) and not a scripted language but the github repo only contains the source code then somewhere in this process it must be compiled.  If I want to use the buildpack as an online buildpack that means that the buildpack has to be "built" after it is cloned down during staging.  Is this the correct path?  Or should it be built and the binary pushed back with the code into github? This seems wrong. Somebody hit me with the clue stick. 

2) Second, somewhat tangent question. I know that CloudFoundry is doing tons of work around envoy/istio, but we have a more pressing need right now.  My company has a need to connect many apps and CloudFoundry is just one small part of our runtime. Would it be possible to write a final build pack for envoy and have it basically bind to PORT and then pass the calls to whatever was the 2nd to last buildpack? java, ruby, go, etc.  Forgive me if this is a silly question, but trying to find the edge of this new multi build pack world. Need to inject some custom ingress/egress rules. 

-- 
--cade



--
José Riguera López <[hidden email]>
_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#8560) | [hidden email] | [hidden email] | Mute This Topic | New Topic

Your Subscription | [hidden email] | Unsubscribe [[hidden email]]

_._,_._,_