[cf-dev] Loggregator Architecture Change: Independently Scalable Syslog

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[cf-dev] Loggregator Architecture Change: Independently Scalable Syslog

Adam Hevenor
Hi All -

The loggregator team is planning a track of work that results in some architectural changes to how syslog drains are implemented. I wanted to share our plans and solicit any feedback about this new approach.

Brief:  Developers and operators can use syslog drains to bind the logs of a specific app to a third party tool, but doing so puts additional strain on logs delivered through the firehose. These changes eliminate this burden and pave the way for more developer features related to syslog while keeping the same user experience.

For more details please see: https://docs.google.com/document/d/1B31BWuPVGYIbQaEVfFhmTmp8_5t8RwGGHY_4q4unojI/edit# 

Please let me know your comments, feedback or ideas about the proposal.
Thanks!
Adam
Reply | Threaded
Open this post in threaded view
|

[cf-dev] Re: Loggregator Architecture Change: Independently Scalable Syslog

Vlad Iovanov
Cancel

On Jan 31, 2017 14:42, "Adam Hevenor" <[hidden email]> wrote:
Hi All -

The loggregator team is planning a track of work that results in some architectural changes to how syslog drains are implemented. I wanted to share our plans and solicit any feedback about this new approach.

Brief:  Developers and operators can use syslog drains to bind the logs of a specific app to a third party tool, but doing so puts additional strain on logs delivered through the firehose. These changes eliminate this burden and pave the way for more developer features related to syslog while keeping the same user experience.

For more details please see: https://docs.google.com/document/d/1B31BWuPVGYIbQaEVfFhmTmp8_5t8RwGGHY_4q4unojI/edit#

Please let me know your comments, feedback or ideas about the proposal.
Thanks!
Adam
Reply | Threaded
Open this post in threaded view
|

[cf-dev] Re: Loggregator Architecture Change: Independently Scalable Syslog

Marco Voelz
In reply to this post by Adam Hevenor

Dear Adam,

 

thanks for sharing the document. Could you please make it accessible for everyone? All I currently get is

 

"You need permission

Want in? Ask for access, or switch to an account with permission. Learn more"

 

Thanks and warm regards

Marco

 

 

From: Adam Hevenor <[hidden email]>
Reply-To: "Discussions about Cloud Foundry projects and the system overall." <[hidden email]>
Date: Tuesday, 31 January 2017 at 23:41
To: "[hidden email]" <[hidden email]>
Subject: [cf-dev] Loggregator Architecture Change: Independently Scalable Syslog

 

Hi All -

 

The loggregator team is planning a track of work that results in some architectural changes to how syslog drains are implemented. I wanted to share our plans and solicit any feedback about this new approach.

 

Brief:  Developers and operators can use syslog drains to bind the logs of a specific app to a third party tool, but doing so puts additional strain on logs delivered through the firehose. These changes eliminate this burden and pave the way for more developer features related to syslog while keeping the same user experience.

 

 

Please let me know your comments, feedback or ideas about the proposal.

Thanks!

Adam

 

Reply | Threaded
Open this post in threaded view
|

[cf-dev] Re: Re: Loggregator Architecture Change: Independently Scalable Syslog

Adam Hevenor
Hi All -

I have updated the sharing settings. Everyone should now be able to see and comment.

Thanks
Adam
Reply | Threaded
Open this post in threaded view
|

[cf-dev] Re: Re: Re: Loggregator Architecture Change: Independently Scalable Syslog

Marco Voelz

Thanks Adam for the permissions. The linked “Inception notes” document [1] seems to also have problems with permissions. Could you please also take care of that?

 

Thanks and warm regards

Marco

 

[1] https://docs.google.com/document/d/1LZCiFSe785BceKg9AM5btrnKucNfL11r1zobRuNy7Bk/edit

 

 

From: Adam Hevenor <[hidden email]>
Reply-To: "Discussions about Cloud Foundry projects and the system overall." <[hidden email]>
Date: Wednesday, 1 February 2017 at 16:39
To: "[hidden email]" <[hidden email]>
Subject: [cf-dev] Re: Re: Loggregator Architecture Change: Independently Scalable Syslog

 

Hi All -

 

I have updated the sharing settings. Everyone should now be able to see and comment.

 

Thanks

Adam

 

Reply | Threaded
Open this post in threaded view
|

[cf-dev] Re: Re: Re: Re: Loggregator Architecture Change: Independently Scalable Syslog

Marco Voelz

+Adam directly to ping again.

 

From: "Voelz, Marco" <[hidden email]>
Reply-To: "Discussions about Cloud Foundry projects and the system overall." <[hidden email]>
Date: Thursday, 2 February 2017 at 21:05
To: "Discussions about Cloud Foundry projects and the system overall." <[hidden email]>
Subject: [cf-dev] Re: Re: Re: Loggregator Architecture Change: Independently Scalable Syslog

 

Thanks Adam for the permissions. The linked “Inception notes” document [1] seems to also have problems with permissions. Could you please also take care of that?

 

Thanks and warm regards

Marco

 

[1] https://docs.google.com/document/d/1LZCiFSe785BceKg9AM5btrnKucNfL11r1zobRuNy7Bk/edit

 

 

From: Adam Hevenor <[hidden email]>
Reply-To: "Discussions about Cloud Foundry projects and the system overall." <[hidden email]>
Date: Wednesday, 1 February 2017 at 16:39
To: "[hidden email]" <[hidden email]>
Subject: [cf-dev] Re: Re: Loggregator Architecture Change: Independently Scalable Syslog

 

Hi All -

 

I have updated the sharing settings. Everyone should now be able to see and comment.

 

Thanks

Adam

 

Reply | Threaded
Open this post in threaded view
|

[cf-dev] Re: Re: Re: Re: Loggregator Architecture Change: Independently Scalable Syslog

Adam Hevenor
Hi Marco - 

I went ahead and changed the sharing settings. That said, keep in mind there were some changes that occurred after the inception so some of the details in the notes may be confusing/misleading. Refer to the proposal for a more accurate description of the final approach.

Adam 

On Fri, Feb 10, 2017 at 2:19 AM, Voelz, Marco <[hidden email]> wrote:

+Adam directly to ping again.

 

From: "Voelz, Marco" <[hidden email]>
Reply-To: "Discussions about Cloud Foundry projects and the system overall." <[hidden email]>
Date: Thursday, 2 February 2017 at 21:05
To: "Discussions about Cloud Foundry projects and the system overall." <[hidden email]>
Subject: [cf-dev] Re: Re: Re: Loggregator Architecture Change: Independently Scalable Syslog

 

Thanks Adam for the permissions. The linked “Inception notes” document [1] seems to also have problems with permissions. Could you please also take care of that?

 

Thanks and warm regards

Marco

 

[1] https://docs.google.com/document/d/1LZCiFSe785BceKg9AM5btrnKucNfL11r1zobRuNy7Bk/edit

 

 

From: Adam Hevenor <[hidden email]>
Reply-To: "Discussions about Cloud Foundry projects and the system overall." <[hidden email]>
Date: Wednesday, 1 February 2017 at 16:39
To: "[hidden email]" <[hidden email]>
Subject: [cf-dev] Re: Re: Loggregator Architecture Change: Independently Scalable Syslog

 

Hi All -

 

I have updated the sharing settings. Everyone should now be able to see and comment.

 

Thanks

Adam

 




--
Adam Hevenor
----------------------------
PM: Loggregator
Reply | Threaded
Open this post in threaded view
|

[cf-dev] Re: Loggregator Architecture Change: Independently Scalable Syslog

Michael Altenhofen
In reply to this post by Adam Hevenor
Hi Adam,

I know I'm a bit late, but I've just added two comments to the document that you've shared in your post.

To us it will be important to understand how that new architecture will affect recipients, i.e. syslog drain providers.
Here, the most interesting questions are:

* will connection handling stay the way it was, i.e. the new component will try to re-establish a connection to a syslog drain in case it was lost?
* will this new architecture also allow for some load-balancing towards a syslog drain, i.e. accept, e.g., multiple addresses?

Thanks and Best Regards

Michael Altenhofen
SAP SE
Reply | Threaded
Open this post in threaded view
|

[cf-dev] Re: Re: Loggregator Architecture Change: Independently Scalable Syslog

Adam Hevenor
Hi Michael -

Thanks for these questions. I'll answer them here:

Our intent is to keep the reconnection logic the same. That said we recently realized that we do not have an equivalent back off strategy for HTTPS drains. We are working on a feature to match this to the previous functionality.

The release does not currently allow for load balancing of a single drain. That certainly is an interesting idea and this architecture is better positioned to consider this feature. Is this a common use case for you?

Adam
Reply | Threaded
Open this post in threaded view
|

[cf-dev] Re: Re: Re: Loggregator Architecture Change: Independently Scalable Syslog

Michael Altenhofen
Hi Adam,

as you might have guessed from my questions, we're interested in more resilient setups for syslog drains.
Not sure whether you've had the chance to listen to my colleague Istvan Ballok's talk at the CF Summit (http://sched.co/AJmc) where he reported on our journey regarding application logging.

Our current solution is implemented as a managed service where the service binding will include a syslog_drain_url of one of our so-called SLEEVE forwarder instances. If one of those instances is going down, I guess it's not that easy to have the corresponding doppler instance (in the current setup) re-connect to a different instance. So, currently we think about using a load balancer in front of those instance cluster, but that introduces another component that may fail.

What "hurts" us at the moment, too, is that we have to re-establish existing bindings on instance restarts, e.g. during an update. This is needed to re-cover "meta data" (app name, space name, org name) for logs that we want to attach to logs. As we currently restrict ourselves to using the official CF API for that, this may take fairly long on large landscapes. We know that there are internal APIs, like the bulk APIs, but as they are marked as "internal", there's no guarantee that they will not change and we want to avoid being trapped by API changes that we didn't spot.

Best

Michael