Re: [CAUTION] Re: [cf-dev] Proposed BOSH logging interface

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [CAUTION] Re: [cf-dev] Proposed BOSH logging interface

Marco Voelz

Dear Jesse,

 

did anything come out of this proposal? Did you end up picking up this track of work?

 

Warm regards

Marco

 

From: <[hidden email]> on behalf of Marco Voelz <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Tuesday, 8. May 2018 at 10:08
To: "[hidden email]" <[hidden email]>, Dmitriy Kalinin <[hidden email]>
Subject: [CAUTION] Re: [cf-dev] Proposed BOSH logging interface

 

Dear Jesse,

 

Thanks for putting this proposal out there. We would be happy to see an automated logfile forwarding mechanism. Here's a couple of comments on your initial points:

* Including the filename in the syslog metadata is very useful and something we'd really like to have. Currently it is something we're working around a bit.

* The appname/tag field should probably contain the release's name as well as a prefix. My proposal here is `<deployment name>.<instance group name>.<job name>`. wdyt?

* We haven't made any particular use of the priority field, so losing control over this field wouldn't matter for out use-cases. Severity is usually something that the actual log message needs to contain, as the logger's severity can only be set on its initial creation, afaik.

* Restricting the depth of recursion seems reasonable. So far, I don't think we're using bosh releases which have more than 1 folder below their /var/vcap/sys/log/<job name>/ folder.

 

Concerning the requirements about permissions on the logfiles you'd want to forward: Did you talk to Dmitriy/the BOSH team about this? With stemcell series 3541.x the permissions on the standard folders below /var/vcap were tightened a bit, so just wanted to make sure that your assumptions are in line with the upcoming changes in the stemcells.

 

Warm regards

Marco


From: [hidden email] <[hidden email]> on behalf of Jesse T. Alford <[hidden email]>
Sent: Tuesday, April 3, 2018 12:55:38 AM
To: [hidden email]
Subject: [cf-dev] Proposed BOSH logging interface

 

Hello! We're the CF Platform Logging team. We maintain `syslog-release` and have been working to improve and regularize platform logging behavior.

 

This is a proposal intended to establish reasonable expectations about what should be logged and what should be forwarded in bosh-deployed cloud systems.

 

Historically, it has been up to each release to provide for their log forwarding, if any. We intend `syslog-release` to provide a consistent interface useful enough to replace all other provisions for the forwarding of logs from bosh jobs.

 

## Proposed Interface

If log forwarding is enabled, some files in `/var/vcap/sys/log` (and its subdirectories, recursively), will have any line written to them forwarded as the MSG portion of an RFC5424 compliant syslog message. Which files are forwarded is governed first by file extension, and secondarily by file permissions.

 

`syslog-release` attempts to read any file ending in `.log`.

(This allows us to avoid forwarding rotated logs, swapfiles, etc.)

It will forward from such files if either of the following are true:

- it is world-readable

- it is readable to the `vcap` group

 

In particular, this means that logs will not be forwarded from files where:

- user and group are root:root

- user and group are vcap:root or vcap:none

- user and group are vcap:vcap, but it is not group-readable

 

…unless they are world-readable.

 

We think that this interface will allow us to avoid running a log forwarder with elevated permissions, while also allowing jobs to, for instance, write DEBUG or similar logs to a file that is not group-readable, thus improving their security and reducing the load on the logging system while still making them available on the ephemeral disk for debugging purposes.

 

## Questions

There are a couple of things around this interface we're especially interested in feedback on, in addition to the obvious "will this be a problem for you" overall question.

 

We may have to have a proviso that the depth of this is not unlimited. This depends somewhat on what is inexpensive to implement and maintain, and is an area we'd appreciate feedback on. Is three levels deep from `/var/vcap/sys/log` (i.e. `/var/vcap/sys/log/jobname/processname/*`) enough? Would four be?

 

In the old way of doing things, more control over the PRI information and other syslog fields was available to release authors. Logs forwarded from files currently all come out as PRI 14, which translates to Facility: User, Severity: Info. Additionally, the appname/tag field is set to the name of the directory containing the log file. Is this enough/good info? If we were to include the filename, too, would that be useful? Sufficient?

 

## Testing with the Proposed Interface

We have recently implemented a feature to help release authors evaluate the proposed interface. If you set `syslog.respect_file_permissions: true`, blackbox will not be run with elevated capabilities, and you'll be able to see what is and isn't forwarded under the proposed interface.

_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#8124) | [hidden email] | [hidden email] | Mute This Topic | New Topic

Your Subscription | [hidden email] | Unsubscribe [[hidden email]]

_._,_._,_
Reply | Threaded
Open this post in threaded view
|

Re: [CAUTION] Re: [cf-dev] Proposed BOSH logging interface

Jesse T. Alford
We haven't done anything beyond proposing the interface and implementing the option to respect permissions.

Since the time of this proposal, BPM has implemented a feature that should allow us to run Blackbox in it, mounting the logs directory as read-only. We haven't tried it yet. Assuming it works, this would also reduce our concerns about running blackbox with read access to the entire file system.

Regarding your other feedback about what should go in the tags or structured data, we've not formally taken any of that on board; development of syslog-release is currently paused. I'd suggest putting these things up as issues on the syslog-release repo;
awareness of those is more likely to be durable enough to remain visible until such time as there's a team on this.

On Wed, Jul 4, 2018 at 4:32 AM Voelz, Marco <[hidden email]> wrote:

Dear Jesse,

 

did anything come out of this proposal? Did you end up picking up this track of work?

 

Warm regards

Marco

 

From: <[hidden email]> on behalf of Marco Voelz <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Tuesday, 8. May 2018 at 10:08
To: "[hidden email]" <[hidden email]>, Dmitriy Kalinin <[hidden email]>
Subject: [CAUTION] Re: [cf-dev] Proposed BOSH logging interface

 

Dear Jesse,

 

Thanks for putting this proposal out there. We would be happy to see an automated logfile forwarding mechanism. Here's a couple of comments on your initial points:

* Including the filename in the syslog metadata is very useful and something we'd really like to have. Currently it is something we're working around a bit.

* The appname/tag field should probably contain the release's name as well as a prefix. My proposal here is `<deployment name>.<instance group name>.<job name>`. wdyt?

* We haven't made any particular use of the priority field, so losing control over this field wouldn't matter for out use-cases. Severity is usually something that the actual log message needs to contain, as the logger's severity can only be set on its initial creation, afaik.

* Restricting the depth of recursion seems reasonable. So far, I don't think we're using bosh releases which have more than 1 folder below their /var/vcap/sys/log/<job name>/ folder.

 

Concerning the requirements about permissions on the logfiles you'd want to forward: Did you talk to Dmitriy/the BOSH team about this? With stemcell series 3541.x the permissions on the standard folders below /var/vcap were tightened a bit, so just wanted to make sure that your assumptions are in line with the upcoming changes in the stemcells.

 

Warm regards

Marco


From: [hidden email] <[hidden email]> on behalf of Jesse T. Alford <[hidden email]>
Sent: Tuesday, April 3, 2018 12:55:38 AM
To: [hidden email]
Subject: [cf-dev] Proposed BOSH logging interface

 

Hello! We're the CF Platform Logging team. We maintain `syslog-release` and have been working to improve and regularize platform logging behavior.

 

This is a proposal intended to establish reasonable expectations about what should be logged and what should be forwarded in bosh-deployed cloud systems.

 

Historically, it has been up to each release to provide for their log forwarding, if any. We intend `syslog-release` to provide a consistent interface useful enough to replace all other provisions for the forwarding of logs from bosh jobs.

 

## Proposed Interface

If log forwarding is enabled, some files in `/var/vcap/sys/log` (and its subdirectories, recursively), will have any line written to them forwarded as the MSG portion of an RFC5424 compliant syslog message. Which files are forwarded is governed first by file extension, and secondarily by file permissions.

 

`syslog-release` attempts to read any file ending in `.log`.

(This allows us to avoid forwarding rotated logs, swapfiles, etc.)

It will forward from such files if either of the following are true:

- it is world-readable

- it is readable to the `vcap` group

 

In particular, this means that logs will not be forwarded from files where:

- user and group are root:root

- user and group are vcap:root or vcap:none

- user and group are vcap:vcap, but it is not group-readable

 

…unless they are world-readable.

 

We think that this interface will allow us to avoid running a log forwarder with elevated permissions, while also allowing jobs to, for instance, write DEBUG or similar logs to a file that is not group-readable, thus improving their security and reducing the load on the logging system while still making them available on the ephemeral disk for debugging purposes.

 

## Questions

There are a couple of things around this interface we're especially interested in feedback on, in addition to the obvious "will this be a problem for you" overall question.

 

We may have to have a proviso that the depth of this is not unlimited. This depends somewhat on what is inexpensive to implement and maintain, and is an area we'd appreciate feedback on. Is three levels deep from `/var/vcap/sys/log` (i.e. `/var/vcap/sys/log/jobname/processname/*`) enough? Would four be?

 

In the old way of doing things, more control over the PRI information and other syslog fields was available to release authors. Logs forwarded from files currently all come out as PRI 14, which translates to Facility: User, Severity: Info. Additionally, the appname/tag field is set to the name of the directory containing the log file. Is this enough/good info? If we were to include the filename, too, would that be useful? Sufficient?

 

## Testing with the Proposed Interface

We have recently implemented a feature to help release authors evaluate the proposed interface. If you set `syslog.respect_file_permissions: true`, blackbox will not be run with elevated capabilities, and you'll be able to see what is and isn't forwarded under the proposed interface.

_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#8165) | [hidden email] | [hidden email] | Mute This Topic | New Topic

Your Subscription | [hidden email] | Unsubscribe [[hidden email]]

_._,_._,_