[cf-dev] Follow up on multiple line log outputs in CF

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[cf-dev] Follow up on multiple line log outputs in CF

Li, George
Hi,

this is a follow up on the archived posting https://groups.google.com/a/cloudfoundry.org/forum/?utm_medium=email&utm_source=footer#!msg/vcap-dev/B1W6_vO0oyo/84X1eAtFsKoJ. I cannot find any new postings on that thread.
I am using Cloud Foundry version "6.11.2-2a26d55-2015-04-27T21:11:44+00:00" and want to know what options I have to handle multiple line logs in a multi-tenant environment. Since multiple instances of multiple applications are all sending logs to a single Logstash server, is it best to avoid having multiple lines in my log? I can live with sticking to single line logs except for outputting exception stack trace, not to mention that I only have control over my code. 

Thanks.

_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Follow up on multiple line log outputs in CF

James Bayer
george,

can you explain the scenario that is impacting you currently? e.g. reading a java stack trace? the app index is shown in the log line. when using the loggregator websocket client to retrieve logs you should also get the application id. perhaps that is enough to correlate whatever UI you have on top of logstash?


On Tue, May 12, 2015 at 7:40 AM, Li, George <[hidden email]> wrote:
Hi,

this is a follow up on the archived posting https://groups.google.com/a/cloudfoundry.org/forum/?utm_medium=email&utm_source=footer#!msg/vcap-dev/B1W6_vO0oyo/84X1eAtFsKoJ. I cannot find any new postings on that thread.
I am using Cloud Foundry version "6.11.2-2a26d55-2015-04-27T21:11:44+00:00" and want to know what options I have to handle multiple line logs in a multi-tenant environment. Since multiple instances of multiple applications are all sending logs to a single Logstash server, is it best to avoid having multiple lines in my log? I can live with sticking to single line logs except for outputting exception stack trace, not to mention that I only have control over my code. 

Thanks.

_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




--
Thank you,

James Bayer

_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Follow up on multiple line log outputs in CF

Li, George
James,

so basically Loggregator does not support such correlation itself and it is totally left to the downstream processor to do the aggregation, right?
When my service write exception stack trace into the log as follows:

2015-05-18 14:40:41 +0000 WARN com.pearson.grid.registrar.commonutils.logging.LoggingMessageObserver [LoggingMessageObserver.java:125] GET http://localhost:3004/ threw exception: FileNotFoundException - E:\Src\Registrar\escrow\service\config\version.properties (The system cannot find the file specified)
java.io.FileInputStream.open(Native Method)
java.io.FileInputStream.<init>(FileInputStream.java:138)
com.pearson.grid.registrar.commonutils.Version.readProperties(Version.java:77)

Loggregator sends the following to downstream logstash:

2015-05-18 14:40:41 +0000 [App/0] OUT [escrow] [pool-3-thread-1052] WARN com.pearson.grid.registrar.commonutils.logging.LoggingMessageObserver [LoggingMessageObserver.java:142] GET http://escrow.cf.pd-cloud.com/ threw exception: FileNotFoundException - config\version.properties (No such file or directory)
2015-05-18 14:40:41 +0000 [App/0] OUT java.io.FileInputStream.open(Native Method)
2015-05-18 14:40:41 +0000 [App/0] OUT java.io.FileInputStream.<init>(FileInputStream.java:146)
2015-05-18 14:40:41 +0000 [App/0] OUT com.pearson.grid.registrar.commonutils.Version.readProperties(Version.java:77)

Ideally we want to keep all these as a single log "event" in the final log. The way to do this in logstash is to config logstash so it would aggregate all lines not staring with a timestamp (or any identifiable marker) to previous lines:

input {
  file {
    codec => multiline {
      pattern => "^%{TIMESTAMP_ISO8601} "
        ...
    }
}

So I think my exact question is if there is a similar way to config loggregator so it does the same thing.

In the previous thread https://groups.google.com/a/cloudfoundry.org/forum/?utm_medium=email&utm_source=footer#!msg/vcap-dev/B1W6_vO0oyo/84X1eAtFsKoJ,  David Lee said "...As for multiline output, we should have fixed this a couple of builds back...", I wonder what was really fixed related to multiline output at that time.

Thanks.

George.

On Sun, May 17, 2015 at 8:26 AM, James Bayer <[hidden email]> wrote:
george,

can you explain the scenario that is impacting you currently? e.g. reading a java stack trace? the app index is shown in the log line. when using the loggregator websocket client to retrieve logs you should also get the application id. perhaps that is enough to correlate whatever UI you have on top of logstash?


On Tue, May 12, 2015 at 7:40 AM, Li, George <[hidden email]> wrote:
Hi,

this is a follow up on the archived posting https://groups.google.com/a/cloudfoundry.org/forum/?utm_medium=email&utm_source=footer#!msg/vcap-dev/B1W6_vO0oyo/84X1eAtFsKoJ. I cannot find any new postings on that thread.
I am using Cloud Foundry version "6.11.2-2a26d55-2015-04-27T21:11:44+00:00" and want to know what options I have to handle multiple line logs in a multi-tenant environment. Since multiple instances of multiple applications are all sending logs to a single Logstash server, is it best to avoid having multiple lines in my log? I can live with sticking to single line logs except for outputting exception stack trace, not to mention that I only have control over my code. 

Thanks.

_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




--
Thank you,

James Bayer


_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Follow up on multiple line log outputs in CF

Mike Youngstrom
I don't believe there is a solution in raw Loggregator to fix this.

Multi line log messages is a major issue we deal with regularly today in our deployment.  Although there are workarounds to the issue like the logstash config you posted, it is somewhat dependent upon how the application logs are formatted which is sub optimal.

I'd personally like to see a syslog port available to the app container that applications can send logs to in addition to channeling STDERR and STDOUT.  That would make issues like much easier to handle downstream and would help eliminate deployed app specific hacks like we have to do today in logstash.

I talked with a LAMB engineer at Summit and they didn't believe syslog for deployed apps was anywhere on the LAMB roadmap.  Can anyone confirm that?  I could have sworn I'd heard mention of syslog for app mentioned in the past.

Mike

On Mon, May 18, 2015 at 9:32 AM, Li, George <[hidden email]> wrote:
James,

so basically Loggregator does not support such correlation itself and it is totally left to the downstream processor to do the aggregation, right?
When my service write exception stack trace into the log as follows:

2015-05-18 14:40:41 +0000 WARN com.pearson.grid.registrar.commonutils.logging.LoggingMessageObserver [LoggingMessageObserver.java:125] GET http://localhost:3004/ threw exception: FileNotFoundException - E:\Src\Registrar\escrow\service\config\version.properties (The system cannot find the file specified)
java.io.FileInputStream.open(Native Method)
java.io.FileInputStream.<init>(FileInputStream.java:138)
com.pearson.grid.registrar.commonutils.Version.readProperties(Version.java:77)

Loggregator sends the following to downstream logstash:

2015-05-18 14:40:41 +0000 [App/0] OUT [escrow] [pool-3-thread-1052] WARN com.pearson.grid.registrar.commonutils.logging.LoggingMessageObserver [LoggingMessageObserver.java:142] GET http://escrow.cf.pd-cloud.com/ threw exception: FileNotFoundException - config\version.properties (No such file or directory)
2015-05-18 14:40:41 +0000 [App/0] OUT java.io.FileInputStream.open(Native Method)
2015-05-18 14:40:41 +0000 [App/0] OUT java.io.FileInputStream.<init>(FileInputStream.java:146)
2015-05-18 14:40:41 +0000 [App/0] OUT com.pearson.grid.registrar.commonutils.Version.readProperties(Version.java:77)

Ideally we want to keep all these as a single log "event" in the final log. The way to do this in logstash is to config logstash so it would aggregate all lines not staring with a timestamp (or any identifiable marker) to previous lines:

input {
  file {
    codec => multiline {
      pattern => "^%{TIMESTAMP_ISO8601} "
        ...
    }
}

So I think my exact question is if there is a similar way to config loggregator so it does the same thing.

In the previous thread https://groups.google.com/a/cloudfoundry.org/forum/?utm_medium=email&utm_source=footer#!msg/vcap-dev/B1W6_vO0oyo/84X1eAtFsKoJ,  David Lee said "...As for multiline output, we should have fixed this a couple of builds back...", I wonder what was really fixed related to multiline output at that time.

Thanks.

George.

On Sun, May 17, 2015 at 8:26 AM, James Bayer <[hidden email]> wrote:
george,

can you explain the scenario that is impacting you currently? e.g. reading a java stack trace? the app index is shown in the log line. when using the loggregator websocket client to retrieve logs you should also get the application id. perhaps that is enough to correlate whatever UI you have on top of logstash?


On Tue, May 12, 2015 at 7:40 AM, Li, George <[hidden email]> wrote:
Hi,

this is a follow up on the archived posting https://groups.google.com/a/cloudfoundry.org/forum/?utm_medium=email&utm_source=footer#!msg/vcap-dev/B1W6_vO0oyo/84X1eAtFsKoJ. I cannot find any new postings on that thread.
I am using Cloud Foundry version "6.11.2-2a26d55-2015-04-27T21:11:44+00:00" and want to know what options I have to handle multiple line logs in a multi-tenant environment. Since multiple instances of multiple applications are all sending logs to a single Logstash server, is it best to avoid having multiple lines in my log? I can live with sticking to single line logs except for outputting exception stack trace, not to mention that I only have control over my code. 

Thanks.

_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




--
Thank you,

James Bayer


_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev



_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Follow up on multiple line log outputs in CF

Li, George
Any build-in support for multiline logging would be nice. Seeing content in one call to log4j's log.error() spreading over numerous logging records really sucks, especially for exception stacktrace.

On Mon, May 18, 2015 at 11:59 AM, Mike Youngstrom <[hidden email]> wrote:
I don't believe there is a solution in raw Loggregator to fix this.

Multi line log messages is a major issue we deal with regularly today in our deployment.  Although there are workarounds to the issue like the logstash config you posted, it is somewhat dependent upon how the application logs are formatted which is sub optimal.

I'd personally like to see a syslog port available to the app container that applications can send logs to in addition to channeling STDERR and STDOUT.  That would make issues like much easier to handle downstream and would help eliminate deployed app specific hacks like we have to do today in logstash.

I talked with a LAMB engineer at Summit and they didn't believe syslog for deployed apps was anywhere on the LAMB roadmap.  Can anyone confirm that?  I could have sworn I'd heard mention of syslog for app mentioned in the past.

Mike

On Mon, May 18, 2015 at 9:32 AM, Li, George <[hidden email]> wrote:
James,

so basically Loggregator does not support such correlation itself and it is totally left to the downstream processor to do the aggregation, right?
When my service write exception stack trace into the log as follows:

2015-05-18 14:40:41 +0000 WARN com.pearson.grid.registrar.commonutils.logging.LoggingMessageObserver [LoggingMessageObserver.java:125] GET http://localhost:3004/ threw exception: FileNotFoundException - E:\Src\Registrar\escrow\service\config\version.properties (The system cannot find the file specified)
java.io.FileInputStream.open(Native Method)
java.io.FileInputStream.<init>(FileInputStream.java:138)
com.pearson.grid.registrar.commonutils.Version.readProperties(Version.java:77)

Loggregator sends the following to downstream logstash:

2015-05-18 14:40:41 +0000 [App/0] OUT [escrow] [pool-3-thread-1052] WARN com.pearson.grid.registrar.commonutils.logging.LoggingMessageObserver [LoggingMessageObserver.java:142] GET http://escrow.cf.pd-cloud.com/ threw exception: FileNotFoundException - config\version.properties (No such file or directory)
2015-05-18 14:40:41 +0000 [App/0] OUT java.io.FileInputStream.open(Native Method)
2015-05-18 14:40:41 +0000 [App/0] OUT java.io.FileInputStream.<init>(FileInputStream.java:146)
2015-05-18 14:40:41 +0000 [App/0] OUT com.pearson.grid.registrar.commonutils.Version.readProperties(Version.java:77)

Ideally we want to keep all these as a single log "event" in the final log. The way to do this in logstash is to config logstash so it would aggregate all lines not staring with a timestamp (or any identifiable marker) to previous lines:

input {
  file {
    codec => multiline {
      pattern => "^%{TIMESTAMP_ISO8601} "
        ...
    }
}

So I think my exact question is if there is a similar way to config loggregator so it does the same thing.

In the previous thread https://groups.google.com/a/cloudfoundry.org/forum/?utm_medium=email&utm_source=footer#!msg/vcap-dev/B1W6_vO0oyo/84X1eAtFsKoJ,  David Lee said "...As for multiline output, we should have fixed this a couple of builds back...", I wonder what was really fixed related to multiline output at that time.

Thanks.

George.

On Sun, May 17, 2015 at 8:26 AM, James Bayer <[hidden email]> wrote:
george,

can you explain the scenario that is impacting you currently? e.g. reading a java stack trace? the app index is shown in the log line. when using the loggregator websocket client to retrieve logs you should also get the application id. perhaps that is enough to correlate whatever UI you have on top of logstash?


On Tue, May 12, 2015 at 7:40 AM, Li, George <[hidden email]> wrote:
Hi,

this is a follow up on the archived posting https://groups.google.com/a/cloudfoundry.org/forum/?utm_medium=email&utm_source=footer#!msg/vcap-dev/B1W6_vO0oyo/84X1eAtFsKoJ. I cannot find any new postings on that thread.
I am using Cloud Foundry version "6.11.2-2a26d55-2015-04-27T21:11:44+00:00" and want to know what options I have to handle multiple line logs in a multi-tenant environment. Since multiple instances of multiple applications are all sending logs to a single Logstash server, is it best to avoid having multiple lines in my log? I can live with sticking to single line logs except for outputting exception stack trace, not to mention that I only have control over my code. 

Thanks.

_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




--
Thank you,

James Bayer


_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev