LogStreamName is not configurable

[Ady88w1]

Describe the feature

I'm migrating a custom logging solution using CloudWatch from the AWS SDK to NLog. This package seems to randomize the log stream name by placing a guid into the LogStream.

Use Case

Each time the executable restarts, a new log stream is created which looses the grouping and search ability you get from grouping logs into a common log stream.

Proposed Solution

Allow the LogStreamName name to be configurable, in the same way that a file name can be configurable when logging to a file.

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

AWS .NET SDK and/or Package version used

NLog 4.7

Targeted .NET Platform

.NET Standard

Operating System and version

Agnostic

[ashishdhingra]

Hi @Ady88w1,

Thanks for opening feature request. Please refer #1 (comment). The way CloudWatch Logs is designed is each process instance needs its own stream.

Thanks,
Ashish

[Ady88w1]

Hi @Ady88w1,

Thanks for opening feature request. Please refer #1 (comment). The way CloudWatch Logs is designed is each process instance needs its own stream.

Thanks, Ashish

Hi @ashishdhingra,

I am aware of that, please read my original post. I am already using CloudWatch, I just wanted to migrate to NLog, or some other logging format. This library seems very opinionated about how to define log streams, which I don't really understand as it's not anywhere else, other than this.

For example, I run long processes on Spot instances, when the spot instance is shut down, and resumed, I want the task to continue using the log stream it was previously using.

e.g. LogStreamName:
/[uniquetaskid]

As the task is unique, there will only ever be one instance using the log stream.

You resume a Log Stream but calling DescribeLogStreams and getting the current uploadSequenceToken.

[Ady88w1]

It seems it's faster to simply create a PR: #196

[Ady88w1]

@normj I'm really unsure how you can say there is no good way to make that happen?

for example (NLog):
<target name="aws" type="AWSTarget" logGroup="my/log" logStreamUniqueKey="${date:format=yyyy-MM-ddTHH.mm.ss.fff}" region="us-east-1" />

Would replicate the existing functionality, alternatively you could use any other Layout Renderer, such as guid.
<target name="aws" type="AWSTarget" logGroup="my/log" logStreamUniqueKey="${guid}" region="us-east-1" />

Considering this is supposed to be an AWS lead project, I'm unsure why you are so insistent to remove AWS CloudWatch features.

[normj]

Okay so you can in NLog have a way of make logStreamUniqueKey work like the existing behavior. You might be able to come up with some schema in the other logging libraries but you know most users would just set the value to a static value.

I am protective against adding this feature because it will make the performance of the library unacceptable when the PutLogEvents is failing due to the sequence token out of date. If we don't make the best practice for using CloudWatch Logs the obvious pick the team will be hammered with GitHub issues about poor performance in prod.

This library is following the same pattern as every other AWS service that I can think of that integrates with CloudWatch Logs and has a log stream per process.

[Ady88w1]

@normj I understand your concern at this, but I think preventing a developer from using CloudWatch in the way they want to use it is not a good solution either (In my case logs of a task can be retrieved by the UI). Maybe describing the property in a way to discourage improper use would be the way forward?

For example, in React you cannot set HTML directly, any HTML in a string is added to a text node to prevent cross site scripting issues. However because there are legitimate reasons why you may want to set raw HTML, there is a property called dangerouslySetInnerHTML. The language of the property is clear that using the property is ill advised, and should only be set if you know what you are doing.

[ryanwilliams83]

I'm hoping to use the EC2 Instance Id as the LogStreamName but it doesn't appear that I have the option to configure it which is really frustrating it.

I can achieve the desired outcome by logging to disk and configuring the CloudWatchLogs agent to upload the logs to CloudWatch in the desired naming convention.

Surely we can get a thread safe implementation of the AWSLogger provider that allows people to nominate the LogStreamName (albeit with documentation that explains the importance of not having two computers or even two applications attempting to write to the same log stream concurrently).