How do we add Ingest Budget to this docker?

[mhobotpplnet]

Just trying to work with some new fields , and was wondering where do I add _budget field ? is it the sources or collector it self?

[maimaisie]

Hi @mhobotpplnet , you can supply your own user.properties file to specify the budget field on the collector.

This section in the README explains how to supply your own properties file when starting the container.

This doc page talks more in detail on what fields are supported in this file. Search for "Ingest Budget" on this page to see how to format this field in user.properties file. It should be something like fields=_budget=<your_budget_field_value>

[mhobotpplnet]

@maimaisie yeah I saw that, I got bit of an issue where I supply the secrets.

So my question is, if I have info below in my docker-compose, and if I add custom user.properties SUMO_GENERATE_USER_PROPERTIES as well, will info below also be added from docker-compose? Or do I need to hardcode info below in custom user.properties file?

      SUMO_CLOBBER: 'false'
      SUMO_COLLECTOR_NAME_PREFIX: "${MYSTUFF_ENVIRONMENT}-"
      SUMO_SOURCES_JSON: /config/sumo/sources.json
      SUMO_SYNC_SOURCES: 'false'

[mhobotpplnet]

Example how it looks w/o custom properties atm

root@sumologic-collector-docker-sumologic-12:/# cat /opt/SumoCollector/config/us
er.properties
#
#Tue May 21 16:48:12 UTC 2019
accessid=myID
name=dev-sumologic-collector-docker-sumologic-12
sources=/config/sumo/sources.json
url=https\://collectors.sumologic.com
ephemeral=true
clobber=false

[maimaisie]

Unfortunately it doesn't support combining custom user.properties file and properties passed from environment variables. So all the properties should be specified through user.properties if you supply your own.

In the future we will add ingest budget assignment through environment variable as well.

[mhobotpplnet]

Show me where you would expose that as an env, I might code it and push it as PR.

[mhobotpplnet]

Probably somewhere here right?

https://github.com/SumoLogic/sumologic-collector-docker/blob/70ca21adb4957ab3e189ddb57c40e044903e92d7/run.sh

[maimaisie]

Yes, basically similar to this: #79

[mhobotpplnet]

Very odd, so I added
SUMO_INGEST_BUDGET=${SUMO_INGEST_BUDGET:_budget=test-budget} at the top of the file where rest of the envs are and for declared options somewhere around line 96 I added ["SUMO_INGEST_BUDGET"]="fields" ran the docker locally and I can't see any user.properties changes. So I did with docker run -e SUMO_INGEST_BUDGET thinking I must add this env then, that did not do anything else.

Any other pointers?

My user.properties file should at the end have
fields=_budget=test-budget

Whatever I do I don't get that extra env in user.properties. just this

#Wed May 22 03:39:24 UTC 2019
accessid=mykey
name=collector_container-a7dda42ffce8
sources=/etc/sumo-sources.json
url=https\://collectors.sumologic.com
ephemeral=true

Update: If I run docker -e for SUMO_COLLECTOR_EPHEMERAL=false , it works just fine which kind of tells me that the sumo wrapper has those variables predefined.

[maimaisie]

Update: If I run docker -e for SUMO_COLLECTOR_EPHEMERAL=false , it works just fine which kind of tells me that the sumo wrapper has those variables predefined.

That is very strange. They should be independent properties. Can you maybe create a PR with what you had so I can review and possibly find where the problem is?

[MattPumphrey]

This would be a great feature to have, as if we are running an entirely dynamic workload that the nodes scale but up and down and the only agent on them is the sumologic collector for log collection, then having a seperate user.properties file when we have been using the generated one does not bode with our ability to use the ingest budgets on scale up of nodes.

I just started looking at this and if I find a workable solution, I will make a pr.