st2 fails to store a Key-Value in the datastore if there is a "/" in the key name.

[RaviTezu]

SUMMARY

st2 fails to store a Key-Value in the datastore if there is a "/" in the key name. I am sorry if I missed anything in the docs, which says that you cannot store a key with "/" in it and please feel free to close this issue if that is the case. Thank you.

STACKSTORM VERSION

st2 3.1.0, on Python 2.7.5

OS, environment, install method:

CentOS/Docker, custom install.

Steps to reproduce the problem

bash-4.2$ st2 key set "foo / bar" "foo_bar"
ERROR: 404 Client Error: Not Found
MESSAGE: The resource could not be found. for url: http://127.0.0.1:9101/v1/keys/foo%20/%20bar

Expected Results

st2 should have accepted the key-value pair and stored it.

Actual Results

st2 has failed to store the key-value pair.

[arm4b]

If we can't save key with / in k/v storage due to API endpoints, - we should at least output validation error.

[RaviTezu]

Hi @armab I would like to take this up and send a fix.

[Kami]

I believe this issue probably lies in st2client / CLI and not the API itself (likely / character is not correctly URL encoded).

I believe we even have some st2api API level tests for keys with / in the name (and if we don't, we should add some).

[RaviTezu]

Hi @Kami
I tried via the st2 command line utility and also from the actions I have implemented, both self.action_service.set_value and self.action_service.get_value are failing with the same error. So I assume, it should be an API level issue.

[Kami]

It could be, although action service also utilizes st2client code which talks to the API :)

If there is not an existing test already, I would start with an API level test for that functionality.

[RaviTezu]

@Kami @armab I see no tests which stores a key-value pair with a "/" in the key name.

Also, I am not able to run the tests in test_kvps.py file using the following command from the root directory as mentioned in this doc on a Ubuntu machine:
nosetests --nocapture st2api/tests/unit/controllers/v1/test_kvps.py
It says ERROR: Failure: ImportError (No module named st2tests.api)

I tried exporting/setting PYTHONPATH to st2 directory but no luck. Could you please help me to get the tests running ? I tried with both Python2 and Python3.

Thanks.

[Kami]

For developing StackStorm platform itself, I would recommend you to use this Vagrant image - https://github.com/StackStorm/st2vagrantdev

In short, you need to run make requirements which will create virtualenv, install all the dependencies and set PYTHONPATH correctly.

[RaviTezu]

@Kami Thanks. This is Great.
I am able to setup the vagrant environment and I can run the tests now. Also, I have added a test which stores a KV and there is a "/" in the key name and it fails. I will dig through the code and see, if we can encode "/" someway.
@armab

[RaviTezu]

@Kami @armab
I modified the serialize method here to quote (from six.moves.urllib_parse import quote) so that the key for example a/b will be converted into a%2Fb. However, the st2 key set "a/b" "some value" is failing with resource could not be found error.

ERROR: 404 Client Error: Not Found
MESSAGE: The resource could not be found. for url: http://127.0.0.1:9101/v1/keys/a%2Fb

I think, I will have to make some modifications on the API as well to accept this kind of keys?
Thank you.

[RaviTezu]

Hi @Kami @armab Do you think, we should be able to encode and store keys with /? or I will try to implement something which will throw an error if the key has / in it. Thank you.

[arm4b]

Yes, I think it's absolutely reasonable to be able to use the / in Key name.
Quick example: https://www.consul.io/docs/commands/kv/put.html#examples

If you could make it work and support your enhancement with tests as well, - that would be a great addition 👍