Skip to content

Implement new JSON Jinja filters #617

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Kami merged 4 commits into from
Oct 2, 2017
Merged

Implement new JSON Jinja filters #617

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
39a30c4
Added documentation about from_json_string, from_yaml_string, and jso…
nmaludy Sep 29, 2017
5e3f421
Fixed build errors for new jinja filters
nmaludy Sep 29, 2017
48729ab
Fixed typo in Jinja docs
nmaludy Sep 29, 2017
468d154
Added more examples of jsonpath_query
nmaludy Sep 29, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/source/datastore.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ Key-Value pairs can also have a TTL associated with them, for automatic expiry.
simple and fully compatible with existing API and CLI commands.

If you want to store a non-string value, you can store a JSON-serialized version, and then
de-serialize it in your action/sensor code.
de-serialize it in your action/sensor code, or using the ``from_json_string``
Jinja filter. See the :doc:`/reference/jinja` documentation for more details.

This may change in future if there is sufficient interest.

Expand Down
159 changes: 129 additions & 30 deletions docs/source/reference/jinja.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,29 @@ with Jinja in the context of |st2|. Please refer to the `Jinja docs
Referencing Datastore Keys in Jinja
------------------------------------

You can use ``{{st2kv.system.foo}}`` to access key ``foo`` from datastore. Note that until
v2.1, the expression to access key ``foo`` from datastore used to be ``{{system.foo}}``
You can use ``{{ st2kv.system.foo }}`` to access key ``foo`` from datastore. Note that until
v2.1, the expression to access key ``foo`` from datastore used to be ``{{ system.foo }}``
but is now deprecated, and the leading ``st2kv.`` namespace is required.

Currently, all data in the datastore is represented as strings. To represent
complext data structures like ``dicts`` and ``lists`` the standard approach is to
convert the data structure into JSON when storing the data, then parse it when
retrieving the data.

.. code-block:: bash

# Pass the result of this expression to the action st2.kv.set
{{ {'complex': 'structure', 'foo': ['x', True]} | to_json_string }}

# Read the data back in using the st2kv and from_json_string filters
{{ st2kv.system.foo | from_json_string }}

.. _jinja-jinja-filters:

Applying Filters with Jinja
----------------------------

To use a filter ``my_filter`` on ``foo``, you use the pipe operator, e.g.: ``{{foo | my_filter}}``.
To use a filter ``my_filter`` on ``foo``, you use the pipe operator, e.g.: ``{{ foo | my_filter }}``.
Please pay attention to the data type and available filters for each data type. Since Jinja is a
text templating language, all your input is converted to text and then manipulations happen on that
value. The necessary casting at the end is done by |st2| based on information you provide in YAML
Expand Down Expand Up @@ -64,11 +77,28 @@ function in Mistral workflows natively supports decryption via the ``decrypt`` p
.. note::

Because of a bug in Mistral, these filters do not currently support the "pipe" operator filter
format (`|`) So, instead of ``'{{ _.input_str | regex_match(_.regex_pattern)}}'`` you would
format (`|`) So, instead of ``'{{ _.input_str | regex_match(_.regex_pattern) }}'`` you would
call the filter like a regular function, moving the previously input value into the first
positional argument position: ``'{{ regex_match(_.input_str, _.regex_pattern)}}'``. This will
positional argument position: ``'{{ regex_match(_.input_str, _.regex_pattern) }}'``. This will
be addressed in a future release.

from_json_string
~~~~~~~~~~~~~~~~

Converts a JSON string into an object or array (opposite of ``to_json_string``).

.. code-block:: bash

{{ value_key | from_json_string }}

from_yaml_string
~~~~~~~~~~~~~~~~

Converts a YAML string into an object or array (opposite of ``to_yaml_string``).

.. code-block:: bash

{{ value_key | from_yaml_string }}

json_escape
~~~~~~~~~~~
Expand All @@ -77,7 +107,76 @@ Adds escape characters to JSON strings.

.. code-block:: bash

{{value_key | json_escape}}
{{ value_key | json_escape }}

jsonpath_query
~~~~~~~~~~~~~~

Provides the ability to extract data from complex ``object`` data using the
`JSONPath <http://goessner.net/articles/JsonPath/>` query language. More specifically
we use the ``jsonpath-rw`` library that has its own extensions, details can be
found on the `jsonpath-rw GitHub page <https://github.com/kennknowles/python-jsonpath-rw/tree/master/jsonpath_rw>`.
Data passed into this function should be of type ``object`` or ``array``.
The result of this function will either be an array of results, or None if the
query did not return any results.
If you would like to test out your JSONPath queries prior to utilizing this filter
an online evaluator can be found `here <http://jsonpath.com/>`.

.. code-block:: bash

# Access an element in a data structure. Each level is delimited by a '.'.
# Each part of the query is the name of the field in the current level
# of the data structure.
#
# input = {'a': {'b': {'c': 1234} } }
# result = [1234]
{{ input | jsonpath_query('a.b.c') }}

# Access an index in an array/list
#
# input = {'animals': ['bird', 'rabbit', 'cat', 'dog', 'cow'] }
# result = ['rabbit']
{{ input | jsonpath_query('animals[1]') }}

# Access all indexes in an array/list
#
# input = {'animals': ['bird', 'rabbit', 'cat', 'dog', 'cow'] }
# result = ['bird', 'rabbit', 'cat', 'dog', 'cow']
{{ input | jsonpath_query('animals[*]') }}
Copy link
Member

@Kami Kami Oct 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work on the docs and examples 👍


# Access a range/slice of indexes in an array/list.
# These expressions can be read mathematically as [first, last)
# Meaning that the index of the first element is inclusive, and the index
# of the last element is exclusive (will not be included).
#
# input = {'animals': ['bird', 'rabbit', 'cat', 'dog', 'sheep'] }
# result = ['rabbit', 'cat']
{{ input | jsonpath_query('animals[1:3]') }}

# If you leave out the first number in the range/slice operator
# it will start at the beginning implicitly. It can be read as:
# "give me all data from the beginning to the index specified"
#
# input = {'animals': ['bird', 'rabbit', 'cat', 'dog', 'sheep'] }
# result = ['bird', 'rabbit']
{{ input | jsonpath_query('animals[:2]') }}

# If you leave out the last number in the range/slice operator
# it will go all the way to the end of the array implicitly.
# It can be read as: "give me all data from the index specified to the end"
#
# input = {'animals': ['bird', 'rabbit', 'cat', 'dog', 'sheep'] }
# result = ['cat', 'dog', 'sheep']
{{ input | jsonpath_query('animals[2:]') }}

# Access a field within every element of an array.
#
# input = {'people': [{'first': 'James', 'last': 'd'},
# {'first': 'Jacob', 'last': 'e'},
# {'first': 'Jayden', 'last': 'f'}]}
# result = ['James', 'Jacob', 'Jayden']
{{ input | jsonpath_query('people[*].first') }}


regex_match
~~~~~~~~~~~
Expand All @@ -86,8 +185,8 @@ Search for the pattern at beginning of the string. Returns True if found, False

.. code-block:: bash

{{value_key | regex_match('x')}}
{{value_key | regex_match("^v(\\d+\\.)?(\\d+\\.)?(\\*|\\d+)$")}}
{{ value_key | regex_match('x') }}
{{ value_key | regex_match("^v(\\d+\\.)?(\\d+\\.)?(\\*|\\d+)$") }}

regex_replace
~~~~~~~~~~~~~
Expand All @@ -100,8 +199,8 @@ Replaces substring that matches pattern with provided replacement value (backref

.. code-block:: bash

{{value_key | regex_replace("x", "y")}}
{{value_key | regex_replace("(blue|white|red)", "beautiful color \\\\1")}}
{{ value_key | regex_replace("x", "y") }}
{{ value_key | regex_replace("(blue|white|red)", "beautiful color \\\\1") }}

regex_search
~~~~~~~~~~~~
Expand All @@ -110,8 +209,8 @@ Search for pattern anywhere in the string. Returns True if found, False if not.

.. code-block:: bash

{{value_key | regex_search("y")}}
{{value_key | regex_search("^v(\\d+\\.)?(\\d+\\.)?(\\*|\\d+)$")}}
{{ value_key | regex_search("y") }}
{{ value_key | regex_search("^v(\\d+\\.)?(\\d+\\.)?(\\*|\\d+)$") }}

regex_substring
~~~~~~~~~~~~~~~
Expand All @@ -121,8 +220,8 @@ Searches for the provided pattern in a string, and returns the first matched reg

.. code-block:: bash

{{value_key | regex_substring("y")}}
{{value_key | regex_substring("^v(\\d+\\.)?(\\d+\\.)?(\\*|\\d+)$")}}
{{ value_key | regex_substring("y") }}
{{ value_key | regex_substring("^v(\\d+\\.)?(\\d+\\.)?(\\*|\\d+)$") }}

to_complex
~~~~~~~~~~
Expand All @@ -131,7 +230,7 @@ Convert data to JSON string (see ``to_json_string`` for a more flexible option)

.. code-block:: bash

{{value_key | to_complex}}
{{ value_key | to_complex }}

to_human_time_from_seconds
~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand All @@ -140,7 +239,7 @@ Given time elapsed in seconds, this filter converts it to human readable form li

.. code-block:: bash

{{ value_key | to_human_time_from_seconds}}
{{ value_key | to_human_time_from_seconds }}

to_json_string
~~~~~~~~~~~~~~
Expand All @@ -149,7 +248,7 @@ Convert data to JSON string.

.. code-block:: bash

{{value_key | to_json_string}}
{{ value_key | to_json_string }}

to_yaml_string
~~~~~~~~~~~~~~
Expand All @@ -158,7 +257,7 @@ Convert data to YAML string.

.. code-block:: bash

{{value_key | to_yaml_string}}
{{ value_key | to_yaml_string }}

use_none
~~~~~~~~
Expand All @@ -167,7 +266,7 @@ If value being filtered is None, this filter will return the string ``%*****__%N

.. code-block:: bash

{{value_key | use_none}}
{{ value_key | use_none }}

version_bump_major
~~~~~~~~~~~~~~~~~~
Expand All @@ -176,7 +275,7 @@ Bumps up the major version of supplied version field.

.. code-block:: bash

{{version | version_bump_major}}
{{ version | version_bump_major }}

version_bump_minor
~~~~~~~~~~~~~~~~~~
Expand All @@ -185,7 +284,7 @@ Bumps up the minor version of supplied version field.

.. code-block:: bash

{{version | version_bump_minor}}
{{ version | version_bump_minor }}

version_bump_patch
~~~~~~~~~~~~~~~~~~
Expand All @@ -194,7 +293,7 @@ Bumps up the patch version of supplied version field.

.. code-block:: bash

{{version | version_bump_patch}}
{{ version | version_bump_patch }}

version_compare
~~~~~~~~~~~~~~~
Expand All @@ -204,7 +303,7 @@ Compare a semantic version to another value. Returns 1 if LHS is greater or -1 i

.. code-block:: bash

{{version | version_compare("0.10.1")}}
{{ version | version_compare("0.10.1") }}

version_equal
~~~~~~~~~~~~~
Expand All @@ -213,19 +312,19 @@ Returns True if LHS version is equal to RHS version.

.. code-block:: bash

{{version | version_equal("0.10.0")}}
{{ version | version_equal("0.10.0") }}

version_less_than
~~~~~~~~~~~~~~~~~

Returns True if LHS version is lesser than RHS version. Both inputs have to follow semantic version
syntax.

E.g. ``{{“1.6.0” | version_less_than("1.7.0")}}``.
E.g. ``{{ “1.6.0” | version_less_than("1.7.0") }}``.

.. code-block:: bash

{{version | version_less_than("0.9.2")}}
{{ version | version_less_than("0.9.2") }}

version_match
~~~~~~~~~~~~~
Expand All @@ -237,19 +336,19 @@ Supports operators ``>``, ``<``, ``==``, ``<=``, and ``>=``.

.. code-block:: bash

{{version | version_match(">0.10.0")}}
{{ version | version_match(">0.10.0") }}

version_more_than
~~~~~~~~~~~~~~~~~

Returns True if LHS version is greater than RHS version. Both inputs have to follow semantic
version syntax.

E.g. ``{{"1.6.0” | version_more_than("1.7.0")}}``.
E.g. ``{{ "1.6.0” | version_more_than("1.7.0") }}``.

.. code-block:: bash

{{version | version_more_than("0.10.1")}}
{{ version | version_more_than("0.10.1") }}

version_strip_patch
~~~~~~~~~~~~~~~~~~~
Expand All @@ -258,4 +357,4 @@ Drops patch version of supplied version field.

.. code-block:: bash

{{version | version_strip_patch}}
{{ version | version_strip_patch }}
Copy link
Member

@Kami Kami Oct 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I'm also not a fan of non-whitespace approach :P