Prometheus gauge

Hi!

I'm trying to add a gauge with a constant value in order to see what helm configuration is deployed.
It's an idea that's similar to #40 but just related to my current config release.

After seeing the code https://github.com/fluent/fluent-plugin-prometheus/blob/master/lib/fluent/plugin/prometheus.rb#L197-L207

I think it should be possible to do something like this:

# Add constant key in order to be able to use a gauge
<filter *>
  @type record_transformer
  <record>
    helm_release {{ .Release.Revision }}
  </record>
</filter>
<filter app*>
  @type prometheus
  <metric>
    name fluentd_last_sent
    type gauge
    key "helm_release"
    desc Use for debugging purposes
    <labels>
      helm_release_name {{ .Release.Name }}
      helm_release_version {{ .Release.Revision }}
    </labels>
  </metric>
</filter>

If I output the record, I can see that the helm_release key is present, but right now if i query the metrics endpoint. I'm not seeing anything:

Am I missing something on the config?

[SuperQ]

This is a bit like a build info metric. Since helm_release is a version string, you probably want to make that a label. And the value should be 1. I think the missing bit here is that unlike counter in the plugin, leaving out the gauge key isn't supported.

I did end up getting it to work:

# Add constant key in order to be able to use a gauge
<filter *>
  @type record_transformer
  enable_ruby
  <record>
    helm_release_timestamp ${ {{ now |unixEpoch }} }
  </record>
</filter>
<filter *>
  @type prometheus
  <metric>
    name fluentd_config_release
    type gauge
    key helm_release_timestamp
    desc Use for debugging purposes
    <labels>
      helm_release_name {{ .Release.Name }}
      helm_release_version {{ .Release.Revision }}
    </labels>
  </metric>
</filter>

I'm not too happy about having to use the ruby syntax on the record_transformer. This was because the prometheus client was complaining if you try to set a value to be a string (only numbers are supported).

I believe this can be solved if gauges are initialized with some default value (configurable), forcing them to be present on the prometheus client.