Skip to content

Commit 559f515

Browse files
api-clients-generation-pipeline[bot]ci.datadog-api-spec
api-clients-generation-pipeline[bot]
and
ci.datadog-api-spec
authored
Adding custom mapper support to Observability Pipelines OCSF Mapper (#3175)
Co-authored-by: ci.datadog-api-spec <packages@datadoghq.com>
1 parent dfd8a87 commit 559f515

19 files changed

Lines changed: 954 additions & 1 deletion

‎.generator/schemas/v2/openapi.yaml‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41369,6 +41369,7 @@ components:
4136941369
example: CloudTrail Account Change
4137041370
oneOf:
4137141371
- $ref: '#/components/schemas/ObservabilityPipelineOcsfMappingLibrary'
41372+
- $ref: '#/components/schemas/ObservabilityPipelineOcsfMappingCustom'
4137241373
ObservabilityPipelineOcsfMapperProcessorType:
4137341374
default: ocsf_mapper
4137441375
description: The processor type. The value should always be `ocsf_mapper`.
@@ -41378,6 +41379,116 @@ components:
4137841379
type: string
4137941380
x-enum-varnames:
4138041381
- OCSF_MAPPER
41382+
ObservabilityPipelineOcsfMappingCustom:
41383+
description: Custom OCSF mapping configuration for transforming logs.
41384+
properties:
41385+
mapping:
41386+
description: A list of field mapping rules for transforming log fields to
41387+
OCSF schema fields.
41388+
items:
41389+
$ref: '#/components/schemas/ObservabilityPipelineOcsfMappingCustomFieldMapping'
41390+
type: array
41391+
metadata:
41392+
$ref: '#/components/schemas/ObservabilityPipelineOcsfMappingCustomMetadata'
41393+
version:
41394+
description: The version of the custom mapping configuration.
41395+
example: 1
41396+
format: int64
41397+
type: integer
41398+
required:
41399+
- mapping
41400+
- metadata
41401+
- version
41402+
type: object
41403+
ObservabilityPipelineOcsfMappingCustomFieldMapping:
41404+
description: Defines a single field mapping rule for transforming a source field
41405+
to an OCSF destination field.
41406+
properties:
41407+
default:
41408+
description: The default value to use if the source field is missing or
41409+
empty.
41410+
example: ''
41411+
dest:
41412+
description: The destination OCSF field path.
41413+
example: device.type
41414+
type: string
41415+
lookup:
41416+
$ref: '#/components/schemas/ObservabilityPipelineOcsfMappingCustomLookup'
41417+
source:
41418+
description: The source field path from the log event.
41419+
example: host.type
41420+
sources:
41421+
description: Multiple source field paths for combined mapping.
41422+
example:
41423+
- field1
41424+
- field2
41425+
value:
41426+
description: A static value to use for the destination field.
41427+
example: static_value
41428+
required:
41429+
- dest
41430+
type: object
41431+
ObservabilityPipelineOcsfMappingCustomLookup:
41432+
description: Lookup table configuration for mapping source values to destination
41433+
values.
41434+
properties:
41435+
default:
41436+
description: The default value to use if no lookup match is found.
41437+
example: unknown
41438+
table:
41439+
description: A list of lookup table entries for value transformation.
41440+
items:
41441+
$ref: '#/components/schemas/ObservabilityPipelineOcsfMappingCustomLookupTableEntry'
41442+
type: array
41443+
type: object
41444+
ObservabilityPipelineOcsfMappingCustomLookupTableEntry:
41445+
description: A single entry in a lookup table for value transformation.
41446+
properties:
41447+
contains:
41448+
description: The substring to match in the source value.
41449+
example: Desktop
41450+
type: string
41451+
equals:
41452+
description: The exact value to match in the source.
41453+
example: desktop
41454+
equals_source:
41455+
description: The source field to match against.
41456+
example: device_type
41457+
type: string
41458+
matches:
41459+
description: A regex pattern to match in the source value.
41460+
example: ^Desktop.*
41461+
type: string
41462+
not_matches:
41463+
description: A regex pattern that must not match the source value.
41464+
example: ^Mobile.*
41465+
type: string
41466+
value:
41467+
description: The value to use when a match is found.
41468+
example: desktop
41469+
type: object
41470+
ObservabilityPipelineOcsfMappingCustomMetadata:
41471+
description: Metadata for the custom OCSF mapping.
41472+
properties:
41473+
class:
41474+
description: The OCSF event class name.
41475+
example: Device Inventory Info
41476+
type: string
41477+
profiles:
41478+
description: A list of OCSF profiles to apply.
41479+
example:
41480+
- container
41481+
items:
41482+
type: string
41483+
type: array
41484+
version:
41485+
description: The OCSF schema version.
41486+
example: 1.3.0
41487+
type: string
41488+
required:
41489+
- class
41490+
- version
41491+
type: object
4138141492
ObservabilityPipelineOcsfMappingLibrary:
4138241493
description: Predefined library mappings for common log formats.
4138341494
enum:

‎docs/datadog_api_client.v2.model.rst‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17861,6 +17861,41 @@ datadog\_api\_client.v2.model.observability\_pipeline\_ocsf\_mapper\_processor\_
1786117861
:members:
1786217862
:show-inheritance:
1786317863

17864+
datadog\_api\_client.v2.model.observability\_pipeline\_ocsf\_mapping\_custom module
17865+
-----------------------------------------------------------------------------------
17866+
17867+
.. automodule:: datadog_api_client.v2.model.observability_pipeline_ocsf_mapping_custom
17868+
:members:
17869+
:show-inheritance:
17870+
17871+
datadog\_api\_client.v2.model.observability\_pipeline\_ocsf\_mapping\_custom\_field\_mapping module
17872+
---------------------------------------------------------------------------------------------------
17873+
17874+
.. automodule:: datadog_api_client.v2.model.observability_pipeline_ocsf_mapping_custom_field_mapping
17875+
:members:
17876+
:show-inheritance:
17877+
17878+
datadog\_api\_client.v2.model.observability\_pipeline\_ocsf\_mapping\_custom\_lookup module
17879+
-------------------------------------------------------------------------------------------
17880+
17881+
.. automodule:: datadog_api_client.v2.model.observability_pipeline_ocsf_mapping_custom_lookup
17882+
:members:
17883+
:show-inheritance:
17884+
17885+
datadog\_api\_client.v2.model.observability\_pipeline\_ocsf\_mapping\_custom\_lookup\_table\_entry module
17886+
---------------------------------------------------------------------------------------------------------
17887+
17888+
.. automodule:: datadog_api_client.v2.model.observability_pipeline_ocsf_mapping_custom_lookup_table_entry
17889+
:members:
17890+
:show-inheritance:
17891+
17892+
datadog\_api\_client.v2.model.observability\_pipeline\_ocsf\_mapping\_custom\_metadata module
17893+
---------------------------------------------------------------------------------------------
17894+
17895+
.. automodule:: datadog_api_client.v2.model.observability_pipeline_ocsf_mapping_custom_metadata
17896+
:members:
17897+
:show-inheritance:
17898+
1786417899
datadog\_api\_client.v2.model.observability\_pipeline\_ocsf\_mapping\_library module
1786517900
------------------------------------------------------------------------------------
1786617901

‎examples/v2/observability-pipelines/ValidatePipeline_3024756866.py‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
"""
2+
Validate an observability pipeline with OCSF mapper custom mapping returns "OK" response
3+
"""
4+
5+
from datadog_api_client import ApiClient, Configuration
6+
from datadog_api_client.v2.api.observability_pipelines_api import ObservabilityPipelinesApi
7+
from datadog_api_client.v2.model.observability_pipeline_config import ObservabilityPipelineConfig
8+
from datadog_api_client.v2.model.observability_pipeline_config_processor_group import (
9+
ObservabilityPipelineConfigProcessorGroup,
10+
)
11+
from datadog_api_client.v2.model.observability_pipeline_data_attributes import ObservabilityPipelineDataAttributes
12+
from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source import (
13+
ObservabilityPipelineDatadogAgentSource,
14+
)
15+
from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source_type import (
16+
ObservabilityPipelineDatadogAgentSourceType,
17+
)
18+
from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination import (
19+
ObservabilityPipelineDatadogLogsDestination,
20+
)
21+
from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination_type import (
22+
ObservabilityPipelineDatadogLogsDestinationType,
23+
)
24+
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapper_processor import (
25+
ObservabilityPipelineOcsfMapperProcessor,
26+
)
27+
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapper_processor_mapping import (
28+
ObservabilityPipelineOcsfMapperProcessorMapping,
29+
)
30+
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapper_processor_type import (
31+
ObservabilityPipelineOcsfMapperProcessorType,
32+
)
33+
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapping_custom import (
34+
ObservabilityPipelineOcsfMappingCustom,
35+
)
36+
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapping_custom_field_mapping import (
37+
ObservabilityPipelineOcsfMappingCustomFieldMapping,
38+
)
39+
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapping_custom_lookup import (
40+
ObservabilityPipelineOcsfMappingCustomLookup,
41+
)
42+
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapping_custom_lookup_table_entry import (
43+
ObservabilityPipelineOcsfMappingCustomLookupTableEntry,
44+
)
45+
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapping_custom_metadata import (
46+
ObservabilityPipelineOcsfMappingCustomMetadata,
47+
)
48+
from datadog_api_client.v2.model.observability_pipeline_spec import ObservabilityPipelineSpec
49+
from datadog_api_client.v2.model.observability_pipeline_spec_data import ObservabilityPipelineSpecData
50+
51+
body = ObservabilityPipelineSpec(
52+
data=ObservabilityPipelineSpecData(
53+
attributes=ObservabilityPipelineDataAttributes(
54+
config=ObservabilityPipelineConfig(
55+
destinations=[
56+
ObservabilityPipelineDatadogLogsDestination(
57+
id="datadog-logs-destination",
58+
inputs=[
59+
"my-processor-group",
60+
],
61+
type=ObservabilityPipelineDatadogLogsDestinationType.DATADOG_LOGS,
62+
),
63+
],
64+
processor_groups=[
65+
ObservabilityPipelineConfigProcessorGroup(
66+
enabled=True,
67+
id="my-processor-group",
68+
include="service:my-service",
69+
inputs=[
70+
"datadog-agent-source",
71+
],
72+
processors=[
73+
ObservabilityPipelineOcsfMapperProcessor(
74+
enabled=True,
75+
id="ocsf-mapper-processor",
76+
include="service:my-service",
77+
mappings=[
78+
ObservabilityPipelineOcsfMapperProcessorMapping(
79+
include="source:custom",
80+
mapping=ObservabilityPipelineOcsfMappingCustom(
81+
mapping=[
82+
ObservabilityPipelineOcsfMappingCustomFieldMapping(
83+
default="",
84+
dest="time",
85+
source="timestamp",
86+
),
87+
ObservabilityPipelineOcsfMappingCustomFieldMapping(
88+
default="",
89+
dest="severity",
90+
source="level",
91+
),
92+
ObservabilityPipelineOcsfMappingCustomFieldMapping(
93+
default="",
94+
dest="device.type",
95+
lookup=ObservabilityPipelineOcsfMappingCustomLookup(
96+
table=[
97+
ObservabilityPipelineOcsfMappingCustomLookupTableEntry(
98+
contains="Desktop",
99+
value="desktop",
100+
),
101+
],
102+
),
103+
source="host.type",
104+
),
105+
],
106+
metadata=ObservabilityPipelineOcsfMappingCustomMetadata(
107+
_class="Device Inventory Info",
108+
profiles=[
109+
"container",
110+
],
111+
version="1.3.0",
112+
),
113+
version=1,
114+
),
115+
),
116+
],
117+
type=ObservabilityPipelineOcsfMapperProcessorType.OCSF_MAPPER,
118+
),
119+
],
120+
),
121+
],
122+
sources=[
123+
ObservabilityPipelineDatadogAgentSource(
124+
id="datadog-agent-source",
125+
type=ObservabilityPipelineDatadogAgentSourceType.DATADOG_AGENT,
126+
),
127+
],
128+
),
129+
name="OCSF Custom Mapper Pipeline",
130+
),
131+
type="pipelines",
132+
),
133+
)
134+
135+
configuration = Configuration()
136+
with ApiClient(configuration) as api_client:
137+
api_instance = ObservabilityPipelinesApi(api_client)
138+
response = api_instance.validate_pipeline(body=body)
139+
140+
print(response)

‎examples/v2/observability-pipelines/ValidatePipeline_3565101276.py‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
"""
2+
Validate an observability pipeline with OCSF mapper library mapping returns "OK" response
3+
"""
4+
5+
from datadog_api_client import ApiClient, Configuration
6+
from datadog_api_client.v2.api.observability_pipelines_api import ObservabilityPipelinesApi
7+
from datadog_api_client.v2.model.observability_pipeline_config import ObservabilityPipelineConfig
8+
from datadog_api_client.v2.model.observability_pipeline_config_processor_group import (
9+
ObservabilityPipelineConfigProcessorGroup,
10+
)
11+
from datadog_api_client.v2.model.observability_pipeline_data_attributes import ObservabilityPipelineDataAttributes
12+
from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source import (
13+
ObservabilityPipelineDatadogAgentSource,
14+
)
15+
from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source_type import (
16+
ObservabilityPipelineDatadogAgentSourceType,
17+
)
18+
from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination import (
19+
ObservabilityPipelineDatadogLogsDestination,
20+
)
21+
from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination_type import (
22+
ObservabilityPipelineDatadogLogsDestinationType,
23+
)
24+
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapper_processor import (
25+
ObservabilityPipelineOcsfMapperProcessor,
26+
)
27+
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapper_processor_mapping import (
28+
ObservabilityPipelineOcsfMapperProcessorMapping,
29+
)
30+
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapper_processor_type import (
31+
ObservabilityPipelineOcsfMapperProcessorType,
32+
)
33+
from datadog_api_client.v2.model.observability_pipeline_spec import ObservabilityPipelineSpec
34+
from datadog_api_client.v2.model.observability_pipeline_spec_data import ObservabilityPipelineSpecData
35+
36+
body = ObservabilityPipelineSpec(
37+
data=ObservabilityPipelineSpecData(
38+
attributes=ObservabilityPipelineDataAttributes(
39+
config=ObservabilityPipelineConfig(
40+
destinations=[
41+
ObservabilityPipelineDatadogLogsDestination(
42+
id="datadog-logs-destination",
43+
inputs=[
44+
"my-processor-group",
45+
],
46+
type=ObservabilityPipelineDatadogLogsDestinationType.DATADOG_LOGS,
47+
),
48+
],
49+
processor_groups=[
50+
ObservabilityPipelineConfigProcessorGroup(
51+
enabled=True,
52+
id="my-processor-group",
53+
include="service:my-service",
54+
inputs=[
55+
"datadog-agent-source",
56+
],
57+
processors=[
58+
ObservabilityPipelineOcsfMapperProcessor(
59+
enabled=True,
60+
id="ocsf-mapper-processor",
61+
include="service:my-service",
62+
type=ObservabilityPipelineOcsfMapperProcessorType.OCSF_MAPPER,
63+
mappings=[
64+
ObservabilityPipelineOcsfMapperProcessorMapping(
65+
include="source:cloudtrail",
66+
mapping="CloudTrail Account Change",
67+
),
68+
],
69+
),
70+
],
71+
),
72+
],
73+
sources=[
74+
ObservabilityPipelineDatadogAgentSource(
75+
id="datadog-agent-source",
76+
type=ObservabilityPipelineDatadogAgentSourceType.DATADOG_AGENT,
77+
),
78+
],
79+
),
80+
name="OCSF Mapper Pipeline",
81+
),
82+
type="pipelines",
83+
),
84+
)
85+
86+
configuration = Configuration()
87+
with ApiClient(configuration) as api_client:
88+
api_instance = ObservabilityPipelinesApi(api_client)
89+
response = api_instance.validate_pipeline(body=body)
90+
91+
print(response)

0 commit comments

Comments
 (0)