diff --git a/data-explorer/create-table-wizard.md b/data-explorer/create-table-wizard.md index f4b3c2238f..72025f1ca3 100644 --- a/data-explorer/create-table-wizard.md +++ b/data-explorer/create-table-wizard.md @@ -3,13 +3,13 @@ title: Create a Table in Azure Data Explorer description: Learn how to easily create a table and manually define the schema in Azure Data Explorer with the table creation wizard. ms.reviewer: aksdi ms.topic: how-to -ms.date: 02/02/2026 +ms.date: 02/12/2026 # Customer intent: As a data engineer, I want to create an empty table in Azure Data Explorer so that I can ingest data and query it. --- # Create a table in Azure Data Explorer -Creating a table is an important step in the process of [data ingestion](ingest-data-overview.md) and [query](/azure/data-explorer/kusto/query/tutorials/learn-common-operators) in Azure Data Explorer. The following article shows how to create a table and schema mapping quickly and easily by using the Azure Data Explorer web UI. +Creating a table is an important step in the process of [data ingestion](ingest-data-overview.md) and [querying](/azure/data-explorer/kusto/query/tutorials/learn-common-operators) in Azure Data Explorer. The following article shows how to create a table and schema mapping quickly and easily by using the Azure Data Explorer web UI. > [!NOTE] > To create a new table based on existing data, see [Get data from file](get-data-file.md) or [Get data from Azure storage](get-data-storage.md). @@ -46,7 +46,7 @@ The **Create table** window opens with the **Destination** tab selected. ## Schema tab -1. Select **Add new column**. The **Edit columns** panel opens. +1. Select **Add new column** to open the **Edit columns** panel. 1. For each column, enter **Column name** and **Data type**. Create more columns by selecting **Add column**. :::image type="content" source="media/create-table-wizard/edit-columns.png" alt-text="Screenshot of Edit columns pane, in which you input the column name and data type in Azure Data Explorer."::: diff --git a/data-explorer/external-table.md b/data-explorer/external-table.md index c09a3e2486..e166bd3b6d 100644 --- a/data-explorer/external-table.md +++ b/data-explorer/external-table.md @@ -1,9 +1,9 @@ --- -title: Create an external table using the Azure Data Explorer web UI wizard in Azure Data Explorer +title: Create an External Table Using the Azure Data Explorer Web UI Wizard in Azure Data Explorer description: Use the wizard experience to create an external table. ms.reviewer: ohbitton ms.topic: how-to -ms.date: 08/25/2022 +ms.date: 02/12/2026 --- # Create an external table using the Azure Data Explorer web UI wizard @@ -28,17 +28,17 @@ The **Create external table** window opens with the **Destination** tab selected ### Destination tab -1. The **Cluster** and **Database** fields are prepopulated. You may select a different destination from the dropdown menu. +1. The **Cluster** and **Database** fields are prepopulated. You can select a different destination from the dropdown menu. 1. In **Table name**, enter a name for your table. > [!TIP] - > Table names can be up to 1024 characters including alphanumeric, hyphens, and underscores. Special characters aren't supported. -1. Select **Next: Source** + > Table names can be up to 1,024 characters including alphanumeric, hyphens, and underscores. Special characters aren't supported. +1. Select **Next: Source**. :::image type="content" source="media/external-table/destination-tab.png" alt-text="Screen capture of the Destination tab with Cluster, Database, and Table name fields."::: ### Source tab -In **Link to containers**, there are two ways to add a container: [Add a container with the **Select container** button](#add-a-container-with-the-select-container-button) and [Add a container with the **Add URL or Add container** button](#add-a-container-with-the-add-url-or-add-container-button). +In **Link to containers**, you can add a container in two ways: [Add a container with the **Select container** button](#add-a-container-with-the-select-container-button) and [Add a container with the **Add URL or Add container** button](#add-a-container-with-the-add-url-or-add-container-button). You can add up to 10 source containers. @@ -48,11 +48,11 @@ You can add up to 10 source containers. :::image type="content" source="media/external-table/select-container.png" alt-text="Screenshot of select container button in source tab."::: -1. Choose the relevant subscription and storage account associated with your container. +1. Choose the subscription and storage account that your container uses. :::image type="content" source="media/select-container-window.png" alt-text="Screenshot of select container window."::: -1. Select the **Add** button. When verification has completed, a green check will appear to the right of the container link. +1. Select **Add**. When verification finishes, a green check appears to the right of the container link. :::image type="content" source="media/external-table/container-verified.png" alt-text="Screenshot of verified container link."::: @@ -62,37 +62,37 @@ You can add up to 10 source containers. :::image type="content" source="media/external-table/add-url-button.png" alt-text="Screenshot of add URL button."::: -1. Enter a [storage connection strings](/kusto/api/connection-strings/storage-connection-strings?view=azure-data-explorer&preserve-view=true) to your source container with read and list permissions. When verification has completed, a green check will appear to the right of the container link. +1. Enter a [storage connection strings](/kusto/api/connection-strings/storage-connection-strings?view=azure-data-explorer&preserve-view=true) to your source container with read and list permissions. When verification finishes, a green check appears to the right of the container link. :::image type="content" source="media/external-table/add-sas-url.png" alt-text="Screenshot of adding SAS URL."::: #### File filters -Use **File filters** to filter the files that the table should include. Files can be filtered according to folder path, file begins with, or file extension. +Use **File filters** to filter the files that the table should include. You can filter files by folder path, file begins with, or file extension. :::image type="content" source="media/external-table/file-filters.png" alt-text="Screenshot of selecting schema-defining file."::: #### Schema-defining file -The first source container will display files below **File filters**. +The first source container displays files below **File filters**. :::image type="content" source="media/external-table/schema-defining-file.png" alt-text="Screen shot of create external table source tab in Azure Data Explorer."::: -1. Choose the schema-defining file by selecting the circle to the left of the file. This file will be used to generate the table schema. +1. Choose the schema-defining file by selecting the circle to the left of the file. This file is used to generate the table schema. 1. Select **Next: schema**. The **Schema** tab opens. ### Schema tab -In the right-hand side of the tab, you can preview your data. On the left-hand side, you can add [partitions](/kusto/management/partitioning-policy?view=azure-data-explorer&preserve-view=true) to your table definitions to access the source data more quickly and achieve better performance. +On the right side of the tab, you can preview your data. On the left side, you can add [partitions](/kusto/management/partitioning-policy?view=azure-data-explorer&preserve-view=true) to your table definitions to access the source data more quickly and achieve better performance. > [!NOTE] -> Mappings are not part of the definition of an external table, and are not supported in this wizard. Mappings can be [configured later](/kusto/management/external-table-mapping-create?view=azure-data-explorer&preserve-view=true) if necessary. Some functionalities, such as deleting the last column in CSV files or changing column names in JSON files, require mappings in order to work correctly. +> Mappings aren't part of the definition of an external table, and the wizard doesn't support them. You can [configure mappings later](/kusto/management/external-table-mapping-create?view=azure-data-explorer&preserve-view=true) if necessary. Some functionalities, such as deleting the last column in CSV files or changing column names in JSON files, require mappings in order to work correctly. 1. Select **Add partition**. :::image type="content" source="media/external-table/view-file.png" alt-text="Screen shot of view file for external table in Azure Data Explorer."::: -1. The partition window opens. A partition is defined over a subpath of the file, which can be altered using the **Path prefix** field. For each partition you wish to apply, fill out the fields as follows: +1. The partition window opens. A partition is defined over a subpath of the file, which you can change by using the **Path prefix** field. For each partition you want to apply, fill out the fields as follows: Field | Description | Required/Optional ---|---|--- @@ -102,21 +102,21 @@ In the right-hand side of the tab, you can preview your data. On the left-hand s Function | The function applied to the data column used for partitioning. | Optional Function argument | Argument to be used in the partition function. | Required if function is used. Path prefix | The subpath of the file on which the partitioning is defined. This prefix changes the URL of the external table, as seen in the **Uri preview** box, and should match the schema-defining file URI. | Optional - Datetime pattern | Format of date that will be used to construct the table URI path. | Optional + Datetime pattern | Format of date that you use to construct the table URI path. | Optional :::image type="content" source="media/external-table/add-partitions.png" alt-text="Screen shot add partitions to external table in Azure Data Explorer." lightbox="media/external-table/add-partitions.png"::: - For example, the partition name *CustomerName* suggests that the value to partition by is in the customer name part of the URL. The above example declared two partitions: one partition over the customer name and one partition over the date embedded in the URL. + For example, the partition name *CustomerName* suggests that the value to partition by is in the customer name part of the URL. The preceding example declares two partitions: one partition over the customer name and one partition over the date embedded in the URL. > [!NOTE] - > Virtual columns appear as part of the schema as the columns data extracted from the file path, and this data can be used later in queries. + > Virtual columns appear as part of the schema as the columns data extracted from the file path, and you can use this data later in queries. 1. Select **Add partition** to add another partition. -1. Select **Save**. The partitions you added now appear in the list of **Partitions** in the left pane. Partitioned columns can't be changed in preview. +1. Select **Save**. The partitions you added now appear in the list of **Partitions** in the left pane. You can't change partitioned columns in preview. :::image type="content" source="media/external-table/schema.png" alt-text="Screenshot of schema external table Azure Data Explorer."::: -1. Select **Next: Create table**. When the table is created, an **External table successfully created** window opens. +1. Select **Next: Create table**. When you create the table, an **External table successfully created** window opens. 1. To view the command used to create the table, select **View command**. :::image type="content" source="media/external-table/successfully-created.png" alt-text="Screenshot of successful creation of external table in Azure Data Explorer."::: @@ -124,7 +124,7 @@ In the right-hand side of the tab, you can preview your data. On the left-hand s ## Query the external table -The resulting table includes data from all the files that fit the criteria defined above. You can query this table using the `external_table()` function. For more information on how to query external tables, see [Querying an external table](data-lake-query-data.md#querying-an-external-table). +The resulting table includes data from all the files that fit the criteria you defined. You can query this table by using the `external_table()` function. For more information on how to query external tables, see [Querying an external table](data-lake-query-data.md#querying-an-external-table). :::image type="content" source="media/external-table/view-table.png" alt-text="Screen shot of table output from querying external table in Azure Data Explorer."::: diff --git a/data-explorer/get-data-file.md b/data-explorer/get-data-file.md index f0080f612c..ab2180bacd 100644 --- a/data-explorer/get-data-file.md +++ b/data-explorer/get-data-file.md @@ -3,9 +3,10 @@ title: Get Data From a File description: Learn how to get data from a local file in Azure Data Explorer. ms.reviewer: sharmaanshul ms.topic: how-to -ms.date: 02/08/2026 +ms.date: 02/12/2026 ms.custom: sfi-image-nochange --- + # Get data from file Data ingestion is the process of loading data from one or more sources into a table in Azure Data Explorer. Once ingested, the data is available for query. In this article, you learn how to get data from a local file into either a new or existing table. diff --git a/data-explorer/ingest-data-cosmos-db-connection.md b/data-explorer/ingest-data-cosmos-db-connection.md index 572e1cd440..89da3e73f0 100644 --- a/data-explorer/ingest-data-cosmos-db-connection.md +++ b/data-explorer/ingest-data-cosmos-db-connection.md @@ -1,24 +1,24 @@ --- -title: Ingest data from Azure Cosmos DB into Azure Data Explorer +title: Ingest Data From Azure Cosmos DB Into Azure Data Explorer description: Learn how to ingest (load) data into Azure Data Explorer from Cosmos DB. ms.reviewer: vplauzon ms.topic: how-to -ms.date: 01/07/2025 +ms.date: 02/12/2026 ms.custom: sfi-image-nochange --- # Ingest data from Azure Cosmos DB into Azure Data Explorer -Azure Data Explorer supports [data ingestion](ingest-data-overview.md) from [Azure Cosmos DB for NoSql](/azure/cosmos-db/nosql/) using a [change feed](/azure/cosmos-db/change-feed). The Cosmos DB change feed data connection is an ingestion pipeline that listens to your Cosmos DB change feed and ingests the data into your Data Explorer table. The change feed listens for new and updated documents but doesn't log deletes. For general information about data ingestion in Azure Data Explorer, see [Azure Data Explorer data ingestion overview](ingest-data-overview.md). +Azure Data Explorer supports [data ingestion](ingest-data-overview.md) from [Azure Cosmos DB for NoSql](/azure/cosmos-db/nosql/) by using a [change feed](/azure/cosmos-db/change-feed). The Cosmos DB change feed data connection is an ingestion pipeline that listens to your Cosmos DB change feed and ingests the data into your Data Explorer table. The change feed listens for new and updated documents but doesn't log deletes. For general information about data ingestion in Azure Data Explorer, see [Azure Data Explorer data ingestion overview](ingest-data-overview.md). -Each data connection listens to a specific Cosmos DB container and ingests data into a specified table (more than one connection can ingest in a single table). The ingestion method supports streaming ingestion (when enabled) and queued ingestion. +Each data connection listens to a specific Cosmos DB container and ingests data into a specified table (more than one connection can ingest into a single table). The ingestion method supports streaming ingestion (when enabled) and queued ingestion. The two main scenarios for using the Cosmos DB change feed data connection are: * Replicating a Cosmos DB container for analytical purposes. For more information, see [Get latest versions of Azure Cosmos DB documents](ingest-data-cosmos-db-queries.md). * Analyzing the document changes in a Cosmos DB container. For more information, see [Considerations](#considerations). -In this article, you'll learn how to set up a Cosmos DB change feed data connection to ingest data into Azure Data Explorer with System Managed Identity. Review the [considerations](#considerations) before you start. +In this article, you learn how to set up a Cosmos DB change feed data connection to ingest data into Azure Data Explorer with System Managed Identity. Review the [considerations](#considerations) before you start. Use the following steps to set up a connector: @@ -33,11 +33,11 @@ Step 3: [Test the data connection](#step-3-test-the-data-connection) - An Azure subscription. Create a [free Azure account](https://azure.microsoft.com/pricing/purchase-options/azure-account?cid=msft_learn). - An Azure Data Explorer cluster and database. [Create a cluster and database](create-cluster-and-database.md). - A container from a [Cosmos DB account for NoSQL](/azure/cosmos-db/nosql/). -- If your Cosmos DB account blocks network access, for example by using a [private endpoint](/azure/cosmos-db/how-to-configure-private-endpoints), you must [create a managed private endpoint](security-network-managed-private-endpoint-create.md) to the Cosmos DB account. This is required for your cluster to invoke the change feed API. +- If your Cosmos DB account blocks network access, for example by using a [private endpoint](/azure/cosmos-db/how-to-configure-private-endpoints), you must [create a managed private endpoint](security-network-managed-private-endpoint-create.md) to the Cosmos DB account. This requirement enables your cluster to invoke the change feed API. ## Step 1: Choose an Azure Data Explorer table and configure its table mapping -Before you create a data connection, create a table where you'll store the ingested data and apply a mapping that matches schema in the source Cosmos DB container. If your scenario requires more than a simple mapping of fields, you can use [update policies to transform and map data](#transform-and-map-data-with-update-policies) ingested from your change feed. +Before you create a data connection, create a table where you store the ingested data and apply a mapping that matches the schema in the source Cosmos DB container. If your scenario requires more than a simple mapping of fields, you can use [update policies to transform and map data](#transform-and-map-data-with-update-policies) ingested from your change feed. The following shows a sample schema of an item in the Cosmos DB container: @@ -75,7 +75,7 @@ Use the following steps to create a table and apply a table mapping: | **_ts** | _timestamp | Uses `DateTimeFromUnixSeconds` to [transform](/kusto/management/mappings?view=azure-data-explorer&preserve-view=true) **\_ts** ([UNIX seconds](https://wikipedia.org/wiki/Unix_time)) to **_timestamp** (`datetime`)) | > [!NOTE] - > We recommend using the following timestamp columns: + > Use the following timestamp columns: > > - **_ts**: Use this column to reconcile data with Cosmos DB. > - **_timestamp**: Use this column to run efficient time filters in your Kusto queries. For more information, see [Query best practice](/kusto/query/best-practices?view=azure-data-explorer&preserve-view=true). @@ -94,19 +94,19 @@ Use the following steps to create a table and apply a table mapping: ### Transform and map data with update policies -If your scenario requires more than a simple mapping of fields, you can use update policies to transform and map data ingested from your change feed. +If your scenario requires more than a simple mapping of fields, use update policies to transform and map data ingested from your change feed. -[Update policies](/kusto/management/update-policy?view=azure-data-explorer&preserve-view=true) are a way to transform data as it's ingested into your table. They're written in Kusto Query Language and are run on the ingestion pipeline. They can be used to transform data from a Cosmos DB change feed ingestion, such as in the following scenarios: +[Update policies](/kusto/management/update-policy?view=azure-data-explorer&preserve-view=true) transform data during ingestion into your table. Write them in Kusto Query Language and run them on the ingestion pipeline. Use them to transform data from a Cosmos DB change feed ingestion, such as in the following scenarios: -- Your documents contain arrays that would be easier to query if they're transformed in multiple rows using the [`mv-expand`](/kusto/management/alter-table-update-policy-command?view=azure-data-explorer&preserve-view=true) operator. -- You want to filter out documents. For example, you can filter out documents by type using the [`where`](/kusto/query/where-operator?view=azure-data-explorer&preserve-view=true) operator. +- Your documents contain arrays that are easier to query if they're transformed into multiple rows by using the [`mv-expand`](/kusto/management/alter-table-update-policy-command?view=azure-data-explorer&preserve-view=true) operator. +- You want to filter out documents. For example, you can filter out documents by type by using the [`where`](/kusto/query/where-operator?view=azure-data-explorer&preserve-view=true) operator. - You have complex logic that can't be represented in a table mapping. For information on how to create and manage update policies, see [Update policy overview](/kusto/management/alter-table-update-policy-command?view=azure-data-explorer&preserve-view=true). ## Step 2: Create a Cosmos DB data connection -You can use the following methods to create the data connector: +Use the following methods to create the data connector: ### [Azure portal](#tab/portal) @@ -116,7 +116,7 @@ You can use the following methods to create the data connector: :::image type="content" source="media/ingest-data-cosmos-db/create-data-connection.png" alt-text="Screenshot of the Getting started tab, showing the Create Cosmos DB data connection option."::: -1. In the Cosmos DB **Create data connection** pane, fill out the form with the information in the table: +1. In the Cosmos DB **Create data connection** pane, fill out the form with the information in the following table: :::image type="content" source="media/ingest-data-cosmos-db/fill-fields.png" alt-text="Screenshot of the data connection pane, showing the form fields with values."::: @@ -131,13 +131,13 @@ You can use the following methods to create the data connector: | **Table name** | Specify the Azure Data Explorer [table name](#step-1-choose-an-azure-data-explorer-table-and-configure-its-table-mapping) to which you want to ingest data. | | **Mapping name** | Optionally, specify the [mapping name](#step-1-choose-an-azure-data-explorer-table-and-configure-its-table-mapping) to use for the data connection. | -1. Optionally, under the **Advanced settings** section, do the following: - 1. Specify the **Event retrieval start date**. This is the time from which the connector will start ingesting data. If you don't specify a time, the connector will start ingesting data from the time you create the data connection. The recommended date format is the ISO 8601 UTC standard, specified as follows: `yyyy-MM-ddTHH:mm:ss.fffffffZ`. - 1. Select **User-assigned** and then select the identity. By Default, the **System-assigned** managed identity is used by the connection. If necessary, you can use a **User-assigned** identity. +1. Optionally, under the **Advanced settings** section, enter the following information: + 1. Specify the **Event retrieval start date**. This value is the time from which the connector starts ingesting data. If you don't specify a time, the connector starts ingesting data from the time you create the data connection. The recommended date format is the ISO 8601 UTC standard, specified as follows: `yyyy-MM-ddTHH:mm:ss.fffffffZ`. + 1. Select **User-assigned** and then select the identity. By default, the connection uses the **System-assigned** managed identity. If necessary, you can use a **User-assigned** identity. :::image type="content" source="media/ingest-data-cosmos-db/advanced-settings.png" alt-text="Screenshot of the data connection pane, showing the Advance settings."::: -1. Select **Create** to crate the data connection. +1. Select **Create** to create the data connection. ### [ARM template](#tab/arm) @@ -149,18 +149,18 @@ To configure your Cosmos DB connection: 1. In the Azure Data Explorer web UI, select **Query** from the left navigation menu, and then select the cluster or database for the data connection. -1. Grant the data connection permission to access your Cosmos DB account. Providing the data connection access to your Cosmos DB allows it to access and retrieve data from your database. You'll need your cluster's principal ID, which you can find in the Azure portal. For more information, see [Configure managed identities for your cluster](configure-managed-identities-cluster.md#add-a-system-assigned-identity). +1. Grant the data connection permission to access your Cosmos DB account. By providing the data connection access to your Cosmos DB, it can access and retrieve data from your database. You need your cluster's principal ID, which you can find in the Azure portal. For more information, see [Configure managed identities for your cluster](configure-managed-identities-cluster.md#add-a-system-assigned-identity). > [!NOTE] > > - The following steps assign these roles to the principal ID: > - [Cosmos DB Built-in Data Reader](/azure/cosmos-db/how-to-setup-rbac#built-in-role-definitions) - > - You can't assign the **Cosmos DB Built-in Data Reader** role using the Azure portal *Role Assignment* feature. + > - You can't assign the **Cosmos DB Built-in Data Reader** role by using the Azure portal *Role Assignment* feature. > - [Cosmos DB Account Reader Role](/azure/role-based-access-control/built-in-roles) Use one of the following options to grant access to your Cosmos DB account: - - **Grant access using the Azure CLI**: Run the CLI command, using information in the following table to replace placeholders with appropriate values: + - **Grant access by using the Azure CLI**: Run the CLI command, using information in the following table to replace placeholders with appropriate values: ```azurecli az cosmosdb sql role assignment create --account-name --resource-group --role-definition-id 00000000-0000-0000-0000-000000000001 --principal-id --scope "/" @@ -173,9 +173,9 @@ To configure your Cosmos DB connection: | **\** | The name of your Cosmos DB account. | | **\** | The name of the resource group that contains your Cosmos DB account. | | **\** | The Azure resource ID (starting with `subscriptions/`) of your Cosmos DB account. | - | **\** | The principal ID of the managed identity assigned to your cluster. You can find your cluster's principle ID in the Azure portal. For more information, see [Configure managed identities for your cluster](configure-managed-identities-cluster.md#add-a-system-assigned-identity). | + | **\** | The principal ID of the managed identity assigned to your cluster. You can find your cluster's principal ID in the Azure portal. For more information, see [Configure managed identities for your cluster](configure-managed-identities-cluster.md#add-a-system-assigned-identity). | - - **Grant access using an ARM Template**: Deploy the following template in the Cosmos DB account resource group: + - **Grant access by using an ARM Template**: Deploy the following template in the Cosmos DB account resource group: ```json { @@ -326,7 +326,7 @@ To configure your Cosmos DB connection: > [!NOTE] > -> Azure Data Explorer has an aggregation (batching) policy for queued data ingestion designed to optimize the ingestion process. The default batching policy is configured to seal a batch once one of the following conditions is true for the batch: a maximum delay time of 5 minutes, total size of one GB, or 1000 blobs. Therefore, you may experience a latency. For more information, see [batching policy](/kusto/management/batching-policy?view=azure-data-explorer&preserve-view=true). To reduce latency, configure your table to support streaming. See [streaming policy](/kusto/management/streaming-ingestion-policy?view=azure-data-explorer&preserve-view=true). +> Azure Data Explorer uses an aggregation (batching) policy for queued data ingestion that optimizes the ingestion process. The default batching policy seals a batch when one of the following conditions is true for the batch: a maximum delay time of five minutes, total size of 1 GB, or 1,000 blobs. Therefore, you might experience latency. For more information, see [batching policy](/kusto/management/batching-policy?view=azure-data-explorer&preserve-view=true). To reduce latency, configure your table to support streaming. See [streaming policy](/kusto/management/streaming-ingestion-policy?view=azure-data-explorer&preserve-view=true). ## Considerations @@ -334,7 +334,7 @@ The following considerations apply to the Cosmos DB change feed: - The change feed doesn't expose *deletion* events. - The Cosmos DB change feed only includes new and updated documents. If you need to know about deleted documents, you can configure your feed use a [soft marker](/azure/cosmos-db/change-feed#change-feed-and-different-operations) to mark a Cosmos DB document as deleted. A property is added to update events that indicate whether a document has been deleted. You can then use the `where` operator in your queries to filter them out. + The Cosmos DB change feed only includes new and updated documents. If you need to know about deleted documents, you can configure your feed to use a [soft marker](/azure/cosmos-db/change-feed#change-feed-and-different-operations) to mark a Cosmos DB document as deleted. A property is added to update events that indicate whether a document is deleted. You can then use the `where` operator in your queries to filter them out. For example, if you map the deleted property to a table column called **IsDeleted**, you can filter out deleted documents with the following query: @@ -359,7 +359,7 @@ The following considerations apply to the Cosmos DB change feed: | A | Carmine | Update | 50 | | B | NeonBlue | Update | 70 | - The change feed API is polled by the data connector at regular intervals, typically every few seconds. Each poll contains changes that occurred in the container between calls, *but only the latest version of change per document*. + The data connector polls the change feed API at regular intervals, typically every few seconds. Each poll contains changes that occurred in the container between calls, *but only the latest version of change per document*. To illustrate the issue, consider a sequence of API calls with timestamps *15*, *35*, *55*, and *75* as shown in the following table: @@ -372,9 +372,9 @@ The following considerations apply to the Cosmos DB change feed: | 55 | A | Carmine | 60 | | 75 | B | NeonBlue | 70 | - Comparing the API results to the list of changes made in the Cosmos DB document, you'll notice that they don't match. The update event to document *A*, highlighted in the change table at timestamp 40, doesn't appear in the results of the API call. + Comparing the API results to the list of changes made in the Cosmos DB document, you notice that they don't match. The update event to document *A*, highlighted in the change table at timestamp 40, doesn't appear in the results of the API call. - To understand why the event doesn't appear, we'll examine the changes to document *A* between the API calls at timestamps 35 and 55. Between these two calls, document *A* changed twice, as follows: + To understand why the event doesn't appear, examine the changes to document *A* between the API calls at timestamps 35 and 55. Between these two calls, document *A* changed twice, as follows: | Document ID | Property **foo** | Event | Document timestamp (**_ts**) | |---|---|---|---| @@ -383,22 +383,22 @@ The following considerations apply to the Cosmos DB change feed: When the API call at timestamp 55 is made, the change feed API returns the latest version of the document. In this case, the latest version of document *A* is the update at timestamp 50, which is the update to property **foo** from *Pink* to *Carmine*. - Because of this scenario, the data connector may miss some intermediate document changes. For example, some events may be missed if the data connection service is down for a few minutes, or if the frequency of document changes is higher than the API polling frequency. However, the latest state of each document is captured. + Because of this scenario, the data connector might miss some intermediate document changes. For example, the data connector might miss some events if the data connection service is down for a few minutes, or if the frequency of document changes is higher than the API polling frequency. However, the latest state of each document is captured. -- Deleting and recreating a Cosmos DB container isn't supported +- Deleting and recreating a Cosmos DB container isn't supported. - Azure Data Explorer keeps track of the change feed by checkpointing the "position" it is at in the feed. This is done using continuation token on each physical partitions of the container. When a container is deleted/recreated, the continuation token is invalid and isn't reset. In this case, you must delete and recreate the data connection. + Azure Data Explorer keeps track of the change feed by checkpointing the "position" it is at in the feed. This process uses a continuation token on each physical partition of the container. When you delete and recreate a container, the continuation token becomes invalid and isn't reset. In this case, you must delete and recreate the data connection. ## Estimate cost -How much does using the Cosmos DB data connection impact your Cosmos DB container's [Request Units (RUs)](/azure/cosmos-db/request-units) usage? +How much does using the Cosmos DB data connection affect your Cosmos DB container's [Request Units (RUs)](/azure/cosmos-db/request-units) usage? The connector invokes the Cosmos DB Change Feed API on each physical partition of your container, to up to once a second. The following costs are associated with these invocations: | Cost | Description | | -- | -- | -| Fixed costs | Fixed costs are about 2 RUs per physical partition every second. | -| Variable costs | Variable costs are about 2% of the RUs used to write documents, though this may vary depending on your scenario. For example, if you write 100 documents to a Cosmos DB container, the cost of writing those documents is 1,000 RUs. The corresponding cost for using the connector to read those document is about 2% the cost to write them, approximately 20 RUs. | +| Fixed costs | Fixed costs are about two RUs per physical partition every second. | +| Variable costs | Variable costs are about 2% of the RUs used to write documents, though this value might vary depending on your scenario. For example, if you write 100 documents to a Cosmos DB container, the cost of writing those documents is 1,000 RUs. The corresponding cost for using the connector to read the documents is about 2% the cost to write them, approximately 20 RUs. | ## Related content diff --git a/data-explorer/ingest-data-kafka.md b/data-explorer/ingest-data-kafka.md index a98c933c98..7b39b20411 100644 --- a/data-explorer/ingest-data-kafka.md +++ b/data-explorer/ingest-data-kafka.md @@ -1,12 +1,13 @@ --- -title: 'Ingest data from Kafka into Azure Data Explorer' +title: Ingest data From Kafka Into Azure Data Explorer description: In this article, you learn how to ingest (load) data into Azure Data Explorer from Kafka. ms.reviewer: ankhanol ms.topic: how-to -ms.date: 08/21/2024 +ms.date: 02/12/2026 #Customer intent: As an integration developer, I want to build integration pipelines from Kafka into Azure Data Explorer, so I can make data available for near real time analytics. --- + # Ingest data from Apache Kafka into Azure Data Explorer [!INCLUDE [ingest-data-kafka](includes/cross-repo/ingest-data-kafka.md)] @@ -53,16 +54,16 @@ This file contains the Kusto sink properties file where you update specific conf ### Managed identity -By default, the Kafka connector uses the application method for authentication during ingestion. To authenticate using managed identity: +By default, the Kafka connector uses the application method for authentication during ingestion. To authenticate by using managed identity: 1. Assign your cluster a managed identity and grant your storage account read permissions. For more information, see [Ingest data using managed identity authentication](ingest-data-managed-identity.md). 1. In your **adx-sink-config.json** file, set `aad.auth.strategy` to `managed_identity` and ensure that `aad.auth.appid` is set to the managed identity client (application) ID. -1. Use a [private instance metadata service token](/azure/active-directory/managed-identities-azure-resources/how-to-use-vm-token) instead of the [Microsoft Entra service principal](#create-a-microsoft-entra-service-principal). +1. Use a [private instance metadata service token](/azure/active-directory/managed-identities-azure-resources/how-to-use-vm-token) instead of the Microsoft Entra service principal. > [!NOTE] -> When using a managed identity, `appId` and `tenant` are deduced from the context of the call site and `password` isn't needed. +> When using managed identity, the connector automatically retrieves the `appId` and `tenant` from the call context, so you don't need to provide a password. [!INCLUDE [ingest-data-kafka-4](includes/cross-repo/ingest-data-kafka-4.md)] diff --git a/data-explorer/ingest-data-managed-identity.md b/data-explorer/ingest-data-managed-identity.md index 43e3beba76..0cb1db37ac 100644 --- a/data-explorer/ingest-data-managed-identity.md +++ b/data-explorer/ingest-data-managed-identity.md @@ -1,27 +1,27 @@ --- -title: Ingest data using managed identity authentication +title: Ingest Data Using Managed Identity Authentication description: Learn how to queue Azure Storage blobs for ingestion using managed identity authentication. ms.reviewer: miwalia ms.topic: how-to -ms.date: 05/09/2023 +ms.date: 02/12/2026 --- # Queue blobs for ingestion using managed identity authentication -When queuing blobs for ingestion from your own storage accounts, you can use managed identities as an alternative to [shared access signature (SAS)](/azure/storage/common/storage-sas-overview) tokens and [Shared Keys](/rest/api/storageservices/authorize-with-shared-key) authentication methods. Managed identities are a more secure way to ingest data as they don't require you to share your customer SAS tokens or shared keys with the service. Instead, a managed identity is assigned to your cluster and is granted read permissions for the storage account used to ingest data. You can revoke these permissions at any time. +When you queue blobs for ingestion from your own storage accounts, use managed identities as an alternative to [shared access signature (SAS)](/azure/storage/common/storage-sas-overview) tokens and [Shared Keys](/rest/api/storageservices/authorize-with-shared-key) authentication methods. Managed identities provide a more secure way to ingest data because you don't need to share your customer SAS tokens or shared keys with the service. Instead, assign a managed identity to your cluster and grant it read permissions for the storage account used to ingest data. You can revoke these permissions at any time. > [!NOTE] > -> * This authentication method only applies to Azure blobs and Azure Data Lake files residing in customer owned storage accounts. It does not apply to local files uploaded using the Kusto SDK. -> * Only queued ingestion is supported. Inline ingestion in Kusto Query Language and direct ingestion using SDK APIs are not supported. +> * This authentication method only applies to Azure blobs and Azure Data Lake files residing in customer owned storage accounts. It doesn't apply to local files uploaded by using the Kusto SDK. +> * Only queued ingestion is supported. Inline ingestion in Kusto Query Language and direct ingestion by using SDK APIs aren't supported. ## Assign a managed identity to your cluster Follow [Managed identities overview](managed-identities-overview.md) to add a System or User Assigned managed identity to your cluster. -If your cluster already has the desired managed identity assigned to it, copy its object ID using the following steps: +If your cluster already has the desired managed identity assigned to it, copy its object ID by using the following steps: -1. Sign in to the [Azure portal](https://portal.azure.com/) using an account associated with the Azure subscription that contains your cluster. +1. Sign in to the [Azure portal](https://portal.azure.com/) by using an account associated with the Azure subscription that contains your cluster. 1. Navigate to your cluster and select **Identity**. 1. Select the appropriate identity type, system or user assigned, and then copy the object ID of the required identity. @@ -30,26 +30,26 @@ If your cluster already has the desired managed identity assigned to it, copy it ## Grant permissions to the managed identity -1. In the Azure portal, navigate to the storage account that contains the data you want to ingest. +1. In the Azure portal, go to the storage account that contains the data you want to ingest. 1. Select **Access Control** and then select **+ Add** > **Add Role Assignment**. 1. Grant the managed identity **Storage Blob Data Reader**, or **Storage Blob Data Contributor** if you intend to use the **DeleteSourceOnSuccess** source option, permissions to the storage account. > [!NOTE] -> Granting **Owner** or **Contributor** permissions is not sufficient and will result in the ingestion failing. +> Granting **Owner** or **Contributor** permissions isn't sufficient and causes the ingestion to fail. :::image type="content" source="media/ingest-data-managed-identity/managed-identity-permissions-on-system-assigned.png" alt-text="Screenshot of the add role assignment page, showing the system assigned role for ingestion using managed identities"::: > [!IMPORTANT] -> In the event of network problems, Azure Storage may return a `Download Forbidden` error. -> This error may occur if you use a private link to access your storage account. -> In such cases, if the permissions are correct, verify the connectivity to your storage account. +> If network problems occur, Azure Storage might return a `Download Forbidden` error. +> This error might happen if you use a private link to access your storage account. +> In such cases, if the permissions are correct, check the connectivity to your storage account. ## Set the managed identity policy in Azure Data Explorer -In order to use the managed identity to ingest data into your cluster, allow the `NativeIngestion` usage option for the selected managed identity. Native ingestion refers to the ability to use an SDK for ingestion from an external source. For more information on the available SDKs, see [Client libraries](/kusto/api/client-libraries?view=azure-data-explorer&preserve-view=true). +To use the managed identity to ingest data into your cluster, grant the `NativeIngestion` usage option to the selected managed identity. Native ingestion refers to the ability to use an SDK for ingestion from an external source. For more information on the available SDKs, see [Client libraries](/kusto/api/client-libraries?view=azure-data-explorer&preserve-view=true). -The usage Managed Identity policy can be defined at the cluster or database level of the target cluster. +You can define the usage Managed Identity policy at the cluster or database level of the target cluster. To apply the policy at the database level, run the following command: @@ -68,25 +68,25 @@ Replace `` with the object ID of the required managed ident > [!NOTE] > You must have the `All Database Admin` permission on the cluster to edit the Managed Identity Policy. -## Queue blobs for ingestion with managed identity using Kusto SDK +## Queue blobs for ingestion with managed identity by using Kusto SDK -When ingesting data using a Kusto [SDK](net-sdk-ingest-data.md), generate your [blob URI using managed identity authentication](/kusto/api/connection-strings/storage-connection-strings?view=azure-data-explorer&preserve-view=true#managed-identity) by appending `;managed_identity={objectId}` to the unauthorized blob URI. If you ingest data using your cluster's system assigned managed identity, you can append `;managed_identity=system` to the blob URI. +When you ingest data by using a Kusto [SDK](net-sdk-ingest-data.md), generate your [blob URI by using managed identity authentication](/kusto/api/connection-strings/storage-connection-strings?view=azure-data-explorer&preserve-view=true#managed-identity) by appending `;managed_identity={objectId}` to the unauthorized blob URI. If you ingest data by using your cluster's system assigned managed identity, you can append `;managed_identity=system` to the blob URI. > [!IMPORTANT] > > You must use a queued ingestion client. -> Using managed identities with direct ingestion or inline ingestion in Kusto Query Language are not supported. +> Using managed identities with direct ingestion or inline ingestion in Kusto Query Language aren't supported. -The following are examples of blob URIs for system and user assigned managed identities. +The following examples show blob URIs for system and user assigned managed identities. * System assigned: `https://demosa.blob.core.windows.net/test/export.csv;managed_identity=system` * User assigned: `https://demosa.blob.core.windows.net/test/export.csv;managed_identity=6a5820b9-fdf6-4cc4-81b9-b416b444fd6d` > [!IMPORTANT] > -> * When using Managed Identities to ingest data with the C# SDK, you must provide a blob size in `StorageSourceOptions`. If the size is not set, the SDK attempts to fill in the blob size by accessing the storage account, resulting in a failure. -> * The *size* parameter should be the raw (uncompressed) data size, and not the blob size. -> * If you do not know the size at the time of ingestion, specify a value of zero (0). The service will attempt to discover the size using the managed identity for authentication. +> * When you use managed identities to ingest data by using the C# SDK, you must provide a blob size in `StorageSourceOptions`. If you don't set the size, the SDK attempts to fill in the blob size by accessing the storage account, resulting in a failure. +> * The *size* parameter should be the raw (uncompressed) data size, not the blob size. +> * If you don't know the size at the time of ingestion, specify a value of zero (0). The service attempts to discover the size by using the managed identity for authentication. ## Related content diff --git a/data-explorer/ingest-data-no-code.md b/data-explorer/ingest-data-no-code.md index 2b631bec2a..3731a38e3e 100644 --- a/data-explorer/ingest-data-no-code.md +++ b/data-explorer/ingest-data-no-code.md @@ -1,9 +1,9 @@ --- -title: 'Tutorial: Ingest monitoring data in Azure Data Explorer without code' +title: "Tutorial: Ingest Monitoring Data in Azure Data Explorer Without Code" description: In this tutorial, you learn how to ingest monitoring data to Azure Data Explorer without one line of code and query that data. ms.reviewer: kerend ms.topic: tutorial -ms.date: 11/09/2022 +ms.date: 02/12/2026 ms.custom: sfi-image-nochange # Customer intent: I want to ingest monitoring data to Azure Data Explorer without one line of code, so that I can explore and analyze my data by using queries. @@ -11,9 +11,9 @@ ms.custom: sfi-image-nochange # Tutorial: Ingest and query monitoring data in Azure Data Explorer -This tutorial will teach you how to ingest data from diagnostic and activity logs to an Azure Data Explorer cluster without writing code. With this simple ingestion method, you can quickly begin querying Azure Data Explorer for data analysis. +This tutorial shows you how to ingest data from diagnostic and activity logs to an Azure Data Explorer cluster without writing code. By using this simple ingestion method, you can quickly start querying Azure Data Explorer for data analysis. -In this tutorial, you'll learn how to: +In this tutorial, you learn how to: > [!div class="checklist"] > @@ -33,17 +33,17 @@ In this tutorial, you'll learn how to: ## Azure Monitor data provider: diagnostic metrics and logs and activity logs -View and understand the data provided by the Azure Monitor diagnostic metrics and logs and activity logs below. You'll create an ingestion pipeline based on these data schemas. Note that each event in a log has an array of records. This array of records will be split later in the tutorial. +View and understand the data provided by the Azure Monitor diagnostic metrics and logs and activity logs in the following sections. You create an ingestion pipeline based on these data schemas. Each event in a log has an array of records. You split this array of records later in the tutorial. ### Examples of diagnostic metrics and logs and activity logs -Azure diagnostic metrics and logs and activity logs are emitted by an Azure service and provide data about the operation of that service. +An Azure service emits Azure diagnostic metrics and logs and activity logs. These logs provide data about the operation of that service. ### [Diagnostic metrics](#tab/diagnostic-metrics) #### Diagnostic metrics example -Diagnostic metrics are aggregated with a time grain of 1 minute. Following is an example of an Azure Data Explorer metric-event schema on query duration: +Diagnostic metrics aggregate data with a time grain of one minute. The following example shows an Azure Data Explorer metric-event schema on query duration: ```json { @@ -78,7 +78,7 @@ Diagnostic metrics are aggregated with a time grain of 1 minute. Following is an #### Diagnostic logs example -Following is an example of an Azure Data Explorer [diagnostic ingestion log](using-diagnostic-logs.md#diagnostic-logs-schema): +The following example shows an Azure Data Explorer [diagnostic ingestion log](using-diagnostic-logs.md#diagnostic-logs-schema): ```json { @@ -136,7 +136,7 @@ Following is an example of an Azure Data Explorer [diagnostic ingestion log](usi #### Activity logs example -Azure activity logs are subscription-level logs that provide insight into the operations performed on resources in your subscription. Following is an example of an activity-log event for checking access: +Azure activity logs are subscription-level logs that provide insight into the operations performed on resources in your subscription. The following example shows an activity-log event for checking access: ```json { @@ -207,7 +207,7 @@ In your Azure Data Explorer *TestDatabase* database, select **Query** to open th ### Create the target tables -The structure of the Azure Monitor logs isn't tabular. You'll manipulate the data and expand each event to one or more records. The raw data will be ingested to an intermediate table named *ActivityLogsRawRecords* for activity logs and *DiagnosticRawRecords* for diagnostic metrics and logs. At that time, the data will be manipulated and expanded. Using an update policy, the expanded data will then be ingested into the *ActivityLogs* table for activity logs, *DiagnosticMetrics* for diagnostic metrics and *DiagnosticLogs* for diagnostic logs. This means that you'll need to create two separate tables for ingesting activity logs and three separate tables for ingesting diagnostic metrics and logs. +The structure of the Azure Monitor logs isn't tabular. You need to manipulate the data and expand each event to one or more records. Ingest the raw data to an intermediate table named *ActivityLogsRawRecords* for activity logs and *DiagnosticRawRecords* for diagnostic metrics and logs. At that time, you manipulate and expand the data. By using an update policy, ingest the expanded data into the *ActivityLogs* table for activity logs, *DiagnosticMetrics* for diagnostic metrics, and *DiagnosticLogs* for diagnostic logs. This process means that you need to create two separate tables for ingesting activity logs and three separate tables for ingesting diagnostic metrics and logs. Use the Azure Data Explorer web UI to create the target tables in the Azure Data Explorer database. @@ -225,7 +225,7 @@ Use the Azure Data Explorer web UI to create the target tables in the Azure Data ![Run query.](media/ingest-data-no-code/run-query.png) -1. Create the intermediate data table named *DiagnosticRawRecords* in the *TestDatabase* database for data manipulation using the following query. Select **Run** to create the table. +1. Create the intermediate data table named *DiagnosticRawRecords* in the *TestDatabase* database for data manipulation by using the following query. Select **Run** to create the table. ```kusto .create table DiagnosticRawRecords (Records:dynamic) @@ -249,7 +249,7 @@ Use the Azure Data Explorer web UI to create the target tables in the Azure Data 1. Select **Run** to create the table. -1. Create the intermediate data table named *DiagnosticRawRecords* in the *TestDatabase* database for data manipulation using the following query. Select **Run** to create the table. +1. Create the intermediate data table named *DiagnosticRawRecords* in the *TestDatabase* database for data manipulation by using the following query. Select **Run** to create the table. ```kusto .create table DiagnosticRawRecords (Records:dynamic) @@ -265,7 +265,7 @@ Use the Azure Data Explorer web UI to create the target tables in the Azure Data #### Create tables for the activity logs -1. Create a table named *ActivityLogs* in the *TestDatabase* database to receive activity log records. To create the table, run the following Azure Data Explorer query: +1. To receive activity log records, create a table named *ActivityLogs* in the *TestDatabase* database. To create the table, run the following Azure Data Explorer query: ```kusto .create table ActivityLogs (Timestamp:datetime, ResourceId:string, OperationName:string, Category:string, ResultType:string, ResultSignature:string, DurationMs:int, IdentityAuthorization:dynamic, IdentityClaims:dynamic, Location:string, Level:string) @@ -287,13 +287,13 @@ Use the Azure Data Explorer web UI to create the target tables in the Azure Data ### Create table mappings - Because the data format is `json`, data mapping is required. The `json` mapping maps each json path to a table column name. JSON paths that include special characters should be escaped as [\'Property Name\']. For more information, see [JSONPath syntax](/kusto/query/jsonpath?view=azure-data-explorer&preserve-view=true). + Because the data format is `json`, you need to create a data mapping. The `json` mapping connects each JSON path to a table column name. If a JSON path has special characters, escape them as [\'Property Name\']. For more information, see [JSONPath syntax](/kusto/query/jsonpath?view=azure-data-explorer&preserve-view=true). ### [Diagnostic metrics / Diagnostic logs](#tab/diagnostic-metrics+diagnostic-logs) #### Map diagnostic metrics and logs to the table -To map the diagnostic metric and log data to the table, use the following query: +Use the following query to map the diagnostic metric and log data to the table: ```kusto .create table DiagnosticRawRecords ingestion json mapping 'DiagnosticRawRecordsMapping' '[{"column":"Records","Properties":{"path":"$.records"}}]' @@ -303,7 +303,7 @@ To map the diagnostic metric and log data to the table, use the following query: #### Map activity logs to the table -To map the activity log data to the table, use the following query: +Use the following query to map the activity log data to the table: ```kusto .create table ActivityLogsRawRecords ingestion json mapping 'ActivityLogsRawRecordsMapping' '[{"column":"Records","Properties":{"path":"$.records"}}]' @@ -317,7 +317,7 @@ To map the activity log data to the table, use the following query: #### Create data update policy for diagnostics metrics -1. Create a [function](/kusto/management/functions?view=azure-data-explorer&preserve-view=true) that expands the collection of diagnostic metric records so that each value in the collection receives a separate row. Use the [`mv-expand`](/kusto/query/mv-expand-operator?view=azure-data-explorer&preserve-view=true) operator: +1. Create a [function](/kusto/management/functions?view=azure-data-explorer&preserve-view=true) that expands the collection of diagnostic metric records so that each value in the collection gets a separate row. Use the [`mv-expand`](/kusto/query/mv-expand-operator?view=azure-data-explorer&preserve-view=true) operator: ```kusto .create function DiagnosticMetricsExpand() { @@ -337,7 +337,7 @@ To map the activity log data to the table, use the following query: } ``` -2. Add the [update policy](/kusto/management/update-policy?view=azure-data-explorer&preserve-view=true) to the target table. This policy will automatically run the query on any newly ingested data in the *DiagnosticRawRecords* intermediate data table and ingest its results into the *DiagnosticMetrics* table: +1. Add the [update policy](/kusto/management/update-policy?view=azure-data-explorer&preserve-view=true) to the target table. This policy automatically runs the query on any newly ingested data in the *DiagnosticRawRecords* intermediate data table and ingests its results into the *DiagnosticMetrics* table: ```kusto .alter table DiagnosticMetrics policy update @'[{"Source": "DiagnosticRawRecords", "Query": "DiagnosticMetricsExpand()", "IsEnabled": "True", "IsTransactional": true}]' @@ -347,7 +347,7 @@ To map the activity log data to the table, use the following query: #### Create data update policy for diagnostics logs -1. Create a [function](/kusto/management/functions) that expands the collection of diagnostic logs records so that each value in the collection receives a separate row. You'll enable ingestion logs on an Azure Data Explorer cluster, and use [ingestion logs schema](using-diagnostic-logs.md?view=azure-data-explorer&preserve-view=true#diagnostic-logs-schema). You'll create one table for succeeded and for failed ingestion, while some of the fields will be empty for succeeded ingestion (ErrorCode for example). Use the [`mv-expand`](/kusto/query/mv-expand-operator?view=azure-data-explorer&preserve-view=true) operator: +1. Create a [function](/kusto/management/functions) that expands the collection of diagnostic logs records so that each value in the collection gets a separate row. You enable ingestion logs on an Azure Data Explorer cluster, and use [ingestion logs schema](using-diagnostic-logs.md?view=azure-data-explorer&preserve-view=true#diagnostic-logs-schema). You create one table for succeeded and for failed ingestion, while some of the fields are empty for succeeded ingestion (ErrorCode, for example). Use the [`mv-expand`](/kusto/query/mv-expand-operator?view=azure-data-explorer&preserve-view=true) operator: ```kusto .create function DiagnosticLogsExpand() { @@ -371,7 +371,7 @@ To map the activity log data to the table, use the following query: } ``` -2. Add the [update policy](/kusto/management/update-policy?view=azure-data-explorer&preserve-view=true) to the target table. This policy will automatically run the query on any newly ingested data in the *DiagnosticRawRecords* intermediate data table and ingest its results into the *DiagnosticLogs* table: +1. Add the [update policy](/kusto/management/update-policy?view=azure-data-explorer&preserve-view=true) to the target table. This policy automatically runs the query on any newly ingested data in the *DiagnosticRawRecords* intermediate data table and ingests its results into the *DiagnosticLogs* table: ```kusto .alter table DiagnosticLogs policy update @'[{"Source": "DiagnosticRawRecords", "Query": "DiagnosticLogsExpand()", "IsEnabled": "True", "IsTransactional": true}]' @@ -381,7 +381,7 @@ To map the activity log data to the table, use the following query: #### Create data update policy for activity logs -1. Create a [function](/kusto/management/functions?view=azure-data-explorer&preserve-view=true) that expands the collection of activity log records so that each value in the collection receives a separate row. Use the [`mv-expand`](/kusto/query/mv-expand-operator?view=azure-data-explorer&preserve-view=true) operator: +1. Create a [function](/kusto/management/functions?view=azure-data-explorer&preserve-view=true) that expands the collection of activity log records so that each value in the collection gets a separate row. Use the [`mv-expand`](/kusto/query/mv-expand-operator?view=azure-data-explorer&preserve-view=true) operator: ```kusto .create function ActivityLogRecordsExpand() { @@ -402,7 +402,7 @@ To map the activity log data to the table, use the following query: } ``` -2. Add the [update policy](/kusto/management/update-policy?view=azure-data-explorer&preserve-view=true) to the target table. This policy will automatically run the query on any newly ingested data in the *ActivityLogsRawRecords* intermediate data table and ingest its results into the *ActivityLogs* table: +1. Add the [update policy](/kusto/management/update-policy?view=azure-data-explorer&preserve-view=true) to the target table. This policy automatically runs the query on any newly ingested data in the *ActivityLogsRawRecords* intermediate data table and ingests its results into the *ActivityLogs* table: ```kusto .alter table ActivityLogs policy update @'[{"Source": "ActivityLogsRawRecords", "Query": "ActivityLogRecordsExpand()", "IsEnabled": "True", "IsTransactional": true}]' @@ -412,13 +412,13 @@ To map the activity log data to the table, use the following query: ## Create an Azure Event Hubs namespace -Azure diagnostic settings enable exporting metrics and logs to a storage account or to an event hub. In this tutorial, we'll route the metrics and logs via an event hub. You'll create an event hub namespace and an event hub for the diagnostic metrics and logs in the following steps. Azure Monitor will create the event hub *insights-operational-logs* for the activity logs. +Azure diagnostic settings enable exporting metrics and logs to a storage account or to an event hub. In this tutorial, you route the metrics and logs through an event hub. You create an event hub namespace and an event hub for the diagnostic metrics and logs in the following steps. Azure Monitor creates the event hub *insights-operational-logs* for the activity logs. 1. Create an event hub by using an Azure Resource Manager template in the Azure portal. To follow the rest of the steps in this article, right-click the **Deploy to Azure** button, and then select **Open in new window**. The **Deploy to Azure** button takes you to the Azure portal. [![Deploy to Azure button.](media/ingest-data-event-hub/deploybutton.png)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2Fquickstarts%2Fmicrosoft.eventhub%2Feventhubs-create-namespace-and-eventhub%2Fazuredeploy.json) -1. Create an event hubs namespace and an event hub for the diagnostic logs. Learn how to [create an event hubs namespace](/azure/event-hubs/event-hubs-create). +1. Create an Event Hubs namespace and an event hub for the diagnostic logs. Learn how to [create an Event Hubs namespace](/azure/event-hubs/event-hubs-create). 1. Fill out the form with the following information. For any settings not listed in the following table, use the default values. @@ -429,7 +429,7 @@ Azure diagnostic settings enable exporting metrics and logs to a storage account | **Location** | Select the region that best meets your needs. | Create the event hub namespace in the same location as other resources. | **Namespace name** | *AzureMonitoringData* | Choose a unique name that identifies your namespace. | **Event hub name** | *DiagnosticData* | The event hub sits under the namespace, which provides a unique scoping container. | - | **Consumer group name** | *adxpipeline* | Create a consumer group name. Consumer groups enable multiple consuming applications to each have a separate view of the event stream. | + | **Consumer group name** | *adxpipeline* | Create a consumer group name. Consumer groups enable multiple consuming applications to each have a separate view of the eventstream. | | | | ## Connect Azure Monitor metrics and logs to your event hub @@ -440,7 +440,7 @@ Now you need to connect your diagnostic metrics and logs and your activity logs #### Connect diagnostic metrics and logs to your event hub -Select a resource from which to export metrics. Several resource types support exporting diagnostic data, including event hubs namespace, Azure Key Vault, Azure IoT Hub, and Azure Data Explorer clusters. In this tutorial, we'll use an Azure Data Explorer cluster as our resource, we'll review query performance metrics and ingestion results logs. +Select a resource from which to export metrics. Several resource types support exporting diagnostic data, including event hubs namespace, Azure Key Vault, Azure IoT Hub, and Azure Data Explorer clusters. In this tutorial, use an Azure Data Explorer cluster as your resource. You can review query performance metrics and ingestion results logs. 1. Select your Kusto cluster in the Azure portal. 1. Select **Diagnostic settings**, and then select the **Turn on diagnostics** link. @@ -481,21 +481,21 @@ Select a resource from which to export metrics. Several resource types support e :::image type="content" source="media/ingest-data-no-code/export-activity-log.PNG" alt-text="Diagnostic settings window with fields to fill out - Azure Data Explorer portal."::: - Do the following steps: + Complete the following steps: 1. Enter a name in the **Diagnostic setting name** field. - 1. On the left-hand side of check boxes, select the platform log(s) you wish to collect from a subscription. + 1. On the left-hand side of check boxes, select the platform logs you want to collect from a subscription. 1. Select the **Stream to an event hub** check box. 1. Select your subscription. 1. In the **Event hub namespace** list, select *AzureMonitoringData*. 1. Optionally, select your **Event hub name**. 1. In the **Event hub policy name** list, select the default event hub policy name. - 1. In the upper-left corner of the window, select **Save**. An event hub with the name *insights-operational-logs* will be created (unless you've selected an Event hub name above). + 1. In the upper-left corner of the window, select **Save**. An event hub with the name *insights-operational-logs* is created unless you select an Event hub name. --- ### See data flowing to your event hubs -1. Wait a few minutes until the connection is defined, and the activity-log export to the event hub is finished. Go to your event hubs namespace to see the event hubs you created. +1. Wait a few minutes until the connection is defined, and the activity-log export to the event hub finishes. Go to your event hubs namespace to see the event hubs you created. ![Event hubs created.](media/ingest-data-no-code/event-hubs-created.png) @@ -533,7 +533,7 @@ Now you need to create the data connections for your diagnostic metrics and logs Target table: - There are two options for routing: *static* and *dynamic*. For this tutorial, you'll use static routing (the default), where you specify the table name, the data format, and the mapping. Leave **My data includes routing info** unselected. + There are two options for routing: *static* and *dynamic*. For this tutorial, you use static routing (the default), where you specify the table name, the data format, and the mapping. Leave **My data includes routing info** unselected. **Setting** | **Suggested value** | **Field description** |---|---|---| @@ -560,7 +560,7 @@ Now you need to create the data connections for your diagnostic metrics and logs Target table: - There are two options for routing: *static* and *dynamic*. For this tutorial, you'll use static routing (the default), where you specify the table name, data format, and mapping. Leave **My data includes routing info** unselected. + There are two options for routing: *static* and *dynamic*. For this tutorial, you use static routing (the default), where you specify the table name, data format, and mapping. Leave **My data includes routing info** unselected. **Setting** | **Suggested value** | **Field description** |---|---|---| @@ -575,7 +575,7 @@ Now you need to create the data connections for your diagnostic metrics and logs ## Query the new tables -You now have a pipeline with data flowing. Ingestion via the cluster takes 5 minutes by default, so allow the data to flow for a few minutes before beginning to query. +You now have a pipeline with data flowing. Ingestion through the cluster takes five minutes by default, so wait a few minutes before you start querying. ### [Diagnostic metrics](#tab/diagnostic-metrics) @@ -599,8 +599,8 @@ Query results: #### Query the diagnostic logs table -This pipeline produces ingestions via an event hub. You'll review the results of these ingestions. -The following query analyzes how many ingestions accrued in a minute, including a sample of `Database`, `Table` and `IngestionSourcePath` for each interval: +This pipeline produces ingestions through an event hub. You can review the results of these ingestions. +The following query analyzes how many ingestions accrued in a minute, including a sample of `Database`, `Table`, and `IngestionSourcePath` for each interval: ```kusto DiagnosticLogs @@ -638,5 +638,5 @@ Query results: ## Related content * [Write queries for Azure Data Explorer](/azure/data-explorer/kusto/query/tutorials/learn-common-operators). -* [Monitor Azure Data Explorer ingestion operations using diagnostic logs](using-diagnostic-logs.md) -* [Use metrics to monitor cluster health](using-metrics.md) +* [Monitor Azure Data Explorer ingestion operations using diagnostic logs](using-diagnostic-logs.md). +* [Use metrics to monitor cluster health](using-metrics.md). diff --git a/data-explorer/kusto/query/tutorials/learn-common-operators.md b/data-explorer/kusto/query/tutorials/learn-common-operators.md index 34c383eb8b..21912113ce 100644 --- a/data-explorer/kusto/query/tutorials/learn-common-operators.md +++ b/data-explorer/kusto/query/tutorials/learn-common-operators.md @@ -1,15 +1,18 @@ --- -title: 'Tutorial: Learn common Kusto Query Language operators' +title: "Tutorial: Learn Common Kusto Query Language Operators" description: This tutorial describes how to write queries using common operators in the Kusto Query Language to meet common query needs. ms.topic: tutorial -ms.date: 02/01/2026 +ms.date: 02/12/2026 +author: hzargari-ms +ms.author: v-hzargari +ms.reviewer: v-hzargari --- # Tutorial: Learn common operators > [!INCLUDE [applies](../../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../../includes/applies-to-version/azure-data-explorer.md)] [!INCLUDE [monitor](../../includes/applies-to-version/monitor.md)] [!INCLUDE [sentinel](../../includes/applies-to-version/sentinel.md)] -Use [Kusto Query Language (KQL)](../index.md) to write queries in [Azure Data Explorer](https://dataexplorer.azure.com/), [Azure Monitor Log Analytics](https://azure.microsoft.com/products/monitor/#overview), [Microsoft Sentinel](https://azure.microsoft.com/products/microsoft-sentinel/), and more. This tutorial introduces the essential KQL operators you can use to access and analyze your data. +You use [Kusto Query Language (KQL)](../index.md) to write queries in [Azure Data Explorer](https://dataexplorer.azure.com/), [Azure Monitor Log Analytics](https://azure.microsoft.com/products/monitor/#overview), [Microsoft Sentinel](https://azure.microsoft.com/products/microsoft-sentinel/), and more. This tutorial introduces the essential KQL operators you use to access and analyze your data. For more specific guidance on how to query logs in Azure Monitor, see [Get started with log queries](/azure/azure-monitor/logs/get-started-queries). @@ -127,7 +130,7 @@ StormEvents ## List unique values -The results of the previous query show that there are multiple types of storms. Use the [distinct](../distinct-operator.md) operator to list all of the unique storm types. +The results of the previous query show multiple types of storms. Use the [distinct](../distinct-operator.md) operator to list all of the unique storm types. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] @@ -214,7 +217,7 @@ There are 146 events that match these conditions. Here's a sample of five of the Use the [between operator](../between-operator.md) to filter data based on a specific time range. -The following query finds all storm events between August 1, 2007, and August 30, 2007, along with their states, event types, start times, and end times. The query returns results sorted in ascending order by start time. +The following query finds all storm events between August 1, 2007, and August 30, 2007, along with their states, event types, start times, and end times. The query sorts the results in ascending order by start time. :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] diff --git a/data-explorer/provision-entra-id-app.md b/data-explorer/provision-entra-id-app.md index 0885d70ee9..225729ad53 100644 --- a/data-explorer/provision-entra-id-app.md +++ b/data-explorer/provision-entra-id-app.md @@ -1,13 +1,13 @@ --- -title: Create a Microsoft Entra application in Azure Data Explorer +title: Create a Microsoft Entra Application in Azure Data Explorer description: Learn how to create a Microsoft Entra application in Azure Data Explorer. ms.topic: how-to -ms.date: 01/11/2024 +ms.date: 02/12/2026 --- # Create a Microsoft Entra application registration in Azure Data Explorer -[Microsoft Entra application authentication](/entra/identity-platform/howto-create-service-principal-portal) is used for applications, such as an unattended service or a scheduled flow, that need to access Azure Data Explorer without a user present. If you're connecting to an Azure Data Explorer database using an application, such as a web app, you should authenticate using service principal authentication. This article details how to create and register a Microsoft Entra service principal and then authorize it to access an Azure Data Explorer database. +Use [Microsoft Entra application authentication](/entra/identity-platform/howto-create-service-principal-portal) for applications, such as an unattended service or a scheduled flow, that need to access Azure Data Explorer without a user present. If you're connecting to an Azure Data Explorer database by using an application, such as a web app, authenticate by using service principal authentication. This article explains how to create and register a Microsoft Entra service principal and then authorize it to access an Azure Data Explorer database. diff --git a/data-explorer/troubleshoot-connect-cluster.md b/data-explorer/troubleshoot-connect-cluster.md index 362f24c7b9..4d8d740cce 100644 --- a/data-explorer/troubleshoot-connect-cluster.md +++ b/data-explorer/troubleshoot-connect-cluster.md @@ -1,37 +1,37 @@ --- -title: Troubleshoot Azure Data Explorer cluster connection failures +title: Troubleshoot Azure Data Explorer Cluster Connection Failures description: This article describes troubleshooting steps for connecting to a cluster in Azure Data Explorer. ms.reviewer: mblythe ms.topic: how-to -ms.date: 01/21/2025 +ms.date: 02/12/2026 --- # Troubleshoot: Failure to connect to a cluster in Azure Data Explorer -If you're not able to connect to a cluster in Azure Data Explorer, follow these steps. +If you can't connect to a cluster in Azure Data Explorer, try the following steps. -1. Ensure the connection string is correct. It should be in the form: `https://..kusto.windows.net`, such as the following example: `https://docscluster.westus.kusto.windows.net`. +1. Make sure the connection string is correct. It should be in the form: `https://..kusto.windows.net`. For example, `https://docscluster.westus.kusto.windows.net`. -1. Ensure you have adequate permissions. Otherwise, you get a response of *unauthorized*. +1. Make sure you have the right permissions. If you don't, you get an *unauthorized* response. For more information about permissions, see [Manage database permissions](manage-database-permissions.md). If necessary, work with your cluster administrator so they can add you to the appropriate role. -1. If you're connecting from an external tenant, ensure the cluster has correct permissions. +1. If you're connecting from an external tenant, make sure the cluster has the correct permissions. For more information about cross tenant scenarios, see [Allow cross-tenant queries and commands](cross-tenant-query-and-commands.md). -1. Ensure your cluster is active. Clusters can automatically stop due to [auto-stop settings](auto-stop-clusters.md). +1. Make sure your cluster is active. Clusters can automatically stop due to [auto-stop settings](auto-stop-clusters.md). - 1. From the [Azure portal](https://ms.portal.azure.com/), navigate to **Azure Data Explorer Clusters** and view your cluster. + 1. From the [Azure portal](https://ms.portal.azure.com/), go to **Azure Data Explorer Clusters** and view your cluster. 1. If the value in the **State** column is **Stopped**, select your cluster to open the overview page. 1. From the **Command bar**, select **Start**. Then try reconnecting. -1. Verify that the cluster wasn't deleted by reviewing your subscription activity log. +1. Check your subscription activity log to verify that the cluster wasn't deleted. 1. Check the [Azure service health dashboard](https://azure.microsoft.com/status/). Look for the status of Azure Data Explorer in the region where you're trying to connect to a cluster. If the status isn't **Good** (green check mark), try connecting to the cluster after the status improves. -1. If you still need assistance solving your issue, open a support request in the [Azure portal](https://portal.azure.com/#blade/Microsoft_Azure_Support/HelpAndSupportBlade/overview). +1. If you still need assistance solving your problem, open a support request in the [Azure portal](https://portal.azure.com/#blade/Microsoft_Azure_Support/HelpAndSupportBlade/overview). diff --git a/data-explorer/web-ui-kql.md b/data-explorer/web-ui-kql.md index 06d75947c4..d1d7ade881 100644 --- a/data-explorer/web-ui-kql.md +++ b/data-explorer/web-ui-kql.md @@ -2,8 +2,10 @@ title: Write Kusto Query Language Queries in the Azure Data Explorer Web UI description: In this article, you learn how to write Kusto Query Language (KQL) queries in the Azure Data Explorer web UI. ms.topic: how-to -ms.date: 02/02/2026 +ms.date: 02/12/2026 +ms.reviewer: v-hzargari --- + # Write Kusto Query Language queries in the Azure Data Explorer web UI The [Azure Data Explorer web UI](https://dataexplorer.azure.com/) query editor offers various features to help you write [Kusto Query Language (KQL)](/kusto/query/index?view=azure-data-explorer&preserve-view=true) queries. Some of these features include built-in KQL Intellisense and autocomplete, inline documentation, and quick fix pop-ups. In this article, you learn what you should know when writing KQL queries in the web UI. @@ -16,7 +18,7 @@ To use KQL Intellisense and autocomplete: 1. Start typing a query in the query editor. -1. KQL Intellisense activates, presenting dropdown options for entities, operators, functions, and more. Move between these options by using the arrow keys on your keyboard, and select one by pressing *Enter* or selecting the option by using your mouse. +1. KQL Intellisense activates, presenting dropdown options for entities, operators, functions, and more. Move between these options by using the arrow keys on your keyboard, and select one by pressing *Enter* or select the option by using your mouse. 1. If you notice underlined keywords, hover over them to reveal errors or suggestions that triggered the highlighting.