Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion chat-builder/android/integration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ dependencies {
implementation 'com.cometchat:chat-uikit-android:5.1.0'

// Optional: voice/video calling
implementation 'com.cometchat:calls-sdk-android:4.1.2'
implementation 'com.cometchat:calls-sdk-android:4.3.1'
}
```

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/moderation-custom-api-create-rule.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
74 changes: 66 additions & 8 deletions moderation/custom/custom-api-overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,72 @@
title: "Overview"
---

CometChat offers AI-powered message moderation to help maintain a safe and respectful chat environment. You can choose between two moderation options:
CometChat allows you to integrate your own moderation logic using a **Custom API**. This "bring your own moderation" approach lets you use any third-party service (OpenAI Moderation, Perspective API, etc.) or your own AI model while CometChat handles message interception and action enforcement.

### **Custom API Moderation**
## How It Works

If you prefer to use a third-party moderation service or your own AI model, CometChat enables integration via a **Custom Moderation API**. With this option, you can:
- **Set Up a Webhook** – Configure an endpoint where messages will be sent for moderation.
- **Customize Authentication** – Add security layers like basic authentication.
- **Contextual Moderation** – Define how many previous messages from the conversation should be included in the webhook request for better analysis.
- **Process Moderation Decisions** – CometChat processes the webhook response and applies moderation actions accordingly.
```mermaid
sequenceDiagram
participant User
participant CometChat
participant Your Webhook

User->>CometChat: Sends message
CometChat->>Your Webhook: POST message + context
Your Webhook->>Your Webhook: Apply moderation logic
Your Webhook->>CometChat: Return decision
CometChat->>CometChat: Apply configured action
CometChat->>User: Message delivered or blocked
```

This moderation provide flexibility to enhance user safety and compliance within your chat platform.
1. **User sends a message** in your chat application
2. **CometChat intercepts** the message and POSTs it to your webhook URL along with optional conversation context (previous messages as plain text)
3. **Your webhook processes** the message using your custom moderation logic
4. **Your webhook responds** with a JSON decision containing `isMatchingCondition`, `confidence`, and optional `reason`
5. **CometChat compares** the returned confidence against your configured threshold and applies the appropriate action (block, flag for review, etc.)

## Getting Started

<Steps>
<Step title="Build Your Moderation Endpoint">
Create a webhook that accepts POST requests with message data. Your endpoint must return JSON with:
- `isMatchingCondition` (boolean) – whether the content violates rules
- `confidence` (number 0.0-1.0) – confidence score of the decision
- `reason` (string, optional) – explanation for flagging
</Step>
<Step title="Create a Custom API List">
In the CometChat Dashboard, go to **Moderation** → **Settings** → **List** and create a new list with:
- **Category**: Custom API
- **URL**: Your webhook endpoint
- **Basic Auth** (optional): Username and password for secure authentication
</Step>
<Step title="Configure Advanced Settings">
Under **Moderation** → **Settings** → **Advanced Settings**, configure:
- **Context messages** (0-10): Number of previous messages to include for context-aware moderation
- **Error handling**: Choose to approve or block messages when your API is unavailable
</Step>
<Step title="Create a Moderation Rule">
Go to **Moderation** → **Rules** and create a rule that:
- Selects your Custom API list
- Sets a confidence threshold (e.g., "greater than 80%")
- Defines actions when content is flagged
<Card title="Custom API Configuration" icon="gear" href="/moderation/custom/custom-api">
Step-by-step guide to configure your custom moderation API
</Card>
</Step>
<Step title="Handle Moderation Events (Optional)">
Set up webhooks to receive notifications when messages are approved or blocked by your moderation logic.
<Card title="Moderation Events" icon="bell" href="/fundamentals/webhooks-overview#moderation-events">
Learn about moderation webhook events
</Card>
</Step>
</Steps>

## Key Features

- **Contextual Moderation** – Include up to 10 previous messages from the conversation for context-aware analysis
- **Secure Authentication** – Optional Basic Auth support for webhook security
- **Confidence-Based Decisions** – Set confidence thresholds to fine-tune when actions are triggered
- **Configurable Error Handling** – Choose to approve or block messages when your API is unavailable or returns an error
- **Full Message Details** – Receive complete message objects including sender/receiver info, timestamps, and metadata
- **Real-time Processing** – Moderation decisions are applied before message delivery
158 changes: 118 additions & 40 deletions moderation/custom/custom-api.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,63 +2,115 @@
title: "Custom API Moderation"
---

CometChat allows you to integrate your own moderation logic using a **Custom API**. With this feature, you can define a webhook URL in the **List Configuration**, where CometChat will send messages for moderation along with relevant context from the conversation (if provided in settings).
CometChat allows you to integrate your own moderation logic using a **Custom API**. With this feature, you can define a webhook URL where CometChat will send messages for moderation along with relevant context from the conversation. This enables you to use your own moderation service, third-party AI moderation APIs, or custom business logic to moderate content.

## **How It Works**
## Integration Steps

1. When a user sends a message, CometChat retrieves the webhook URL from the configured **List**.
2. The message, along with previous conversation messages (if a context window is set in settings), is sent to the webhook.
3. The webhook (your external API) processes the data using your custom moderation logic.
4. The webhook responds with a structured decision containing details about the moderation outcome.
5. CometChat processes the response and applies the moderation decision in real-time.
### Step 1: Create a Custom API List

This approach gives you complete control over moderation, allowing you to implement **custom filtering, AI-based analysis, or any other logic** on your own servers.
First, you need to create a list that defines your Custom API webhook configuration.

## Integration
1. Navigate to the [CometChat Dashboard](https://app.cometchat.com) and select your app.

### Step 1: Configure Custom API Settings
2. Go to **Moderation** → **Settings** from the left-hand menu.

1. **Login to the CometChat Dashboard**
3. Click on the **List** tab.

* Navigate to [CometChat Dashboard](https://app.cometchat.com) and select your app.
4. Click **Add List** to create a new list.

2. **Navigate to Moderation Settings**
5. Fill in the required fields:
- **Name**: Enter a descriptive name for your custom moderation list (e.g., `custom-moderation`)
- **ID**: Enter a unique identifier for the list (e.g., `custom-moderation-list`)
- **Category**: Select **Custom API** from the dropdown
- **URL**: Enter the webhook URL where CometChat will POST message data for moderation

* Go to **Moderation → Settings** in the left-hand menu.
6. **Enable Basic Auth** (Optional but Recommended):
- Toggle on **Enable Basic Auth** for added security
- Enter your **Basic Auth Username**
- Enter your **Basic Auth Password**
- CometChat will include these credentials in the `Authorization` header when calling your webhook

3. **Open Custom API Settings Tab**
7. Click **Save** to create the list.

* Click on the **Custom API** tab within the Moderation Settings.
### Step 2: Configure Advanced Settings

4. **Fill in the Custom API Configuration**
Next, configure the context and error handling settings for your Custom API.

* **Set Action on API Error**
1. Navigate to **Moderation** → **Settings**.

* Define how the system should respond if the Custom API is unavailable (e.g., "Allow message" or "Block message").
2. Click on the **Advanced Settings** tab.

* **Set Context Window**
3. Scroll down to the **Custom API** section.

* Specify the number of previous messages in a conversation that will be used for context.
4. Configure the following options:

5. **Click Save Settings**
- **The number of messages sent to Custom API for context**:
- Set the number of previous messages to include as context (0-10)
- This enables context-aware moderation by providing conversation history
- Set to `0` if you don't need context and only want to moderate individual messages

### Step 2: Enable Custom API Moderation
- **On Custom API Error**:
- **Approve Message** (Default): Messages are automatically approved if your webhook is unavailable or returns an error
- **Block Message**: Messages are blocked when your webhook is inaccessible, ensuring no unmoderated content passes through

1. Navigate to **Moderation → Rules**.
2. Click **"Create New Rule"**.
3. Select **Custom API** as the moderation type.
4. The rule you create should be of type **"Text Contains"** or **"Image Contains"**.
5. Save the rule.
5. Click **Save** to apply the settings.

## Payload Sent to Webhook
<Frame>
<img src="/images/moderation-custom-api-advanced-settings.png" alt="Custom API Advanced Settings" />
</Frame>

When a message is sent, CometChat invokes your webhook with a payload that includes:
### Step 3: Create a Moderation Rule

* he latest message (the one just sent) — provided in full detail (entire message object)
Now, map your Custom API list to a moderation rule to activate the moderation.

* The previous messages — provided as plain text only, for context (based on the context window setting)
1. Navigate to **Moderation** → **Rules**.

This structure allows you to apply moderation logic to the current message while considering its surrounding context.
2. Click **Create New Rule**.

3. Fill in the mandatory fields:
- **Name**: Enter a name for the rule (e.g., `test-custom-api`)
- **ID**: Enter a unique identifier (e.g., `testcustomapi`)
- **Description**: Optionally describe what this rule does

4. Configure the **Conditions**:
- Select the content type to moderate (e.g., **Text**)
- Select **Contains**
- Select **Custom API**
- Choose the Custom API list you created in Step 1 from the dropdown
- Set the **confidence** threshold (e.g., `greater than 80%`)
- Higher percentages result in stricter matching
- Lower percentages allow more content to be flagged

5. Configure the **Actions** to perform when content is flagged (block, flag for review, etc.).

6. Click **Save** to create the rule.

7. **Enable the rule** by toggling it on.

<Frame>
<img src="/images/moderation-custom-api-create-rule.png" alt="Create Custom API Rule"/>
</Frame>

Once enabled, all messages matching your filter criteria will be sent to your webhook URL for moderation.

---

## Webhook Request

When a message is sent, CometChat will POST the message data to your configured webhook URL.

### Headers

| Header | Description |
|--------|-------------|
| `Content-Type` | `application/json` |
| `Authorization` | Basic auth credentials (if configured) |

### Payload Structure

The payload includes:
- **Context messages**: Previous messages in the conversation (as plain text) based on your context window setting
- **Latest message**: The message being moderated (full message object with all details)

```json
{
Expand Down Expand Up @@ -122,14 +174,40 @@ This structure allows you to apply moderation logic to the current message while
}
```

## Webhook Response Format
---

## Webhook Response

Your webhook must return a JSON response indicating the moderation decision.

### When content violates rules

```json
{
"isMatchingCondition": true,
"confidence": 0.95,
"reason": "Contains prohibited content"
}
```

The webhook should return a response in the following format:
### When content is safe

```javascript
{
isMatchingCondition: true, // True if the message violates the rule
confidence: 0.95, // Confidence score of the decision
reason: "Contains hate speech" // Reason for flagging
```json
{
"isMatchingCondition": false,
"confidence": 0.98,
"reason": ""
}
```

### Response Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `isMatchingCondition` | boolean | Yes | `true` if the message violates the rule, `false` if safe |
| `confidence` | number | Yes | Confidence score of the decision (0.0 - 1.0). This is compared against the threshold set in your rule conditions |
| `reason` | string | No | Reason for flagging (can be empty for safe content) |

<Note>
The `confidence` value returned by your webhook is compared against the confidence threshold configured in your moderation rule. For example, if your rule is set to trigger when confidence is "greater than 80%" and your webhook returns `0.95`, the rule action will be executed.
</Note>
2 changes: 1 addition & 1 deletion sdk/android/calling-overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Calling is a core feature of CometChat.

v2.4+ onwards, Voice & Video Calling functionality has been moved to a separate library. Please add the following dependency to your app-level `build.gradle` file in case you plan on using the Voice and Video Calling feature.

`dependencies { implementation 'com.cometchat:calls-sdk-android:4.1.0 }`
`dependencies { implementation 'com.cometchat:calls-sdk-android:4.3.1 }`

</Note>

Expand Down
2 changes: 1 addition & 1 deletion sdk/android/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ dependencies {

<Note>

v2.4+ onwards, Voice & Video Calling functionality has been moved to a separate library. In case you plan to use the calling feature, please add the Calling dependency `implementation 'com.cometchat:calls-sdk-android:4.1.0'` in the dependencies section of the app-level `build.gradle` file.
v2.4+ onwards, Voice & Video Calling functionality has been moved to a separate library. In case you plan to use the calling feature, please add the Calling dependency `implementation 'com.cometchat:calls-sdk-android:4.3.1'` in the dependencies section of the app-level `build.gradle` file.

</Note>

Expand Down
2 changes: 1 addition & 1 deletion sdk/android/setup-calling.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ Then, add CometChatCalls to the app level `build.gradle` file in the `dependenci
<Tab title="Groovy">
```java
dependencies {
implementation 'com.cometchat:calls-sdk-android:4.1.0'
implementation 'com.cometchat:calls-sdk-android:4.3.1'
}
```

Expand Down
2 changes: 1 addition & 1 deletion sdk/android/setup.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ dependencies {

<Note>

In case you plan to use the calling feature, please add the Calling dependency `implementation 'com.cometchat:calls-sdk-android:4.1.0'` in the dependencies section of the app-level `build.gradle` file.
In case you plan to use the calling feature, please add the Calling dependency `implementation 'com.cometchat:calls-sdk-android:4.3.1'` in the dependencies section of the app-level `build.gradle` file.

</Note>

Expand Down
2 changes: 1 addition & 1 deletion ui-kit/android/getting-started.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ dependencies {
implementation 'com.cometchat:chat-uikit-android:5.1.0'

// (Optional) Include this if your app uses voice/video calling features
implementation 'com.cometchat:calls-sdk-android:4.1.0'
implementation 'com.cometchat:calls-sdk-android:4.3.1'
}
```

Expand Down
Loading