Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/moderation-custom-api-create-rule.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
37 changes: 28 additions & 9 deletions moderation/custom/custom-api-overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,19 +21,36 @@ sequenceDiagram
```

1. **User sends a message** in your chat application
2. **CometChat intercepts** the message and calls your webhook endpoint
2. **CometChat intercepts** the message and POSTs it to your webhook URL along with optional conversation context (previous messages as plain text)
3. **Your webhook processes** the message using your custom moderation logic
4. **Your webhook responds** with a decision (violation detected or not)
5. **CometChat applies the action** you configured (block, flag, allow, etc.)
4. **Your webhook responds** with a JSON decision containing `isMatchingCondition`, `confidence`, and optional `reason`
5. **CometChat compares** the returned confidence against your configured threshold and applies the appropriate action (block, flag for review, etc.)

## Getting Started

<Steps>
<Step title="Build Your Moderation Endpoint">
Create a webhook that receives messages and returns moderation decisions. You can use any third-party moderation service or your own AI model.
Create a webhook that accepts POST requests with message data. Your endpoint must return JSON with:
- `isMatchingCondition` (boolean) – whether the content violates rules
- `confidence` (number 0.0-1.0) – confidence score of the decision
- `reason` (string, optional) – explanation for flagging
</Step>
<Step title="Configure Custom API in CometChat">
Set up your webhook URL, authentication, and moderation rules in the CometChat Dashboard.
<Step title="Create a Custom API List">
In the CometChat Dashboard, go to **Moderation** → **Settings** → **List** and create a new list with:
- **Category**: Custom API
- **URL**: Your webhook endpoint
- **Basic Auth** (optional): Username and password for secure authentication
</Step>
<Step title="Configure Advanced Settings">
Under **Moderation** → **Settings** → **Advanced Settings**, configure:
- **Context messages** (0-10): Number of previous messages to include for context-aware moderation
- **Error handling**: Choose to approve or block messages when your API is unavailable
</Step>
<Step title="Create a Moderation Rule">
Go to **Moderation** → **Rules** and create a rule that:
- Selects your Custom API list
- Sets a confidence threshold (e.g., "greater than 80%")
- Defines actions when content is flagged
<Card title="Custom API Configuration" icon="gear" href="/moderation/custom/custom-api">
Step-by-step guide to configure your custom moderation API
</Card>
Expand All @@ -48,7 +65,9 @@ sequenceDiagram

## Key Features

- **Contextual Moderation** – Include previous messages from the conversation for better analysis
- **Error Handling** – Configure fallback behavior when your API is unavailable
- **Flexible Rules** – Apply custom moderation to text or image content
- **Contextual Moderation** – Include up to 10 previous messages from the conversation for context-aware analysis
- **Secure Authentication** – Optional Basic Auth support for webhook security
- **Confidence-Based Decisions** – Set confidence thresholds to fine-tune when actions are triggered
- **Configurable Error Handling** – Choose to approve or block messages when your API is unavailable or returns an error
- **Full Message Details** – Receive complete message objects including sender/receiver info, timestamps, and metadata
- **Real-time Processing** – Moderation decisions are applied before message delivery
178 changes: 95 additions & 83 deletions moderation/custom/custom-api.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,67 +2,115 @@
title: "Custom API Moderation"
---

CometChat allows you to integrate your own moderation logic using a **Custom API**. With this feature, you can define a webhook URL where CometChat will send messages for moderation along with relevant context from the conversation.
CometChat allows you to integrate your own moderation logic using a **Custom API**. With this feature, you can define a webhook URL where CometChat will send messages for moderation along with relevant context from the conversation. This enables you to use your own moderation service, third-party AI moderation APIs, or custom business logic to moderate content.

## Integration
## Integration Steps

### Step 1: Configure Custom API Settings
### Step 1: Create a Custom API List

1. **Login to the CometChat Dashboard**
- Navigate to [CometChat Dashboard](https://app.cometchat.com) and select your app.
First, you need to create a list that defines your Custom API webhook configuration.

2. **Navigate to Moderation Settings**
- Go to **Moderation → Settings** in the left-hand menu.
1. Navigate to the [CometChat Dashboard](https://app.cometchat.com) and select your app.

3. **Open Custom API Settings Tab**
- Click on the **Custom API** tab within the Moderation Settings.
2. Go to **Moderation** → **Settings** from the left-hand menu.

4. **Fill in the Custom API Configuration**
3. Click on the **List** tab.

- **Webhook URL**
- Enter the endpoint URL where CometChat will send messages for moderation.
4. Click **Add List** to create a new list.

- **Authentication (Optional)**
- Enable Basic Authentication to secure your webhook endpoint.
- Provide a username and password that CometChat will include in the `Authorization` header.
5. Fill in the required fields:
- **Name**: Enter a descriptive name for your custom moderation list (e.g., `custom-moderation`)
- **ID**: Enter a unique identifier for the list (e.g., `custom-moderation-list`)
- **Category**: Select **Custom API** from the dropdown
- **URL**: Enter the webhook URL where CometChat will POST message data for moderation

- **Set Action on API Error**
- Define how the system should respond if the Custom API is unavailable:
- **Allow message** – Messages are delivered even if moderation fails.
- **Block message** – Messages are blocked when moderation is unavailable.
6. **Enable Basic Auth** (Optional but Recommended):
- Toggle on **Enable Basic Auth** for added security
- Enter your **Basic Auth Username**
- Enter your **Basic Auth Password**
- CometChat will include these credentials in the `Authorization` header when calling your webhook

- **Set Context Window**
- Specify the number of previous messages in a conversation to include for context (0-10).
7. Click **Save** to create the list.

5. **Click Save Settings**
### Step 2: Configure Advanced Settings

### Step 2: Create a Moderation Rule
Next, configure the context and error handling settings for your Custom API.

1. Navigate to **Moderation → Rules**.
2. Click **"Create New Rule"**.
3. Select **Custom API** as the moderation type.
4. Choose the rule type:
- **Text Contains** – For text message moderation
- **Image Contains** – For image message moderation
5. Configure the action to take when content is flagged (block, flag for review, etc.).
6. Save the rule.
1. Navigate to **Moderation** → **Settings**.

2. Click on the **Advanced Settings** tab.

3. Scroll down to the **Custom API** section.

4. Configure the following options:

- **The number of messages sent to Custom API for context**:
- Set the number of previous messages to include as context (0-10)
- This enables context-aware moderation by providing conversation history
- Set to `0` if you don't need context and only want to moderate individual messages

- **On Custom API Error**:
- **Approve Message** (Default): Messages are automatically approved if your webhook is unavailable or returns an error
- **Block Message**: Messages are blocked when your webhook is inaccessible, ensuring no unmoderated content passes through

5. Click **Save** to apply the settings.

<Frame>
<img src="/images/moderation-custom-api-advanced-settings.png" alt="Custom API Advanced Settings" />
</Frame>

### Step 3: Create a Moderation Rule

Now, map your Custom API list to a moderation rule to activate the moderation.

1. Navigate to **Moderation** → **Rules**.

2. Click **Create New Rule**.

3. Fill in the mandatory fields:
- **Name**: Enter a name for the rule (e.g., `test-custom-api`)
- **ID**: Enter a unique identifier (e.g., `testcustomapi`)
- **Description**: Optionally describe what this rule does

4. Configure the **Conditions**:
- Select the content type to moderate (e.g., **Text**)
- Select **Contains**
- Select **Custom API**
- Choose the Custom API list you created in Step 1 from the dropdown
- Set the **confidence** threshold (e.g., `greater than 80%`)
- Higher percentages result in stricter matching
- Lower percentages allow more content to be flagged

5. Configure the **Actions** to perform when content is flagged (block, flag for review, etc.).

6. Click **Save** to create the rule.

7. **Enable the rule** by toggling it on.

<Frame>
<img src="/images/moderation-custom-api-create-rule.png" alt="Create Custom API Rule"/>
</Frame>

Once enabled, all messages matching your filter criteria will be sent to your webhook URL for moderation.

---

## Webhook Request

### Headers
When a message is sent, CometChat will POST the message data to your configured webhook URL.

When CometChat calls your webhook, it includes the following headers:
### Headers

| Header | Description |
|--------|-------------|
| `Content-Type` | `application/json` |
| `Authorization` | Basic auth credentials (if configured) |

### Payload
### Payload Structure

The payload includes:
- The latest message (the one just sent) — provided in full detail (entire message object)
- The previous messages — provided as plain text only, for context (based on the context window setting)
- **Context messages**: Previous messages in the conversation (as plain text) based on your context window setting
- **Latest message**: The message being moderated (full message object with all details)

```json
{
Expand Down Expand Up @@ -126,6 +174,8 @@ The payload includes:
}
```

---

## Webhook Response

Your webhook must return a JSON response indicating the moderation decision.
Expand All @@ -136,7 +186,7 @@ Your webhook must return a JSON response indicating the moderation decision.
{
"isMatchingCondition": true,
"confidence": 0.95,
"reason": "Contains hate speech"
"reason": "Contains prohibited content"
}
```

Expand All @@ -152,50 +202,12 @@ Your webhook must return a JSON response indicating the moderation decision.

### Response Fields

| Field | Type | Description |
|-------|------|-------------|
| `isMatchingCondition` | boolean | `true` if the message violates the rule, `false` if safe |
| `confidence` | number | Confidence score of the decision (0.0 - 1.0) |
| `reason` | string | Reason for flagging (can be empty for safe content) |

## Example Webhook Implementation

Here's a simple Node.js/Express example:

```javascript
const express = require('express');
const app = express();

app.use(express.json());

app.post('/moderate', (req, res) => {
const { contextMessages } = req.body;

// Get the latest message (last item in array)
const latestEntry = contextMessages[contextMessages.length - 1];
const senderId = Object.keys(latestEntry)[0];
const messageData = latestEntry[senderId];

// Extract text content
const text = typeof messageData === 'string'
? messageData
: messageData.data?.text || '';

// Your moderation logic here
const isViolation = containsBadContent(text);

res.json({
isMatchingCondition: isViolation,
confidence: 0.95,
reason: isViolation ? 'Content policy violation' : ''
});
});

function containsBadContent(text) {
// Implement your moderation logic
// Could call OpenAI Moderation API, Perspective API, etc.
return false;
}
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `isMatchingCondition` | boolean | Yes | `true` if the message violates the rule, `false` if safe |
| `confidence` | number | Yes | Confidence score of the decision (0.0 - 1.0). This is compared against the threshold set in your rule conditions |
| `reason` | string | No | Reason for flagging (can be empty for safe content) |

app.listen(3000);
```
<Note>
The `confidence` value returned by your webhook is compared against the confidence threshold configured in your moderation rule. For example, if your rule is set to trigger when confidence is "greater than 80%" and your webhook returns `0.95`, the rule action will be executed.
</Note>