feat: committees implementation [CM-1066] by mbani01 · Pull Request #3995 · linuxfoundation/crowd.dev

mbani01 · 2026-04-02T13:58:00Z

This pull request introduces support for "committees" as a new platform throughout the data pipeline, including database, integrations, and attribute configuration. The main changes add new activity types for committee membership, implement the data extraction and transformation logic for committee events, and update the type system and configuration to recognize "committees" as a first-class source.

Support for "committees" platform and data pipeline:

Added two new activity types, added-to-committee and removed-from-committee, to the activityTypes table for tracking committee membership events.
Introduced a new source query and transformer for the "committees" platform in the Snowflake connectors, enabling extraction and transformation of committee membership data. [1] [2]
Registered the committees source and transformer in the integrations index, and defined its data source name. [1] [2] [3]

Type system and configuration updates:

Added "committees" to the PlatformType, OrganizationSource, and OrganizationAttributeSource enums, and included it in the organization attribute source priority list, ensuring proper handling and prioritization. [1] [2] [3] [4]
Defined new types and scoring grid for committee activities in the integrations library, and re-exported them for usage across the codebase. [1] [2]

Note

Medium Risk
Introduces a new end-to-end ingestion path (Snowflake query + transformer) and new DB activity types, which can affect export volumes and downstream activity/organization attribution if the join/dedup logic is wrong.

Overview
Adds Committees as a first-class platform in the data pipeline.

Creates new activity types (added-to-committee, removed-from-committee) via a DB migration and introduces shared integration types/scoring for these events.

Implements a new Snowflake connector source query (with incremental/segment-aware export logic and non-prod scoping) plus a transformer that emits committee membership activities and optional organization domain identities, and registers this new data source in the Snowflake platform registry.

Updates org attribution plumbing by adding COMMITTEES to PlatformType, OrganizationSource, OrganizationAttributeSource, and the org attribute source priority list.

^{Written by Cursor Bugbot for commit b3da22c. This will update automatically on new commits. Configure here.}

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

github-actions · 2026-04-02T13:58:17Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2026-04-02T13:58:32Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2026-04-02T13:59:54Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

services/apps/snowflake_connectors/src/integrations/committees/committees/transformer.ts

Copilot

Pull request overview

Adds “committees” as a first-class platform across the pipeline (types/config, Snowflake connector extraction+transform, and DB activity type registration) so committee membership events can be exported and represented as activities.

Changes:

Added committees to platform/org source enums and organization attribute source priority.
Implemented Snowflake connector source query + transformer for committee membership events.
Added DB migration to register new committee membership activity types.

Reviewed changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
services/libs/types/src/enums/platforms.ts	Adds `PlatformType.COMMITTEES`.
services/libs/types/src/enums/organizations.ts	Adds `COMMITTEES` to org source enums.
services/libs/integrations/src/integrations/index.ts	Re-exports committees integration types.
services/libs/integrations/src/integrations/committees/types.ts	Defines committee activity types + scoring grid.
services/libs/data-access-layer/src/organizations/attributesConfig.ts	Adds committees to org attribute source priority list.
services/apps/snowflake_connectors/src/integrations/types.ts	Adds committees datasource name.
services/apps/snowflake_connectors/src/integrations/index.ts	Registers committees platform and datasource.
services/apps/snowflake_connectors/src/integrations/committees/committees/buildSourceQuery.ts	Adds Snowflake query for committee membership export.
services/apps/snowflake_connectors/src/integrations/committees/committees/transformer.ts	Transforms exported rows into committee membership activities.
backend/src/database/migrations/V1775064222__addCommitteesActivityTypes.sql	Inserts committees activity types into `activityTypes`.
backend/src/database/migrations/U1775064222__addCommitteesActivityTypes.sql	Undo migration placeholder (empty).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-02T14:04:55Z

services/apps/snowflake_connectors/src/integrations/committees/committees/transformer.ts

+    const activity: IActivityData = {
+      type,
+      platform: PlatformType.COMMITTEES,
+      timestamp: (row.LASTMODIFIEDDATE as string | null) || null,
+      score: COMMITTEES_GRID[type].score,
+      sourceId: committeeId,
+      sourceParentId: null,
+      member: {


activity.sourceId is set to committeeId (COMMITTEE_ID). That value is shared by many rows/members, so activities will collide on the unique key (tenant/platform/type/sourceId/segmentId) and later rows can overwrite or fail to insert. Use a per-membership unique identifier like row.SFID for sourceId (and keep COMMITTEE_ID as an attribute and/or sourceParentId).

Same as above

Copilot · 2026-04-02T14:04:56Z

services/apps/snowflake_connectors/src/integrations/committees/committees/transformer.ts

+      return null
+    }
+
+    const committeeId = (row.COMMITTEE_ID as string).trim()


committeeId is computed via (row.COMMITTEE_ID as string).trim() without null/undefined protection. If COMMITTEE_ID is missing or not a string for any row, this will throw and safeTransformRow will skip the row. Consider using optional chaining + explicit skip/log when COMMITTEE_ID is not present.

Suggested change

const committeeId = (row.COMMITTEE_ID as string).trim()

const rawCommitteeId = row.COMMITTEE_ID

const committeeId =

typeof rawCommitteeId === 'string' && rawCommitteeId.trim().length > 0

? rawCommitteeId.trim()

: null

if (!committeeId) {

log.warn(

{ sfid: row.SFID, rawCommitteeId: row.COMMITTEE_ID, email },

'Skipping row: missing or invalid committeeId',

)

return null

}

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

github-actions · 2026-04-02T14:47:29Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

CLAassistant · 2026-04-02T14:51:09Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

joanagmaia · 2026-04-02T15:56:52Z

services/apps/snowflake_connectors/src/integrations/committees/committees/transformer.ts

+      identities.push({
+        platform: PlatformType.COMMITTEES,
+        value: email,
+        type: MemberIdentityType.USERNAME,
+        verified: true,
+        verifiedBy: PlatformType.COMMITTEES,


We should also add type email on this case no?

Correct!
I realized the logic is almost the same across SF connectors, I'm moving to a unified method for identity building to avoid repetition and inconsistencies.

services/apps/snowflake_connectors/src/integrations/committees/committees/transformer.ts

joanagmaia · 2026-04-02T16:01:45Z

services/apps/snowflake_connectors/src/integrations/committees/committees/transformer.ts

+    const activity: IActivityData = {
+      type,
+      platform: PlatformType.COMMITTEES,
+      timestamp: (row.LASTMODIFIEDDATE as string | null) || null,
+      score: COMMITTEES_GRID[type].score,
+      sourceId: committeeId,
+      sourceParentId: null,
+      member: {


Same as above

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

services/apps/snowflake_connectors/src/integrations/committees/committees/transformer.ts

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

cursor

Cursor Bugbot has reviewed your changes and found 3 potential issues.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

cursor · 2026-04-03T12:15:36Z

services/apps/snowflake_connectors/src/integrations/committees/committees/transformer.ts

+      platform: PlatformType.COMMITTEES,
+      timestamp: activityTimestamp,
+      score: COMMITTEES_GRID[type].score,
+      sourceId: `${committeeId}-${row.SFID}`,


Same sourceId for add and remove causes overwrites

High Severity

The activity sourceId is ${committeeId}-${row.SFID}, which is identical for both "added-to-committee" and "removed-from-committee" events on the same membership record. When a record transitions from active to soft-deleted (FIVETRAN_DELETED flips to true), the removal activity will share the same sourceId as the original addition activity, causing the downstream pipeline to overwrite the "added" event with the "removed" event. The type (or another discriminator) needs to be part of the sourceId to produce distinct activities.

Additional Locations (1)

services/apps/snowflake_connectors/src/integrations/committees/committees/transformer.ts#L39-L42

cursor · 2026-04-03T12:15:36Z

services/libs/integrations/src/integrations/committees/types.ts

+
+export const COMMITTEES_GRID: Record<CommitteesActivityType, IActivityScoringGrid> = {
+  [CommitteesActivityType.ADDED_TO_COMMITTEE]: { score: 1 },
+  [CommitteesActivityType.REMOVED_FROM_COMMITTEE]: { score: 1 },


Removal activity score inconsistent with codebase pattern

Medium Severity

REMOVED_FROM_COMMITTEE has score: 1, which is inconsistent with the established codebase scoring convention. Analogous removal activities use negative scores: UNSTAR is -2 (reversal of STAR at 2), and MEMBER_LEAVE is -2 (reversal of MEMBER_JOIN at 2). A committee removal with a positive score would incorrectly boost member engagement metrics.

cursor · 2026-04-03T12:15:36Z

services/apps/snowflake_connectors/src/integrations/committees/committees/buildSourceQuery.ts

+
+  -- Updated committee memberships since last export
+  ${select}
+    AND c.LASTMODIFIEDDATE > '${sinceTimestamp}'


Incremental export misses soft-deleted records for removal events

High Severity

The incremental filter uses only c.LASTMODIFIEDDATE > sinceTimestamp to find changed records, but the transformer relies on _FIVETRAN_DELETED to generate removed-from-committee activities. When Fivetran soft-deletes a record, it sets _FIVETRAN_DELETED and updates _FIVETRAN_SYNCED, but LASTMODIFIEDDATE (a Salesforce source column) may retain its original value. This means newly deleted records won't match the incremental filter, and removal activities will never be generated in incremental exports. No other connector in the codebase uses _FIVETRAN_DELETED, so this gap is unique to committees. The filter needs to also check _FIVETRAN_SYNCED for deleted records.

feat: committees implementation (WIP)

5f5e903

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

mbani01 self-assigned this Apr 2, 2026

Copilot AI review requested due to automatic review settings April 2, 2026 13:58

Copilot started reviewing on behalf of mbani01 April 2, 2026 13:58 View session

cursor bot reviewed Apr 2, 2026

View reviewed changes

services/apps/snowflake_connectors/src/integrations/committees/committees/transformer.ts Outdated Show resolved Hide resolved

Copilot AI reviewed Apr 2, 2026

View reviewed changes

fix: activity timestamp and use cncf for staging tests

bdc9da9

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

mbani01 changed the title ~~feat: committees implementation~~ feat: committees implementation [CM-1066] Apr 2, 2026

mbani01 requested a review from joanagmaia April 2, 2026 14:48

joanagmaia reviewed Apr 2, 2026

View reviewed changes

fix: add channel & use unique sourceId

2f5b708

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

mbani01 force-pushed the feat/snowflake_committees_implementation branch from cb61fdd to 2f5b708 Compare April 3, 2026 11:50

Merge branch 'main' into feat/snowflake_committees_implementation

5f1703e

cursor bot reviewed Apr 3, 2026

View reviewed changes

services/apps/snowflake_connectors/src/integrations/committees/committees/transformer.ts Outdated Show resolved Hide resolved

mbani01 added 3 commits April 3, 2026 13:05

fix: member identity

6debc97

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

chore: pass explicity null platform username

510704b

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

fix: lint

b3da22c

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

cursor bot reviewed Apr 3, 2026

View reviewed changes

-    const committeeId = (row.COMMITTEE_ID as string).trim()
+    const rawCommitteeId = row.COMMITTEE_ID
+    const committeeId =
+      typeof rawCommitteeId === 'string' && rawCommitteeId.trim().length > 0
+        ? rawCommitteeId.trim()
+        : null
+    if (!committeeId) {
+      log.warn(
+        { sfid: row.SFID, rawCommitteeId: row.COMMITTEE_ID, email },
+        'Skipping row: missing or invalid committeeId',
+      )
+      return null
+    }

Conversation

mbani01 commented Apr 2, 2026 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

joanagmaia Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

CLAassistant commented Apr 2, 2026

Uh oh!

joanagmaia Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

mbani01 Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joanagmaia Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor bot Apr 3, 2026

Choose a reason for hiding this comment

Same sourceId for add and remove causes overwrites

Uh oh!

cursor bot Apr 3, 2026

Choose a reason for hiding this comment

Removal activity score inconsistent with codebase pattern

Uh oh!

cursor bot Apr 3, 2026

Choose a reason for hiding this comment

Incremental export misses soft-deleted records for removal events

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mbani01 commented Apr 2, 2026 •

edited by cursor bot

Loading