v0.7.8: security hardening, code hygiene, MSFT oauth provider, new atlassian assets and google calendar tools

.agents/skills/add-connector/SKILL.md

@@ -12,8 +12,8 @@ You are an expert at adding knowledge base connectors to Sim. A connector syncs
 When the user asks you to create a connector:
 1. Use Context7 or WebFetch to read the service's API documentation
 2. Determine the auth mode: **OAuth** (if Sim already has an OAuth provider for the service) or **API key** (if the service uses API key / Bearer token auth)
-3. Create the connector directory and config
-4. Register it in the connector registry
+3. Create the connector directory: a client-safe `meta.ts` (declarative metadata) plus the runtime module that spreads it
+4. Register it in BOTH the server registry and the client-safe meta registry
 
 ## Hard Rule: No Guessed Response Or Document Schemas
 
@@ -32,13 +32,19 @@ If the source schema is unknown, do one of these instead:
 
 ## Directory Structure
 
+Each connector is split into a client-safe metadata file and a server-only runtime file. This mirrors the `XBlockMeta` / `BLOCK_META_REGISTRY` split in `apps/sim/blocks` — client components (the knowledge UI) only need the metadata (icon, name, auth, config fields), so the runtime functions (which pull server-only helpers like `input-validation.server` → `undici` → `node:net`) must stay out of the client bundle.
+
 Create files in `apps/sim/connectors/{service}/`:
 ```
 connectors/{service}/
-├── index.ts          # Barrel export
-└── {service}.ts      # ConnectorConfig definition
+├── index.ts          # Barrel export (re-exports the runtime connector)
+├── meta.ts           # ConnectorMeta — client-safe declarative metadata
+└── {service}.ts      # ConnectorConfig — spreads the meta + adds runtime functions
 ```
 
+- `meta.ts` exports `{service}ConnectorMeta: ConnectorMeta`. It imports ONLY the icon from `@/components/icons`, `import type { ConnectorMeta } from '@/connectors/types'`, and any pure-data constants. It must NEVER import server/runtime code.
+- `{service}.ts` exports `{service}Connector: ConnectorConfig`. It imports the meta via `import { {service}ConnectorMeta } from '@/connectors/{service}/meta'`, spreads it as the first property, and holds the runtime functions (which may import server-only helpers like `@/lib/knowledge/documents/utils`).
+
 ## Authentication
 
 Connectors use a discriminated union for auth config (`ConnectorAuthConfig` in `connectors/types.ts`):
@@ -55,19 +61,17 @@ For services with existing OAuth providers in `apps/sim/lib/oauth/types.ts`. The
 ### API key mode
 For services that use API key / Bearer token auth. The modal shows a password input with the configured `label` and `placeholder`. The API key is encrypted at rest using AES-256-GCM and stored in a dedicated `encryptedApiKey` column on the connector record. The sync engine decrypts it automatically — connectors receive the raw access token in `listDocuments`, `getDocument`, and `validateConfig`.
 
-## ConnectorConfig Structure
+## Connector Structure (meta.ts + runtime)
+
+The declarative metadata lives in `meta.ts` (`ConnectorMeta`). The runtime functions live in `{service}.ts` (`ConnectorConfig`), which spreads the meta as its first property.
 
-### OAuth connector example
+### `meta.ts` — client-safe metadata
 
 ```typescript
-import { createLogger } from '@sim/logger'
 import { {Service}Icon } from '@/components/icons'
-import { fetchWithRetry } from '@/lib/knowledge/documents/utils'
-import type { ConnectorConfig, ExternalDocument, ExternalDocumentList } from '@/connectors/types'
-
-const logger = createLogger('{Service}Connector')
+import type { ConnectorMeta } from '@/connectors/types'
 
-export const {service}Connector: ConnectorConfig = {
+export const {service}ConnectorMeta: ConnectorMeta = {
   id: '{service}',
   name: '{Service}',
   description: 'Sync documents from {Service} into your knowledge base',
@@ -85,6 +89,26 @@ export const {service}Connector: ConnectorConfig = {
     // Supports 'short-input' and 'dropdown' types
   ],
 
+  // Optional: tag definitions are metadata too — declare them here
+  // tagDefinitions: [ ... ],
+}
+```
+
+Keep `meta.ts` free of any server/runtime import. Only the icon, the `ConnectorMeta` type, and pure-data constants belong here.
+
+### `{service}.ts` — runtime (OAuth example)
+
+```typescript
+import { createLogger } from '@sim/logger'
+import { fetchWithRetry } from '@/lib/knowledge/documents/utils'
+import { {service}ConnectorMeta } from '@/connectors/{service}/meta'
+import type { ConnectorConfig, ExternalDocument, ExternalDocumentList } from '@/connectors/types'
+
+const logger = createLogger('{Service}Connector')
+
+export const {service}Connector: ConnectorConfig = {
+  ...{service}ConnectorMeta,
+
   listDocuments: async (accessToken, sourceConfig, cursor) => {
     // Return metadata stubs with contentDeferred: true (if per-doc content fetch needed)
     // Or full documents with content (if list API returns content inline)
@@ -111,8 +135,11 @@ Only map fields in `listDocuments`, `getDocument`, `validateConfig`, and `mapTag
 
 ### API key connector example
 
+The split is identical — `auth` lives in `meta.ts`, runtime functions in `{service}.ts`.
+
 ```typescript
-export const {service}Connector: ConnectorConfig = {
+// meta.ts
+export const {service}ConnectorMeta: ConnectorMeta = {
   id: '{service}',
   name: '{Service}',
   description: 'Sync documents from {Service} into your knowledge base',
@@ -126,6 +153,11 @@ export const {service}Connector: ConnectorConfig = {
   },
 
   configFields: [ /* ... */ ],
+}
+
+// {service}.ts
+export const {service}Connector: ConnectorConfig = {
+  ...{service}ConnectorMeta,
   listDocuments: async (accessToken, sourceConfig, cursor) => { /* ... */ },
   getDocument: async (accessToken, sourceConfig, externalId) => { /* ... */ },
   validateConfig: async (accessToken, sourceConfig) => { /* ... */ },
@@ -499,7 +531,9 @@ If the service already has an icon in `apps/sim/components/icons.tsx` (from a to
 
 ## Registering
 
-Add one line to `apps/sim/connectors/registry.ts`:
+Register in BOTH registries, keeping the same alphabetical-by-id ordering in each.
+
+1. **Server registry** — `apps/sim/connectors/registry.server.ts` (server-only full registry; holds full connectors with runtime functions, imported by the sync engine and knowledge API routes):
 
 ```typescript
 import { {service}Connector } from '@/connectors/{service}'
@@ -510,6 +544,19 @@ export const CONNECTOR_REGISTRY: ConnectorRegistry = {
 }
 ```
 
+2. **Client-safe meta registry** — `apps/sim/connectors/registry.ts` (imports each connector's `meta.ts` only, so client components can use it without pulling server-only code; the metadata counterpart to `BLOCK_META_REGISTRY`):
+
+```typescript
+import { {service}ConnectorMeta } from '@/connectors/{service}/meta'
+
+export const CONNECTOR_META_REGISTRY: ConnectorMetaRegistry = {
+  // ... existing connector metas ...
+  {service}: {service}ConnectorMeta,
+}
+```
+
+`registry.ts` exports `CONNECTOR_META_REGISTRY: ConnectorMetaRegistry` plus the helpers `getConnectorMeta(id)` and `getAllConnectorMeta()`, importing each `@/connectors/{service}/meta` directly — never the runtime module. `registry.server.ts` exports `CONNECTOR_REGISTRY: ConnectorRegistry`.
+
 ## Reference Implementations
 
 - **OAuth + contentDeferred**: `apps/sim/connectors/google-drive/google-drive.ts` — file download with metadata-based hash, `orderBy` for deterministic pagination
@@ -520,7 +567,8 @@ export const CONNECTOR_REGISTRY: ConnectorRegistry = {
 
 ## Checklist
 
-- [ ] Created `connectors/{service}/{service}.ts` with full ConnectorConfig
+- [ ] Created `connectors/{service}/meta.ts` with `{service}ConnectorMeta: ConnectorMeta` (icon, name, auth, configFields, tagDefinitions) — no server/runtime imports
+- [ ] Created `connectors/{service}/{service}.ts` with `{service}Connector: ConnectorConfig` spreading the meta + runtime functions
 - [ ] Created `connectors/{service}/index.ts` barrel export
 - [ ] **Auth configured correctly:**
   - OAuth: `auth.provider` matches an existing `OAuthService` in `lib/oauth/types.ts`
@@ -542,4 +590,5 @@ export const CONNECTOR_REGISTRY: ConnectorRegistry = {
 - [ ] All external API calls use `fetchWithRetry` (not raw `fetch`)
 - [ ] All optional config fields validated in `validateConfig`
 - [ ] Icon exists in `components/icons.tsx` (or asked user to provide SVG)
-- [ ] Registered in `connectors/registry.ts`
+- [ ] Registered the full connector in `connectors/registry.server.ts`
+- [ ] Registered the meta in `connectors/registry.ts` (same alphabetical-by-id ordering as registry.server.ts)

.agents/skills/db-migrate/SKILL.md

@@ -0,0 +1,89 @@
+---
+name: db-migrate
+description: Author or review a Drizzle DB migration for zero-downtime safety — expand/contract phasing, backward-compatibility with the deployed app version, and writing the `-- migration-safe` acknowledgment the check:migrations lint requires. Use when adding/editing files under `packages/db/migrations/` or changing `packages/db/schema.ts`.
+---
+
+# DB Migrate Skill
+
+You make schema changes that survive a deploy without downtime. The `check:migrations` lint (`scripts/check-migrations-safety.ts`) is the deterministic gate; you are the judgment that decides whether a flagged change is actually safe and writes the annotation that satisfies it.
+
+## The window (why this matters)
+
+A deploy runs the migration, then rolls out the new app image via blue/green. The two are **not atomic and cannot be** — during cutover the old task set keeps serving against the **already-migrated** schema. So:
+
+> Every migration must be backward-compatible with the app version that is *already deployed*.
+
+If a migration drops a column the old code still reads, renames one, or adds a `NOT NULL` the old inserts don't populate, the old code throws until traffic fully shifts — the downtime we're guarding against. You can't fix this by reordering the pipeline; the only fix is discipline.
+
+## Expand / contract
+
+Split every breaking change across **two deploys**:
+
+1. **Expand** (this PR): additive, backward-compatible schema + code that tolerates *both* the old and new shape.
+2. **Contract** (a later PR, after expand is fully deployed): remove the old thing, now that nothing reads it.
+
+Never put expand and contract in the same PR. If this PR both removes the code that used a column *and* drops the column, the old code is still live during cutover — split it.
+
+### Per-operation playbook
+
+| You want to | Do (deploy 1 = expand) | Do (deploy 2 = contract) |
+|---|---|---|
+| Add a required column | `ADD COLUMN` nullable or `DEFAULT`; code writes it | backfill, then `SET NOT NULL` |
+| Rename a column/table | add the new name; code dual-writes / reads new-then-old | drop the old name |
+| Drop a column/table | stop all reads/writes in code; ship it | `DROP` (annotate) |
+| Change a column type | add a new column of the new type; dual-write | backfill, swap reads, drop old |
+| Add FK / CHECK | `ADD CONSTRAINT ... NOT VALID` | `VALIDATE CONSTRAINT` separately |
+| Index an existing table | `COMMIT;` breakpoint → `SET lock_timeout = 0` → `CREATE INDEX CONCURRENTLY IF NOT EXISTS` (see `packages/db/scripts/migrate.ts`) | — |
+| Drop an index | `COMMIT;` breakpoint → `DROP INDEX CONCURRENTLY` — plain `DROP INDEX` takes ACCESS EXCLUSIVE on the table | — |
+| Backfill data | batched + idempotent `UPDATE` (keyset/`WHERE`, bounded) | — |
+
+A `CREATE INDEX`, `ADD COLUMN`, or `ADD CONSTRAINT` against a table **created in the same migration** is always safe (no rows, no live traffic) — the lint already suppresses those.
+
+## Tracking the contract (don't let it rot)
+
+The contract half is deferred to a later deploy — and that is exactly when it gets forgotten, leaving dead columns, orphaned tables, and `NOT NULL`s that never land. Every deferred contract must become a durable, greppable TODO.
+
+When an expand defers a drop, leave a **`contract-pending`** marker on the legacy column/table in `packages/db/schema.ts` — that is the file you will be editing when you finally do the drop, so the reminder lives where the work happens:
+
+```ts
+// contract-pending(after #5035 is fully deployed): drop once permission-check.ts stops reading it
+workspaceId: text('workspace_id'),
+```
+
+Format: `contract-pending(<precondition>): <what to drop> — <why it's safe once the precondition holds>`. The precondition names the PR/release that removes the last reader and **must be fully deployed** before the contract ships.
+
+- **The TODO list is a grep** — always accurate, never drifts: `grep -rn "contract-pending" packages/db apps/sim`. Run it when starting migration work to see what is owed.
+- For anything with a real owner or schedule, also open a tracking issue and put its number in the marker.
+- **Close the loop in the contract PR:** the contract migration's `-- migration-safe:` annotation references the expand, and you **delete the `contract-pending` marker** in the same PR:
+  ```sql
+  -- migration-safe: contract of #5035 — workspace_id readers removed there, deployed 2026-06-10
+  ALTER TABLE "permission_group" DROP COLUMN "workspace_id";
+  ```
+- An expand merged **without** a marker for the drop it defers, or a contract merged **without** removing its marker, is a bug — flag it in review.
+
+## The judgment the lint can't do
+
+The lint flags risky *shapes*; it cannot know whether a given drop is *safe right now*. For each flagged statement, do the work it can't:
+
+1. **Is the dependency gone?** Grep the app for the table/column: search `apps/sim` and `packages` for the column name, the Drizzle field (camelCase), and the table object. If any live read/write remains, it is **not** safe — fix the code first.
+2. **Did the expand already ship?** The removal of that read/write must be in a deploy that is *already out*, not this same PR. If it's in this PR, split: land the code change now, do the destructive migration in a follow-up after it deploys.
+3. **Backfills:** confirm the `UPDATE`/`DELETE` is batched (bounded `WHERE`/keyset, not a single whole-table statement), idempotent (safe to replay — a failed migration re-runs unjournaled files from the top), and safe under concurrent writes from the still-live old app.
+
+## Workflow
+
+1. Edit `packages/db/schema.ts`, then `cd packages/db && bunx drizzle-kit generate` to produce the SQL. If this is an expand that defers a drop, leave a `contract-pending` marker on the legacy column (see "Tracking the contract"). If this is the contract, delete the marker it resolves.
+2. Hand-edit the generated SQL where the playbook requires it: `CONCURRENTLY` + `COMMIT;` breakpoint for indexes on existing tables, `NOT VALID` for constraints, batching for backfills.
+3. Run `bun run check:migrations` (base defaults to `origin/staging`).
+   - **Hard errors** (`add-not-null-no-default`, `rename`, `index-not-concurrent`, `constraint-not-valid`, …): rewrite into expand/contract. Do **not** try to annotate them away — the lint won't accept it.
+   - **Annotate tier** (`drop-table`, `drop-column`, `drop-default`, `set-not-null`, `alter-type`, `drop-index`): only after you've confirmed steps 1–3 above, add a comment on the line directly above the statement:
+     ```sql
+     -- migration-safe: `secret` read removed in v0.6.1 (#1234), shipped two deploys ago
+     ALTER TABLE "webhook" DROP COLUMN "secret";
+     ```
+     The reason must be specific and name the PR/version that removed the dependency. An empty reason fails the lint.
+   - **Warnings** (`data-backfill`): non-blocking, but confirm the batching/idempotency before merging.
+4. Verify locally: `cd packages/db && bun run db:migrate` against a dev DB.
+
+## Hard rule
+
+Never annotate a destructive statement just to make the lint pass. The annotation is a claim that you verified the old code no longer depends on it. If you can't make that claim truthfully, the change belongs in a later deploy — tell the user to split it.

.agents/skills/ship/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: ship
-description: Commit, push, and open a PR to staging in one shot
+description: Commit, push, and open a PR to staging in one shot — runs the cleanup pass and, when migrations changed, the db-migrate safety review first
 ---
 
 # Ship Command
@@ -16,12 +16,17 @@ When the user runs `/ship`:
   - Types: `fix`, `feat`, `improvement`, `chore`
   - Scope: short identifier (e.g., `undo-redo`, `api`, `ui`)
   - Keep it concise
-3. **Run pre-ship checks** from the repo root before staging:
+3. **Run the cleanup pass** — only if the diff modifies UI code (any `.tsx` file, or anything under `apps/sim/components/`, `apps/sim/hooks/`, or `apps/sim/stores/`): `/cleanup`
+  - The six code-quality skills (effects, memo, callbacks, state, React Query, emcn) only apply to React code, so skip this step entirely when no UI was touched. When it runs, it applies fixes so they land in this commit.
+4. **Run migration safety** — only if the diff touches `packages/db/migrations/**` or `packages/db/schema.ts`:
+  - Run `/db-migrate` to review the migration for zero-downtime safety (expand/contract phasing, backward-compatibility with the deployed app version).
+  - `bun run check:migrations origin/staging` must pass (staging is the PR base). Do not silence a flagged statement with a `-- migration-safe:` annotation unless `/db-migrate` confirmed the old code no longer depends on it; otherwise split the destructive change into a later deploy.
+5. **Run pre-ship checks** from the repo root before staging:
   - `bun run lint` to fix formatting issues
   - `bun run check:api-validation:strict` to catch boundary contract failures before CI
-4. **Stage and commit** the changes with the generated message
-5. **Push to origin** using the current branch name
-6. **Create a PR** to staging with a description in the user's voice
+6. **Stage and commit** the changes with the generated message
+7. **Push to origin** using the current branch name
+8. **Create a PR** to staging with a description in the user's voice
 
 ## Commit Message Format
 
@@ -77,7 +82,7 @@ gh pr create --base staging --title "COMMIT_MESSAGE" --body "PR_BODY"
 
 - Short, direct bullet points
 - No unnecessary explanation
-- "Tested manually" is acceptable for testing section; include lint and boundary validation results when run
+- "Tested manually" is acceptable for testing section; include lint, boundary validation, and (when migrations changed) `check:migrations` results when run
 - Checkboxes filled in appropriately
 - No screenshots section unless UI changes