Skip to content

feat: post analytics cron#3943

Merged
rebelchris merged 6 commits into
mainfrom
feat-post-analytics-cron
Jun 15, 2026
Merged

feat: post analytics cron#3943
rebelchris merged 6 commits into
mainfrom
feat-post-analytics-cron

Conversation

@rebelchris

Copy link
Copy Markdown
Contributor

No description provided.

@rebelchris rebelchris requested a review from capJavert June 15, 2026 11:53
@pulumi

pulumi Bot commented Jun 15, 2026

Copy link
Copy Markdown

🍹 The Update (preview) for dailydotdev/api/prod (at 7911c16) was successful.

✨ Neo Code Review

This change replaces real-time CDC-driven achievement tracking for post analytics with a new every-15-minute cron job, backed by a new DB index on `post_analytics.updatedAt`. The main risks are a brief Debezium downtime during its forced replacement and a potential share lock on `post_analytics` during index creation. Overall: 🟡 Moderate Risk.

This PR migrates post analytics achievement processing from a real-time CDC (Change Data Capture) event handler to a scheduled polling cron job running every 15 minutes. The onPostAnalyticsChange handler is removed from the CDC worker (primary.ts), and a new post-analytics-achievements cron is introduced that queries post_analytics rows updated since the last run, then calls syncUsersRetroactiveAchievements in batches.

Three supporting changes accompany this:

  1. A DB migration adds IDX_post_analytics_updatedAt on post_analytics.updatedAt — required for the cron's time-windowed query to perform efficiently.
  2. syncUsersRetroactiveAchievements gains an optional eventTypes filter so the cron can scope processing to analytics-related achievement types only.
  3. The Debezium table.include.list gains public.opportunity_user and public.user_marketing_cta, causing the vpc-native-debezium-props Secret to be replaced and the Debezium Deployment to be replaced (it mounts this secret via an init container).

🟡 Warningvpc-native-debezium-deployment is being replaced (delete + create with Recreate strategy). Debezium will have a brief gap in CDC event processing during the restart. Any post_analytics CDC events that were in-flight or queued during the window are now irrelevant for achievements (since that handler is removed), but other CDC-driven features relying on Debezium will experience a short interruption.

🟡 Warning — There is a transition gap to consider: once the CDC handler is removed and before the new cron first runs, post_analytics changes that occur will only be picked up on the next cron tick (up to 15 minutes later). Achievement progress during that window will be delayed but not lost, since the cron uses updatedAt to look back.

🔵 Info — The DB migration job (vpc-native-api-db-migration-2c752a61) will create the IDX_post_analytics_updatedAt index. Building this index on a potentially large post_analytics table could take time; CREATE INDEX IF NOT EXISTS (non-concurrent) will hold a share lock during creation. If this table is large, consider whether a CONCURRENTLY build would be safer.

Resource Changes

    Name                                                       Type                           Operation
-   vpc-native-api-db-migration-4698e453                       kubernetes:batch/v1:Job        delete
~   vpc-native-update-trending-cron                            kubernetes:batch/v1:CronJob    update
~   vpc-native-update-highlighted-views-cron                   kubernetes:batch/v1:CronJob    update
~   vpc-native-update-achievement-rarity-cron                  kubernetes:batch/v1:CronJob    update
~   vpc-native-squad-posts-analytics-refresh-cron              kubernetes:batch/v1:CronJob    update
~   vpc-native-user-profile-updated-sync-cron                  kubernetes:batch/v1:CronJob    update
+   vpc-native-post-analytics-achievements-cron                kubernetes:batch/v1:CronJob    create
~   vpc-native-worker-job-deployment                           kubernetes:apps/v1:Deployment  update
~   vpc-native-update-tags-str-cron                            kubernetes:batch/v1:CronJob    update
~   vpc-native-channel-highlights-cron                         kubernetes:batch/v1:CronJob    update
~   vpc-native-personalized-digest-cron                        kubernetes:batch/v1:CronJob    update
-   vpc-native-api-clickhouse-migration-4698e453               kubernetes:batch/v1:Job        delete
~   vpc-native-update-source-public-threshold-cron             kubernetes:batch/v1:CronJob    update
+   vpc-native-api-clickhouse-migration-2c752a61               kubernetes:batch/v1:Job        create
~   vpc-native-daily-digest-cron                               kubernetes:batch/v1:CronJob    update
~   vpc-native-temporal-deployment                             kubernetes:apps/v1:Deployment  update
~   vpc-native-clean-zombie-images-cron                        kubernetes:batch/v1:CronJob    update
~   vpc-native-update-views-cron                               kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-stale-user-transactions-cron              kubernetes:batch/v1:CronJob    update
~   vpc-native-generic-referral-reminder-cron                  kubernetes:batch/v1:CronJob    update
~   vpc-native-post-lifecycle-state-clickhouse-cron            kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-channel-highlights-cron                   kubernetes:batch/v1:CronJob    update
~   vpc-native-expire-super-agent-trial-cron                   kubernetes:batch/v1:CronJob    update
~   vpc-native-post-analytics-clickhouse-cron                  kubernetes:batch/v1:CronJob    update
~   vpc-native-materialize-yearly-best-post-archives-cron      kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-expired-better-auth-sessions-cron         kubernetes:batch/v1:CronJob    update
~   vpc-native-generate-search-invites-cron                    kubernetes:batch/v1:CronJob    update
~   vpc-native-calculate-top-readers-cron                      kubernetes:batch/v1:CronJob    update
~   vpc-native-validate-active-users-cron                      kubernetes:batch/v1:CronJob    update
~   vpc-native-channel-digests-cron                            kubernetes:batch/v1:CronJob    update
+-  vpc-native-debezium-props                                  kubernetes:core/v1:Secret      create-replacement
~   vpc-native-update-tag-materialized-views-cron              kubernetes:batch/v1:CronJob    update
~   vpc-native-user-profile-analytics-history-clickhouse-cron  kubernetes:batch/v1:CronJob    update
~   vpc-native-hourly-notification-cron                        kubernetes:batch/v1:CronJob    update
~   vpc-native-rotate-weekly-quests-cron                       kubernetes:batch/v1:CronJob    update
~   vpc-native-user-profile-analytics-clickhouse-cron          kubernetes:batch/v1:CronJob    update
~   vpc-native-deployment                                      kubernetes:apps/v1:Deployment  update
~   vpc-native-user-posts-analytics-refresh-cron               kubernetes:batch/v1:CronJob    update
~   vpc-native-private-deployment                              kubernetes:apps/v1:Deployment  update
~   vpc-native-sync-subscription-with-cio-cron                 kubernetes:batch/v1:CronJob    update
~   vpc-native-post-analytics-history-day-clickhouse-cron      kubernetes:batch/v1:CronJob    update
... and 16 other changes

Comment thread src/cron/postAnalyticsAchievements.ts Outdated
@rebelchris rebelchris requested a review from capJavert June 15, 2026 12:20
@rebelchris rebelchris merged commit c596a3d into main Jun 15, 2026
10 checks passed
@rebelchris rebelchris deleted the feat-post-analytics-cron branch June 15, 2026 13:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants