Skip to content

fix: replace manual anomalies with a hampel filter #1997

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
matthewp wants to merge 6 commits into
base: main
Choose a base branch
from
+221 −452
Open

fix: replace manual anomalies with a hampel filter #1997

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a29c7d3
Replace manual anomalies with a hampel filter
matthewp Mar 8, 2026
9cf5d8b
Fix typechecking
matthewp Mar 8, 2026
55388db
Remove i18n keys
matthewp Mar 8, 2026
7dc8c0b
Fix false positives for boundary points and low-volume packages
matthewp Mar 11, 2026
88334f4
Add tweakable Hampel filter sliders to chart UI
matthewp Mar 11, 2026
bfc1bcb
[autofix.ci] apply automated fixes
autofix-ci[bot] Mar 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 43 additions & 62 deletions app/components/Package/TrendsChart.vue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import {
endDateOnlyToUtcMs,
DEFAULT_PREDICTION_POINTS,
} from '~/utils/chart-data-prediction'
import { applyBlocklistCorrection, getAnomaliesForPackages } from '~/utils/download-anomalies'
import { applyHampelCorrection } from '~/utils/download-anomalies'
import { copyAltTextForTrendLineChart, sanitise, loadFile, applyEllipsis } from '~/utils/charts'

import('vue-data-ui/style.css')
Expand Down Expand Up @@ -976,12 +976,10 @@ const effectiveDataSingle = computed<EvolutionData>(() => {
}

if (isDownloadsMetric.value && data.length) {
const pkg = effectivePackageNames.value[0] ?? props.packageName ?? ''
if (settings.value.chartFilter.anomaliesFixed) {
data = applyBlocklistCorrection({
data,
packageName: pkg,
granularity: displayedGranularity.value,
data = applyHampelCorrection(data, {
halfWindow: settings.value.chartFilter.hampelWindow,
threshold: settings.value.chartFilter.hampelThreshold,
})
}
}
Expand Down Expand Up @@ -1026,7 +1024,10 @@ const chartData = computed<{
let data = state.evolutionsByPackage[pkg] ?? []
if (isDownloadsMetric.value && data.length) {
if (settings.value.chartFilter.anomaliesFixed) {
data = applyBlocklistCorrection({ data, packageName: pkg, granularity })
data = applyHampelCorrection(data, {
halfWindow: settings.value.chartFilter.hampelWindow,
threshold: settings.value.chartFilter.hampelThreshold,
})
}
}
const points = extractSeriesPoints(granularity, data)
Expand Down Expand Up @@ -1556,20 +1557,6 @@ const chartConfig = computed<VueUiXyConfig>(() => {
const isDownloadsMetric = computed(() => selectedMetric.value === 'downloads')
const showCorrectionControls = shallowRef(false)

const packageAnomalies = computed(() => getAnomaliesForPackages(effectivePackageNames.value))
const hasAnomalies = computed(() => packageAnomalies.value.length > 0)

function formatAnomalyDate(dateStr: string) {
const [y, m, d] = dateStr.split('-').map(Number)
if (!y || !m || !d) return dateStr
return new Intl.DateTimeFormat(locale.value, {
year: 'numeric',
month: 'short',
day: 'numeric',
timeZone: 'UTC',
}).format(new Date(Date.UTC(y, m - 1, d)))
}

// Trigger data loading when the metric is switched
watch(selectedMetric, value => {
if (!isMounted.value) return
Expand Down Expand Up @@ -1717,69 +1704,63 @@ watch(selectedMetric, value => {
class="accent-[var(--accent-color,var(--fg-subtle))]"
/>
</label>
</div>
<div v-if="showCorrectionControls" class="grid grid-cols-2 sm:flex items-end gap-3">
<label class="flex flex-col gap-1 flex-1">
<span class="text-2xs font-mono text-fg-subtle tracking-wide uppercase">
{{ $t('package.trends.hampel_window') }}
<span class="text-fg-muted">({{ settings.chartFilter.hampelWindow }})</span>
</span>
<input
v-model.number="settings.chartFilter.hampelWindow"
type="range"
min="1"
max="10"
step="1"
class="accent-[var(--accent-color,var(--fg-subtle))]"
/>
</label>
<label class="flex flex-col gap-1 flex-1">
<span class="text-2xs font-mono text-fg-subtle tracking-wide uppercase">
{{ $t('package.trends.hampel_threshold') }}
<span class="text-fg-muted">({{ settings.chartFilter.hampelThreshold }})</span>
</span>
<input
v-model.number="settings.chartFilter.hampelThreshold"
type="range"
min="1"
max="10"
step="0.5"
class="accent-[var(--accent-color,var(--fg-subtle))]"
/>
</label>
<div class="flex flex-col gap-1 shrink-0">
<span
class="text-2xs font-mono text-fg-subtle tracking-wide uppercase flex items-center justify-between"
>
{{ $t('package.trends.known_anomalies') }}
<TooltipApp interactive :to="inModal ? '#chart-modal' : undefined">
<TooltipApp :to="inModal ? '#chart-modal' : undefined">
<button
type="button"
class="i-lucide:info w-3.5 h-3.5 text-fg-muted cursor-help"
:aria-label="$t('package.trends.known_anomalies')"
/>
<template #content>
<div class="flex flex-col gap-3">
<p class="text-xs text-fg-muted">
{{ $t('package.trends.known_anomalies_description') }}
</p>
<div v-if="hasAnomalies">
<p class="text-xs text-fg-subtle font-medium">
{{ $t('package.trends.known_anomalies_ranges') }}
</p>
<ul class="text-xs text-fg-subtle list-disc list-inside">
<li v-for="a in packageAnomalies" :key="`${a.packageName}-${a.start}`">
{{
isMultiPackageMode
? $t('package.trends.known_anomalies_range_named', {
packageName: a.packageName,
start: formatAnomalyDate(a.start),
end: formatAnomalyDate(a.end),
})
: $t('package.trends.known_anomalies_range', {
start: formatAnomalyDate(a.start),
end: formatAnomalyDate(a.end),
})
}}
</li>
</ul>
</div>
<p v-else class="text-xs text-fg-muted">
{{ $t('package.trends.known_anomalies_none', effectivePackageNames.length) }}
</p>
<div class="flex justify-end">
<LinkBase
to="https://github.com/npmx-dev/npmx.dev/edit/main/app/utils/download-anomalies.data.ts"
class="text-xs text-accent"
>
{{ $t('package.trends.known_anomalies_contribute') }}
</LinkBase>
</div>
</div>
<p class="text-xs text-fg-muted">
{{ $t('package.trends.known_anomalies_description') }}
</p>
</template>
</TooltipApp>
</span>
<label
class="flex items-center gap-1.5 text-2xs font-mono text-fg-subtle cursor-pointer h-4"
:class="{ 'opacity-50 pointer-events-none': !hasAnomalies }"
>
<input
:checked="settings.chartFilter.anomaliesFixed && hasAnomalies"
:checked="settings.chartFilter.anomaliesFixed"
@change="
settings.chartFilter.anomaliesFixed = ($event.target as HTMLInputElement).checked
"
type="checkbox"
:disabled="!hasAnomalies"
class="accent-[var(--accent-color,var(--fg-subtle))]"
/>
{{ $t('package.trends.apply_correction') }}
Expand Down
9 changes: 4 additions & 5 deletions app/components/Package/WeeklyDownloadStats.vue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import { useCssVariables } from '~/composables/useColors'
import type { WeeklyDataPoint } from '~/types/chart'
import { applyDataCorrection } from '~/utils/chart-data-correction'
import { OKLCH_NEUTRAL_FALLBACK, lightenOklch } from '~/utils/colors'
import { applyBlocklistCorrection } from '~/utils/download-anomalies'
import { applyHampelCorrection } from '~/utils/download-anomalies'
import type { RepoRef } from '#shared/utils/git-providers'
import type { VueUiSparklineConfig, VueUiSparklineDatasetItem } from 'vue-data-ui'
import { onKeyDown } from '@vueuse/core'
Expand Down Expand Up @@ -186,10 +186,9 @@ const correctedDownloads = computed<WeeklyDataPoint[]>(() => {
let data = weeklyDownloads.value as WeeklyDataPoint[]
if (!data.length) return data
if (settings.value.chartFilter.anomaliesFixed) {
data = applyBlocklistCorrection({
data,
packageName: props.packageName,
granularity: 'weekly',
data = applyHampelCorrection(data, {
halfWindow: settings.value.chartFilter.hampelWindow,
threshold: settings.value.chartFilter.hampelThreshold,
}) as WeeklyDataPoint[]
}
data = applyDataCorrection(data, settings.value.chartFilter) as WeeklyDataPoint[]
Expand Down
4 changes: 4 additions & 0 deletions app/composables/useSettings.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ export interface AppSettings {
averageWindow: number
smoothingTau: number
anomaliesFixed: boolean
hampelWindow: number
hampelThreshold: number
Comment on lines +48 to +49
Copy link
Contributor

@coderabbitai coderabbitai bot Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

find . -type f -name "useSettings.ts" | head -5

Repository: npmx-dev/npmx.dev

Length of output: 93

🌐 Web query:

VueUse useLocalStorage mergeDefaults shallow merge documentation

💡 Result:

In VueUse, useLocalStorage() uses the same options as useStorage(), and the mergeDefaults behavior is documented on the useStorage page. (vueuse.org)

  • mergeDefaults: true merges the default value with the value read from storage, and for objects it does a shallow merge (top-level keys only). (vueuse.org)
  • For a deep merge, pass a custom function to mergeDefaults (the docs explicitly show this pattern). (vueuse.org)

Example (custom merge = deep merge):

import { useLocalStorage } from '@vueuse/core'
// import deepMerge from '...'

const state = useLocalStorage('key', { a: { b: 1 } }, {
  mergeDefaults: (storageValue, defaults) => deepMerge(defaults, storageValue),
})

Citations:

🏁 Script executed:

cat -n ./app/composables/useSettings.ts

Repository: npmx-dev/npmx.dev

Length of output: 7350

🏁 Script executed:

# Search for migration or normalisation logic in settings/preferences
rg -i "migrat|normaliz|hydrat" --type ts --type tsx -A 3 app/composables/ 2>/dev/null | head -50

Repository: npmx-dev/npmx.dev

Length of output: 43

🏁 Script executed:

# Check usePreferencesProvider.ts for migration patterns
find . -type f -name "*usePreferences*" -o -name "*preferences*" | grep -E "\.(ts|tsx)$"

Repository: npmx-dev/npmx.dev

Length of output: 236

🏁 Script executed:

head -150 ./app/composables/usePreferencesProvider.ts

Repository: npmx-dev/npmx.dev

Length of output: 2609

🏁 Script executed:

# Search for any settings normalisation, migration, or post-hydration logic
rg "chartFilter|hampel" --type ts --type tsx -B 3 -A 3 app/ | head -100

Repository: npmx-dev/npmx.dev

Length of output: 88

🏁 Script executed:

# Check if there's any middleware or plugin that normalises settings
rg "DEFAULT_SETTINGS|STORAGE_KEY|npmx-settings" --type ts --type tsx -B 2 -A 2 app/

Repository: npmx-dev/npmx.dev

Length of output: 88

🏁 Script executed:

# Search for chartFilter and hampel references
rg "chartFilter|hampel" --type ts -B 3 -A 3 app/

Repository: npmx-dev/npmx.dev

Length of output: 1232

🏁 Script executed:

# Search for settings normalisation anywhere
rg "DEFAULT_SETTINGS|STORAGE_KEY.*npmx-settings" --type ts -B 2 -A 2 app/

Repository: npmx-dev/npmx.dev

Length of output: 885

Deep-merge nested chartFilter settings on hydration.

VueUse's useLocalStorage with mergeDefaults: true only performs a shallow merge at the top level. Returning users with an existing stored chartFilter object will not receive the new hampelWindow and hampelThreshold fields—the entire default chartFilter is replaced, leaving the new settings unset until explicitly written. VueUse documentation recommends a custom merge function for nested defaults.

Suggested fix 
   if (!settingsRef) {
     settingsRef = useLocalStorage<AppSettings>(STORAGE_KEY, DEFAULT_SETTINGS, {
-      mergeDefaults: true,
+      mergeDefaults: (storageValue, defaults) => ({
+        ...defaults,
+        ...storageValue,
+        connector: { ...defaults.connector, ...(storageValue.connector ?? {}) },
+        sidebar: { ...defaults.sidebar, ...(storageValue.sidebar ?? {}) },
+        chartFilter: {
+          ...defaults.chartFilter,
+          ...(storageValue.chartFilter ?? {}),
+        },
+      }),
     })
   }

predictionPoints: number
}
}
Expand All @@ -69,6 +71,8 @@ const DEFAULT_SETTINGS: AppSettings = {
averageWindow: 0,
smoothingTau: 1,
anomaliesFixed: true,
hampelWindow: 3,
hampelThreshold: 3,
predictionPoints: 4,
},
}
Expand Down
30 changes: 0 additions & 30 deletions app/utils/download-anomalies.data.ts
View file
Open in desktop

This file was deleted.

Loading
Loading