-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathtorchdr.user_guide.html
More file actions
931 lines (715 loc) · 61.2 KB
/
torchdr.user_guide.html
File metadata and controls
931 lines (715 loc) · 61.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
<!DOCTYPE html>
<html lang="en" data-content_root="./" >
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>User Guide — TorchDR 0.4 documentation</title>
<script data-cfasync="false">
document.documentElement.dataset.mode = localStorage.getItem("mode") || "";
document.documentElement.dataset.theme = localStorage.getItem("theme") || "";
</script>
<!--
this give us a css class that will be invisible only if js is disabled
-->
<noscript>
<style>
.pst-js-only { display: none !important; }
</style>
</noscript>
<!-- Loaded before other Sphinx assets -->
<link href="_static/styles/theme.css?digest=8878045cc6db502f8baf" rel="stylesheet" />
<link href="_static/styles/pydata-sphinx-theme.css?digest=8878045cc6db502f8baf" rel="stylesheet" />
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=8f2a1f02" />
<link rel="stylesheet" type="text/css" href="_static/sg_gallery.css?v=d2d258e8" />
<link rel="stylesheet" type="text/css" href="_static/sg_gallery-binder.css?v=f4aeca0c" />
<link rel="stylesheet" type="text/css" href="_static/sg_gallery-dataframe.css?v=2082cf3c" />
<link rel="stylesheet" type="text/css" href="_static/sg_gallery-rendered-html.css?v=1277b6f3" />
<!-- So that users can add custom icons -->
<script src="_static/scripts/fontawesome.js?digest=8878045cc6db502f8baf"></script>
<!-- Pre-loaded scripts that we'll load fully later -->
<link rel="preload" as="script" href="_static/scripts/bootstrap.js?digest=8878045cc6db502f8baf" />
<link rel="preload" as="script" href="_static/scripts/pydata-sphinx-theme.js?digest=8878045cc6db502f8baf" />
<script src="_static/documentation_options.js?v=7f00635f"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script>window.MathJax = {"tex": {"equationNumbers": {"autoNumber": "AMS", "useLabelIds": true}}}</script>
<script defer="defer" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
<script>DOCUMENTATION_OPTIONS.pagename = 'torchdr.user_guide';</script>
<script src="_static/pypi-icon.js?v=3850e9a1"></script>
<link rel="icon" href="_static/logo.ico"/>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="DRModule" href="stubs/torchdr.DRModule.html" />
<link rel="prev" title="Torch Dimensionality Reduction" href="index.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
<meta name="docsearch:version" content="" />
</head>
<body data-bs-spy="scroll" data-bs-target=".bd-toc-nav" data-offset="180" data-bs-root-margin="0px 0px -60%" data-default-mode="">
<div id="pst-skip-link" class="skip-link d-print-none"><a href="#main-content">Skip to main content</a></div>
<div id="pst-scroll-pixel-helper"></div>
<button type="button" class="btn rounded-pill" id="pst-back-to-top">
<i class="fa-solid fa-arrow-up"></i>Back to top</button>
<dialog id="pst-search-dialog">
<form class="bd-search d-flex align-items-center"
action="search.html"
method="get">
<i class="fa-solid fa-magnifying-glass"></i>
<input type="search"
class="form-control"
name="q"
placeholder="Search the docs ..."
aria-label="Search the docs ..."
autocomplete="off"
autocorrect="off"
autocapitalize="off"
spellcheck="false"/>
<span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd>K</kbd></span>
</form>
</dialog>
<div class="pst-async-banner-revealer d-none">
<aside id="bd-header-version-warning" class="d-none d-print-none" aria-label="Version warning"></aside>
</div>
<header class="bd-header navbar navbar-expand-lg bd-navbar d-print-none">
<div class="bd-header__inner bd-page-width">
<button class="pst-navbar-icon sidebar-toggle primary-toggle" aria-label="Site navigation">
<span class="fa-solid fa-bars"></span>
</button>
<div class="col-lg-3 navbar-header-items__start">
<div class="navbar-item">
<a class="navbar-brand logo" href="index.html">
<img src="_static/torchdr_logo.png" class="logo__image only-light" alt="TorchDR 0.4 documentation - Home"/>
<img src="_static/torchdr_logo.png" class="logo__image only-dark pst-js-only" alt="TorchDR 0.4 documentation - Home"/>
</a></div>
</div>
<div class="col-lg-9 navbar-header-items">
<div class="me-auto navbar-header-items__center">
<div class="navbar-item">
<nav>
<ul class="bd-navbar-elements navbar-nav">
<li class="nav-item ">
<a class="nav-link nav-internal" href="index.html">
Torch Dimensionality Reduction
</a>
</li>
<li class="nav-item current active">
<a class="nav-link nav-internal" href="#">
User Guide
</a>
</li>
<li class="nav-item ">
<a class="nav-link nav-internal" href="all.html">
API and Modules
</a>
</li>
<li class="nav-item ">
<a class="nav-link nav-internal" href="auto_examples/index.html">
Gallery
</a>
</li>
<li class="nav-item ">
<a class="nav-link nav-internal" href="torchdr.releases.html">
Releases
</a>
</li>
<li class="nav-item dropdown">
<button class="btn dropdown-toggle nav-item" type="button"
data-bs-toggle="dropdown" aria-expanded="false"
aria-controls="pst-nav-more-links">
More
</button>
<ul id="pst-nav-more-links" class="dropdown-menu">
<li class=" ">
<a class="nav-link dropdown-item nav-internal" href="torchdr.contributing.html">
Contributing
</a>
</li>
<li class=" ">
<a class="nav-link dropdown-item nav-internal" href="torchdr.bibliography.html">
Bibliography
</a>
</li>
</ul>
</li>
</ul>
</nav></div>
</div>
<div class="navbar-header-items__end">
<div class="navbar-item navbar-persistent--container">
<button class="btn search-button-field search-button__button pst-js-only" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="fa-solid fa-magnifying-glass"></i>
<span class="search-button__default-text">Search</span>
<span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd class="kbd-shortcut__modifier">K</kbd></span>
</button>
</div>
<div class="navbar-item">
<button class="btn btn-sm nav-link pst-navbar-icon theme-switch-button pst-js-only" aria-label="Color mode" data-bs-title="Color mode" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="theme-switch fa-solid fa-sun fa-lg" data-mode="light" title="Light"></i>
<i class="theme-switch fa-solid fa-moon fa-lg" data-mode="dark" title="Dark"></i>
<i class="theme-switch fa-solid fa-circle-half-stroke fa-lg" data-mode="auto" title="System Settings"></i>
</button></div>
<div class="navbar-item"><ul class="navbar-icon-links"
aria-label="Icon Links">
<li class="nav-item">
<a href="https://github.com/TorchDR/TorchDR" title="GitHub" class="nav-link pst-navbar-icon" rel="noopener" target="_blank" data-bs-toggle="tooltip" data-bs-placement="bottom"><i class="fa-brands fa-github fa-lg" aria-hidden="true"></i>
<span class="sr-only">GitHub</span></a>
</li>
<li class="nav-item">
<a href="https://pypi.org/project/torchdr/" title="PyPI" class="nav-link pst-navbar-icon" rel="noopener" target="_blank" data-bs-toggle="tooltip" data-bs-placement="bottom"><i class="fa-custom fa-pypi fa-lg" aria-hidden="true"></i>
<span class="sr-only">PyPI</span></a>
</li>
</ul></div>
</div>
</div>
<div class="navbar-persistent--mobile">
<button class="btn search-button-field search-button__button pst-js-only" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="fa-solid fa-magnifying-glass"></i>
<span class="search-button__default-text">Search</span>
<span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd class="kbd-shortcut__modifier">K</kbd></span>
</button>
</div>
<button class="pst-navbar-icon sidebar-toggle secondary-toggle" aria-label="On this page">
<span class="fa-solid fa-outdent"></span>
</button>
</div>
</header>
<div class="bd-container">
<div class="bd-container__inner bd-page-width">
<dialog id="pst-primary-sidebar-modal"></dialog>
<div id="pst-primary-sidebar" class="bd-sidebar-primary bd-sidebar">
<div class="sidebar-header-items sidebar-primary__section">
<div class="sidebar-header-items__center">
<div class="navbar-item">
<nav>
<ul class="bd-navbar-elements navbar-nav">
<li class="nav-item ">
<a class="nav-link nav-internal" href="index.html">
Torch Dimensionality Reduction
</a>
</li>
<li class="nav-item current active">
<a class="nav-link nav-internal" href="#">
User Guide
</a>
</li>
<li class="nav-item ">
<a class="nav-link nav-internal" href="all.html">
API and Modules
</a>
</li>
<li class="nav-item ">
<a class="nav-link nav-internal" href="auto_examples/index.html">
Gallery
</a>
</li>
<li class="nav-item ">
<a class="nav-link nav-internal" href="torchdr.releases.html">
Releases
</a>
</li>
<li class="nav-item ">
<a class="nav-link nav-internal" href="torchdr.contributing.html">
Contributing
</a>
</li>
<li class="nav-item ">
<a class="nav-link nav-internal" href="torchdr.bibliography.html">
Bibliography
</a>
</li>
</ul>
</nav></div>
</div>
<div class="sidebar-header-items__end">
<div class="navbar-item">
<button class="btn btn-sm nav-link pst-navbar-icon theme-switch-button pst-js-only" aria-label="Color mode" data-bs-title="Color mode" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="theme-switch fa-solid fa-sun fa-lg" data-mode="light" title="Light"></i>
<i class="theme-switch fa-solid fa-moon fa-lg" data-mode="dark" title="Dark"></i>
<i class="theme-switch fa-solid fa-circle-half-stroke fa-lg" data-mode="auto" title="System Settings"></i>
</button></div>
<div class="navbar-item"><ul class="navbar-icon-links"
aria-label="Icon Links">
<li class="nav-item">
<a href="https://github.com/TorchDR/TorchDR" title="GitHub" class="nav-link pst-navbar-icon" rel="noopener" target="_blank" data-bs-toggle="tooltip" data-bs-placement="bottom"><i class="fa-brands fa-github fa-lg" aria-hidden="true"></i>
<span class="sr-only">GitHub</span></a>
</li>
<li class="nav-item">
<a href="https://pypi.org/project/torchdr/" title="PyPI" class="nav-link pst-navbar-icon" rel="noopener" target="_blank" data-bs-toggle="tooltip" data-bs-placement="bottom"><i class="fa-custom fa-pypi fa-lg" aria-hidden="true"></i>
<span class="sr-only">PyPI</span></a>
</li>
</ul></div>
</div>
</div>
<div class="sidebar-primary-items__start sidebar-primary__section">
<div class="sidebar-primary-item">
<nav class="bd-docs-nav bd-links"
aria-label="Section Navigation">
<p class="bd-links__title" role="heading" aria-level="1">Section Navigation</p>
<div class="bd-toc-item navbar-nav"><ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="stubs/torchdr.DRModule.html">DRModule</a></li>
</ul>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="stubs/torchdr.AffinityMatcher.html">AffinityMatcher</a></li>
</ul>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="stubs/torchdr.Affinity.html">Affinity</a></li>
</ul>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="stubs/torchdr.LogAffinity.html">LogAffinity</a></li>
</ul>
</div>
</nav></div>
</div>
<div class="sidebar-primary-items__end sidebar-primary__section">
<div class="sidebar-primary-item">
<div id="ethical-ad-placement"
class="flat"
data-ea-publisher="readthedocs"
data-ea-type="readthedocs-sidebar"
data-ea-manual="true">
</div></div>
</div>
</div>
<main id="main-content" class="bd-main" role="main">
<div class="bd-content">
<div class="bd-article-container">
<div class="bd-header-article d-print-none">
<div class="header-article-items header-article__inner">
<div class="header-article-items__start">
<div class="header-article-item">
<nav aria-label="Breadcrumb" class="d-print-none">
<ul class="bd-breadcrumbs">
<li class="breadcrumb-item breadcrumb-home">
<a href="index.html" class="nav-link" aria-label="Home">
<i class="fa-solid fa-home"></i>
</a>
</li>
<li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">User Guide</span></li>
</ul>
</nav>
</div>
</div>
</div>
</div>
<div id="searchbox"></div>
<article class="bd-article">
<section id="module-torchdr">
<span id="user-guide"></span><span id="id1"></span><h1>User Guide<a class="headerlink" href="#module-torchdr" title="Link to this heading">#</a></h1>
<nav class="contents local" id="table-of-contents">
<p class="topic-title">Table of Contents</p>
<ul class="simple">
<li><p><a class="reference internal" href="#overview" id="id9">Overview</a></p></li>
<li><p><a class="reference internal" href="#dimensionality-reduction-methods" id="id10">Dimensionality Reduction Methods</a></p></li>
<li><p><a class="reference internal" href="#affinities" id="id11">Affinities</a></p></li>
<li><p><a class="reference internal" href="#scalability-backends" id="id12">Scalability & Backends</a></p></li>
</ul>
</nav>
<section id="overview">
<h2><a class="toc-backref" href="#id9" role="doc-backlink">Overview</a><a class="headerlink" href="#overview" title="Link to this heading">#</a></h2>
<section id="general-formulation-of-dimensionality-reduction">
<h3>General Formulation of Dimensionality Reduction<a class="headerlink" href="#general-formulation-of-dimensionality-reduction" title="Link to this heading">#</a></h3>
<p>DR aims to construct a low-dimensional representation (or embedding) <span class="math notranslate nohighlight">\(\mathbf{Z} = (\mathbf{z}_1, ..., \mathbf{z}_n)^\top\)</span> of an input dataset <span class="math notranslate nohighlight">\(\mathbf{X} = (\mathbf{x}_1, ..., \mathbf{x}_n)^\top\)</span> that best preserves its geometry, encoded via a pairwise affinity matrix <span class="math notranslate nohighlight">\(\mathbf{P}\)</span>. To this end, DR methods optimize <span class="math notranslate nohighlight">\(\mathbf{Z}\)</span> such that a pairwise affinity matrix in the embedding space (denoted <span class="math notranslate nohighlight">\(\mathbf{Q}\)</span>) matches <span class="math notranslate nohighlight">\(\mathbf{P}\)</span>. This general problem is as follows</p>
<div class="math notranslate nohighlight">
\[\min_{\mathbf{Z}} \: \mathcal{L}( \mathbf{P}, \mathbf{Q}) \quad \text{(DR)}\]</div>
<p>where <span class="math notranslate nohighlight">\(\mathcal{L}\)</span> is typically the <span class="math notranslate nohighlight">\(\ell_2\)</span> or cross-entropy loss.</p>
<p>TorchDR is structured around the above formulation <span class="math notranslate nohighlight">\(\text{(DR)}\)</span>.
The input affinity <span class="math notranslate nohighlight">\(\mathbf{P}\)</span> is constructed using an <a class="reference internal" href="stubs/torchdr.Affinity.html#torchdr.Affinity" title="torchdr.Affinity"><code class="xref py py-class docutils literal notranslate"><span class="pre">Affinity</span></code></a> object.
For the embedding affinity <span class="math notranslate nohighlight">\(\mathbf{Q}\)</span>, TorchDR offers flexibility: it can be defined as a separate <a class="reference internal" href="stubs/torchdr.Affinity.html#torchdr.Affinity" title="torchdr.Affinity"><code class="xref py py-class docutils literal notranslate"><span class="pre">Affinity</span></code></a> object or, more commonly, the embedding kernel is integrated directly into the loss function for efficiency.</p>
<p>All modules follow the <code class="docutils literal notranslate"><span class="pre">sklearn</span></code> <span id="id2">[<a class="reference internal" href="torchdr.bibliography.html#id18" title="Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, and others. Scikit-learn: machine learning in python. the Journal of machine Learning research, 12:2825–2830, 2011.">Pedregosa <em>et al.</em>, 2011</a>]</span> API and can be used in <a class="reference external" href="https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html">sklearn pipelines</a>.</p>
</section>
<section id="gpu-support">
<h3>GPU Support<a class="headerlink" href="#gpu-support" title="Link to this heading">#</a></h3>
<p>TorchDR is built on top of <code class="docutils literal notranslate"><span class="pre">torch</span></code> <span id="id3">[<a class="reference internal" href="torchdr.bibliography.html#id17" title="Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, and others. Pytorch: an imperative style, high-performance deep learning library. Advances in neural information processing systems, 2019.">Paszke <em>et al.</em>, 2019</a>]</span>, offering GPU support and automatic differentiation. To utilize GPU support, set <code class="xref py py-attr docutils literal notranslate"><span class="pre">device="cuda"</span></code> when initializing any module; for CPU computations, set <code class="xref py py-attr docutils literal notranslate"><span class="pre">device="cpu"</span></code>. DR particularly benefits from GPU acceleration as most computations involve matrix reductions that are highly parallelizable.</p>
<p>The following sections cover the available DR methods, the affinity building blocks, and advanced scalability options.</p>
</section>
</section>
<section id="dimensionality-reduction-methods">
<h2><a class="toc-backref" href="#id10" role="doc-backlink">Dimensionality Reduction Methods</a><a class="headerlink" href="#dimensionality-reduction-methods" title="Link to this heading">#</a></h2>
<p>TorchDR provides a wide range of dimensionality reduction (DR) methods. All DR estimators inherit the structure of the <a class="reference internal" href="stubs/torchdr.DRModule.html#torchdr.DRModule" title="torchdr.DRModule"><code class="xref py py-meth docutils literal notranslate"><span class="pre">DRModule()</span></code></a> class:</p>
<div class="pst-scrollable-table-container"><table class="autosummary longtable table autosummary">
<tbody>
<tr class="row-odd"><td><p><a class="reference internal" href="stubs/torchdr.DRModule.html#torchdr.DRModule" title="torchdr.DRModule"><code class="xref py py-obj docutils literal notranslate"><span class="pre">torchdr.DRModule</span></code></a></p></td>
<td><p>Base class for DR methods.</p></td>
</tr>
</tbody>
</table>
</div>
<p>They are <code class="xref py py-class docutils literal notranslate"><span class="pre">sklearn.base.BaseEstimator</span></code> and <code class="xref py py-class docutils literal notranslate"><span class="pre">sklearn.base.TransformerMixin</span></code> classes which can be called with the <code class="docutils literal notranslate"><span class="pre">fit_transform</span></code> method.</p>
<p>Outside of <a class="reference internal" href="#spectral-section"><span class="std std-ref">Spectral Methods</span></a>, a closed-form solution to the DR problem is typically not available. The problem can then be solved using <a class="reference external" href="https://pytorch.org/docs/stable/optim.html">gradient-based optimizers</a>.</p>
<p>For users who want full control over both input and output affinities, the <a class="reference internal" href="stubs/torchdr.AffinityMatcher.html#torchdr.AffinityMatcher" title="torchdr.AffinityMatcher"><code class="xref py py-class docutils literal notranslate"><span class="pre">AffinityMatcher</span></code></a> class provides a flexible base that accepts separate <code class="xref py py-attr docutils literal notranslate"><span class="pre">affinity_in</span></code> and <code class="xref py py-attr docutils literal notranslate"><span class="pre">affinity_out</span></code> objects.</p>
<div class="pst-scrollable-table-container"><table class="autosummary longtable table autosummary">
<tbody>
<tr class="row-odd"><td><p><a class="reference internal" href="stubs/torchdr.AffinityMatcher.html#torchdr.AffinityMatcher" title="torchdr.AffinityMatcher"><code class="xref py py-obj docutils literal notranslate"><span class="pre">torchdr.AffinityMatcher</span></code></a></p></td>
<td><p>Perform dimensionality reduction by matching two affinity matrices.</p></td>
</tr>
</tbody>
</table>
</div>
<p>In what follows we briefly present two families of DR algorithms: neighbor embedding methods and spectral methods.</p>
<section id="neighbor-embedding">
<span id="neighbor-embedding-section"></span><h3>Neighbor Embedding<a class="headerlink" href="#neighbor-embedding" title="Link to this heading">#</a></h3>
<p>TorchDR aims to implement most popular <strong>neighbor embedding (NE)</strong> algorithms.
In these methods, <span class="math notranslate nohighlight">\(\mathbf{P}\)</span> and <span class="math notranslate nohighlight">\(\mathbf{Q}\)</span> can be viewed as <strong>soft neighborhood graphs</strong>, hence the term <em>neighbor embedding</em>.</p>
<p>NE objectives share a common structure: they minimize a weighted sum of an attractive term and a repulsive term. The attractive term is often the cross-entropy between the input and output affinities, while the repulsive term is typically a function of the output affinities only. Thus, the NE problem can be formulated as:</p>
<div class="math notranslate nohighlight">
\[\min_{\mathbf{Z}} \: - \lambda \sum_{ij} P_{ij} \log Q_{ij} + \mathcal{L}_{\mathrm{rep}}(\mathbf{Q}) \:.\]</div>
<p>In the above, <span class="math notranslate nohighlight">\(\mathcal{L}_{\mathrm{rep}}(\mathbf{Q})\)</span> represents the repulsive part of the loss function while <span class="math notranslate nohighlight">\(\lambda\)</span> is a hyperparameter that controls the balance between attraction and repulsion. The latter is called <code class="xref py py-attr docutils literal notranslate"><span class="pre">early_exaggeration_coeff</span></code> in TorchDR because it is often set to a value larger than one at the beginning of the optimization.</p>
<p>Many NE methods can be represented within this framework. See below for some examples.</p>
<div class="pst-scrollable-table-container"><table class="table">
<thead>
<tr class="row-odd"><th class="head"><p><strong>Method</strong></p></th>
<th class="head"><p><strong>Repulsive term</strong> <span class="math notranslate nohighlight">\(\mathcal{L}_{\mathrm{rep}}\)</span></p></th>
<th class="head"><p><strong>Input affinity</strong> <span class="math notranslate nohighlight">\(\mathbf{P}\)</span></p></th>
<th class="head"><p><strong>Embedding kernel</strong> <span class="math notranslate nohighlight">\(\mathbf{Q}\)</span></p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p><a class="reference internal" href="gen_modules/torchdr.SNE.html#torchdr.SNE" title="torchdr.SNE"><code class="xref py py-class docutils literal notranslate"><span class="pre">SNE</span></code></a></p></td>
<td><p><span class="math notranslate nohighlight">\(\sum_{i} \log(\sum_j Q_{ij})\)</span></p></td>
<td><p><a class="reference internal" href="gen_modules/torchdr.EntropicAffinity.html#torchdr.EntropicAffinity" title="torchdr.EntropicAffinity"><code class="xref py py-class docutils literal notranslate"><span class="pre">EntropicAffinity</span></code></a></p></td>
<td><p><span class="math notranslate nohighlight">\(Q_{ij} = \exp(- \| \mathbf{z}_i - \mathbf{z}_j \|^2)\)</span></p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="gen_modules/torchdr.TSNE.html#torchdr.TSNE" title="torchdr.TSNE"><code class="xref py py-class docutils literal notranslate"><span class="pre">TSNE</span></code></a></p></td>
<td><p><span class="math notranslate nohighlight">\(\log(\sum_{ij} Q_{ij})\)</span></p></td>
<td><p><a class="reference internal" href="gen_modules/torchdr.EntropicAffinity.html#torchdr.EntropicAffinity" title="torchdr.EntropicAffinity"><code class="xref py py-class docutils literal notranslate"><span class="pre">EntropicAffinity</span></code></a></p></td>
<td><p><span class="math notranslate nohighlight">\(Q_{ij} = (1 + \| \mathbf{z}_i - \mathbf{z}_j \|^2)^{-1}\)</span></p></td>
</tr>
<tr class="row-even"><td><p><a class="reference internal" href="gen_modules/torchdr.TSNEkhorn.html#torchdr.TSNEkhorn" title="torchdr.TSNEkhorn"><code class="xref py py-class docutils literal notranslate"><span class="pre">TSNEkhorn</span></code></a></p></td>
<td><p><span class="math notranslate nohighlight">\(\sum_{ij} Q_{ij}\)</span></p></td>
<td><p><a class="reference internal" href="gen_modules/torchdr.SymmetricEntropicAffinity.html#torchdr.SymmetricEntropicAffinity" title="torchdr.SymmetricEntropicAffinity"><code class="xref py py-class docutils literal notranslate"><span class="pre">SymmetricEntropicAffinity</span></code></a></p></td>
<td><p><a class="reference internal" href="gen_modules/torchdr.SinkhornAffinity.html#torchdr.SinkhornAffinity" title="torchdr.SinkhornAffinity"><code class="xref py py-class docutils literal notranslate"><span class="pre">SinkhornAffinity(base_kernel="student")</span></code></a></p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="gen_modules/torchdr.InfoTSNE.html#torchdr.InfoTSNE" title="torchdr.InfoTSNE"><code class="xref py py-class docutils literal notranslate"><span class="pre">InfoTSNE</span></code></a></p></td>
<td><p><span class="math notranslate nohighlight">\(\sum_i \log(\sum_{j \in \mathrm{Neg}(i)} Q_{ij})\)</span></p></td>
<td><p><a class="reference internal" href="gen_modules/torchdr.EntropicAffinity.html#torchdr.EntropicAffinity" title="torchdr.EntropicAffinity"><code class="xref py py-class docutils literal notranslate"><span class="pre">EntropicAffinity</span></code></a></p></td>
<td><p><span class="math notranslate nohighlight">\(Q_{ij} = (1 + \| \mathbf{z}_i - \mathbf{z}_j \|^2)^{-1}\)</span></p></td>
</tr>
<tr class="row-even"><td><p><a class="reference internal" href="gen_modules/torchdr.UMAP.html#torchdr.UMAP" title="torchdr.UMAP"><code class="xref py py-class docutils literal notranslate"><span class="pre">UMAP</span></code></a></p></td>
<td><p><span class="math notranslate nohighlight">\(- \sum_{i, j \in \mathrm{Neg}(i)} \log (1 - Q_{ij})\)</span></p></td>
<td><p><a class="reference internal" href="gen_modules/torchdr.UMAPAffinity.html#torchdr.UMAPAffinity" title="torchdr.UMAPAffinity"><code class="xref py py-class docutils literal notranslate"><span class="pre">UMAPAffinity</span></code></a></p></td>
<td><p><span class="math notranslate nohighlight">\(Q_{ij} = (1 + a \| \mathbf{z}_i - \mathbf{z}_j \|^{2b})^{-1}\)</span></p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="gen_modules/torchdr.LargeVis.html#torchdr.LargeVis" title="torchdr.LargeVis"><code class="xref py py-class docutils literal notranslate"><span class="pre">LargeVis</span></code></a></p></td>
<td><p><span class="math notranslate nohighlight">\(- \sum_{i, j \in \mathrm{Neg}(i)} \log (1 - Q_{ij})\)</span></p></td>
<td><p><a class="reference internal" href="gen_modules/torchdr.EntropicAffinity.html#torchdr.EntropicAffinity" title="torchdr.EntropicAffinity"><code class="xref py py-class docutils literal notranslate"><span class="pre">EntropicAffinity</span></code></a></p></td>
<td><p><span class="math notranslate nohighlight">\(Q_{ij} = (1 + \| \mathbf{z}_i - \mathbf{z}_j \|^2)^{-1}\)</span></p></td>
</tr>
</tbody>
</table>
</div>
<p>In the above table, <span class="math notranslate nohighlight">\(\mathrm{Neg}(i)\)</span> denotes the set of negative samples
for point <span class="math notranslate nohighlight">\(i\)</span>. They are usually sampled uniformly at random from the dataset.</p>
</section>
<section id="spectral-methods">
<span id="spectral-section"></span><h3>Spectral Methods<a class="headerlink" href="#spectral-methods" title="Link to this heading">#</a></h3>
<p>Spectral methods correspond to choosing the scalar product affinity <span class="math notranslate nohighlight">\(P_{ij} = \langle \mathbf{z}_i, \mathbf{z}_j \rangle\)</span> for the embeddings and the <span class="math notranslate nohighlight">\(\ell_2\)</span> loss.</p>
<div class="math notranslate nohighlight">
\[\min_{\mathbf{Z}} \: \sum_{ij} ( P_{ij} - \langle \mathbf{z}_i, \mathbf{z}_j \rangle )^{2}\]</div>
<p>When <span class="math notranslate nohighlight">\(\mathbf{P}\)</span> is positive semi-definite, this problem is commonly known as kernel Principal Component Analysis <span id="id4">[<a class="reference internal" href="torchdr.bibliography.html#id11" title="Jihun Ham, Daniel D Lee, Sebastian Mika, and Bernhard Schölkopf. A kernel view of the dimensionality reduction of manifolds. In Proceedings of the twenty-first international conference on Machine learning, 47. 2004.">Ham <em>et al.</em>, 2004</a>]</span> and an optimal solution is given by</p>
<div class="math notranslate nohighlight">
\[\mathbf{Z}^{\star} = (\sqrt{\lambda_1} \mathbf{v}_1, ..., \sqrt{\lambda_d} \mathbf{v}_d)^\top\]</div>
<p>where <span class="math notranslate nohighlight">\(\lambda_1, ..., \lambda_d\)</span> are the largest eigenvalues of the centered kernel matrix <span class="math notranslate nohighlight">\(\mathbf{P}\)</span> and <span class="math notranslate nohighlight">\(\mathbf{v}_1, ..., \mathbf{v}_d\)</span> are the corresponding eigenvectors.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>PCA (available at <a class="reference internal" href="gen_modules/torchdr.PCA.html#torchdr.PCA" title="torchdr.PCA"><code class="xref py py-class docutils literal notranslate"><span class="pre">torchdr.PCA</span></code></a>) corresponds to choosing <span class="math notranslate nohighlight">\(P_{ij} = \langle \mathbf{x}_i, \mathbf{x}_j \rangle\)</span>.</p>
</div>
</section>
</section>
<section id="affinities">
<h2><a class="toc-backref" href="#id11" role="doc-backlink">Affinities</a><a class="headerlink" href="#affinities" title="Link to this heading">#</a></h2>
<p>Affinities are the essential building blocks of dimensionality reduction methods.
TorchDR provides a wide range of affinities. See <a class="reference internal" href="all.html#api-and-modules"><span class="std std-ref">API and Modules</span></a> for a complete list.</p>
<section id="base-structure">
<h3>Base structure<a class="headerlink" href="#base-structure" title="Link to this heading">#</a></h3>
<p>Affinities inherit the structure of the following <a class="reference internal" href="stubs/torchdr.Affinity.html#torchdr.Affinity" title="torchdr.Affinity"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Affinity()</span></code></a> class.</p>
<div class="pst-scrollable-table-container"><table class="autosummary longtable table autosummary">
<tbody>
<tr class="row-odd"><td><p><a class="reference internal" href="stubs/torchdr.Affinity.html#torchdr.Affinity" title="torchdr.Affinity"><code class="xref py py-obj docutils literal notranslate"><span class="pre">torchdr.Affinity</span></code></a></p></td>
<td><p>Base class for affinity matrices.</p></td>
</tr>
</tbody>
</table>
</div>
<p>If computations can be performed in log domain, the <a class="reference internal" href="stubs/torchdr.LogAffinity.html#torchdr.LogAffinity" title="torchdr.LogAffinity"><code class="xref py py-meth docutils literal notranslate"><span class="pre">LogAffinity()</span></code></a> class should be used.</p>
<div class="pst-scrollable-table-container"><table class="autosummary longtable table autosummary">
<tbody>
<tr class="row-odd"><td><p><a class="reference internal" href="stubs/torchdr.LogAffinity.html#torchdr.LogAffinity" title="torchdr.LogAffinity"><code class="xref py py-obj docutils literal notranslate"><span class="pre">torchdr.LogAffinity</span></code></a></p></td>
<td><p>Base class for affinity matrices in log domain.</p></td>
</tr>
</tbody>
</table>
</div>
<p>Affinities are objects that can directly be called. The outputted affinity matrix is a <strong>square matrix of size (n, n)</strong> where n is the number of input samples.</p>
<p>Here is an example with the <a class="reference internal" href="gen_modules/torchdr.NormalizedGaussianAffinity.html#torchdr.NormalizedGaussianAffinity" title="torchdr.NormalizedGaussianAffinity"><code class="xref py py-class docutils literal notranslate"><span class="pre">NormalizedGaussianAffinity</span></code></a>:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span><span class="w"> </span><span class="nn">torch</span><span class="o">,</span><span class="w"> </span><span class="nn">torchdr</span>
<span class="gp">>>></span>
<span class="gp">>>> </span><span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
<span class="gp">>>> </span><span class="n">data</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">affinity</span> <span class="o">=</span> <span class="n">torchdr</span><span class="o">.</span><span class="n">NormalizedGaussianAffinity</span><span class="p">(</span><span class="n">normalization_dim</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">affinity_matrix</span> <span class="o">=</span> <span class="n">affinity</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">affinity_matrix</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span>
<span class="go">(100, 100)</span>
</pre></div>
</div>
</section>
<section id="spotlight-on-affinities-based-on-entropic-projections">
<h3>Spotlight on affinities based on entropic projections<a class="headerlink" href="#spotlight-on-affinities-based-on-entropic-projections" title="Link to this heading">#</a></h3>
<p>A widely used family of affinities focuses on <strong>controlling the entropy</strong> of the affinity matrix. It is notably a crucial component of Neighbor-Embedding methods (see <a class="reference internal" href="#neighbor-embedding-section"><span class="std std-ref">Neighbor Embedding</span></a>).</p>
<p>These affinities are normalized such that each row sums to one, allowing the affinity matrix to be viewed as a <strong>Markov transition matrix</strong>. An adaptive bandwidth parameter then determines how the mass from each point spreads to its neighbors. The bandwidth is based on the <code class="xref py py-attr docutils literal notranslate"><span class="pre">perplexity</span></code> hyperparameter which controls the number of effective neighbors for each point.</p>
<p>The resulting affinities can be viewed as a soft approximation of a k-nearest neighbor graph, where <code class="xref py py-attr docutils literal notranslate"><span class="pre">perplexity</span></code> takes the role of k. This allows for capturing more nuances than binary weights, as closer neighbors receive a higher weight compared to those farther away. Ultimately, <code class="xref py py-attr docutils literal notranslate"><span class="pre">perplexity</span></code> is an interpretable hyperparameter that governs the scale of dependencies represented in the affinity.</p>
<p>The following table outlines the aspects controlled by different formulations of entropic affinities. <strong>Marginal</strong> indicates whether each row of the affinity matrix has a controlled sum. <strong>Symmetry</strong> indicates whether the affinity matrix is symmetric. <strong>Entropy</strong> indicates whether each row of the affinity matrix has controlled entropy, dictated by the <code class="xref py py-attr docutils literal notranslate"><span class="pre">perplexity</span></code> hyperparameter.</p>
<div class="pst-scrollable-table-container"><table class="table">
<thead>
<tr class="row-odd"><th class="head"><p><strong>Affinity (associated DR method)</strong></p></th>
<th class="head"><p><strong>Marginal</strong></p></th>
<th class="head"><p><strong>Symmetry</strong></p></th>
<th class="head"><p><strong>Entropy</strong></p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p><a class="reference internal" href="gen_modules/torchdr.NormalizedGaussianAffinity.html#torchdr.NormalizedGaussianAffinity" title="torchdr.NormalizedGaussianAffinity"><code class="xref py py-class docutils literal notranslate"><span class="pre">NormalizedGaussianAffinity</span></code></a></p></td>
<td><p>✅</p></td>
<td><p>❌</p></td>
<td><p>❌</p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="gen_modules/torchdr.SinkhornAffinity.html#torchdr.SinkhornAffinity" title="torchdr.SinkhornAffinity"><code class="xref py py-class docutils literal notranslate"><span class="pre">SinkhornAffinity</span></code></a></p></td>
<td><p>✅</p></td>
<td><p>✅</p></td>
<td><p>❌</p></td>
</tr>
<tr class="row-even"><td><p><a class="reference internal" href="gen_modules/torchdr.EntropicAffinity.html#torchdr.EntropicAffinity" title="torchdr.EntropicAffinity"><code class="xref py py-class docutils literal notranslate"><span class="pre">EntropicAffinity</span></code></a></p></td>
<td><p>✅</p></td>
<td><p>❌</p></td>
<td><p>✅</p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="gen_modules/torchdr.SymmetricEntropicAffinity.html#torchdr.SymmetricEntropicAffinity" title="torchdr.SymmetricEntropicAffinity"><code class="xref py py-class docutils literal notranslate"><span class="pre">SymmetricEntropicAffinity</span></code></a></p></td>
<td><p>✅</p></td>
<td><p>✅</p></td>
<td><p>✅</p></td>
</tr>
</tbody>
</table>
</div>
<p>More details on these affinities can be found in the <a class="reference external" href="https://proceedings.neurips.cc/paper_files/paper/2023/file/8b54ecd9823fff6d37e61ece8f87e534-Paper-Conference.pdf">SNEkhorn paper</a> <span id="id5">[<a class="reference internal" href="torchdr.bibliography.html#id4" title="Hugues Van Assel, Titouan Vayer, Rémi Flamary, and Nicolas Courty. Snekhorn: dimension reduction with symmetric entropic affinities. Advances in Neural Information Processing Systems, 2024.">Van Assel <em>et al.</em>, 2024</a>]</span>.</p>
</section>
<section id="examples-using-entropicaffinity">
<h3>Examples using <code class="docutils literal notranslate"><span class="pre">EntropicAffinity</span></code>:<a class="headerlink" href="#examples-using-entropicaffinity" title="Link to this heading">#</a></h3>
<div class="sphx-glr-thumbnails"><div class="sphx-glr-thumbcontainer" tooltip="We show the adaptivity property of entropic affinities on a toy simulated dataset with heteroscedastic noise."><img alt="" src="_images/sphx_glr_demo_ea_adaptivity_thumb.png" />
<p><a class="reference internal" href="auto_examples/affinities/demo_ea_adaptivity.html"><span class="doc">Entropic Affinities can adapt to varying noise levels</span></a></p>
<div class="sphx-glr-thumbnail-title">Entropic Affinities can adapt to varying noise levels</div>
</div><div class="sphx-glr-thumbcontainer" tooltip="We illustrate the basic usage of TorchDR with different neighbor embedding methods on the SNARE-seq gene expression dataset with given cell type labels."><img alt="" src="_images/sphx_glr_demo_ne_methods_affinity_matcher_thumb.png" />
<p><a class="reference internal" href="auto_examples/basics/demo_ne_methods_affinity_matcher.html"><span class="doc">Neighbor Embedding on genomics & equivalent affinity matcher formulation</span></a></p>
<div class="sphx-glr-thumbnail-title">Neighbor Embedding on genomics & equivalent affinity matcher formulation</div>
</div></div></section>
<section id="other-various-affinities">
<h3>Other various affinities<a class="headerlink" href="#other-various-affinities" title="Link to this heading">#</a></h3>
<p>TorchDR features other affinities that can be used in various contexts.</p>
<p>For instance, the UMAP <span id="id6">[<a class="reference internal" href="torchdr.bibliography.html#id9" title="Leland McInnes, John Healy, and James Melville. Umap: uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018.">McInnes <em>et al.</em>, 2018</a>]</span> algorithm relies on the affinity <a class="reference internal" href="gen_modules/torchdr.UMAPAffinity.html#torchdr.UMAPAffinity" title="torchdr.UMAPAffinity"><code class="xref py py-class docutils literal notranslate"><span class="pre">UMAPAffinity</span></code></a> in input space.
<a class="reference internal" href="gen_modules/torchdr.UMAPAffinity.html#torchdr.UMAPAffinity" title="torchdr.UMAPAffinity"><code class="xref py py-class docutils literal notranslate"><span class="pre">UMAPAffinity</span></code></a> follows a similar construction as entropic affinities to ensure a constant number of effective neighbors, with <code class="xref py py-attr docutils literal notranslate"><span class="pre">n_neighbors</span></code> playing the role of the <code class="xref py py-attr docutils literal notranslate"><span class="pre">perplexity</span></code> hyperparameter.</p>
<p>Another example is the doubly stochastic normalization of a base affinity under the <span class="math notranslate nohighlight">\(\ell_2\)</span> geometry that has recently been proposed for DR <span id="id7">[<a class="reference internal" href="torchdr.bibliography.html#id10" title="Stephen Zhang, Gilles Mordant, Tetsuya Matsumoto, and Geoffrey Schiebinger. Manifold learning with sparse regularised optimal transport. arXiv preprint arXiv:2307.09816, 2023.">Zhang <em>et al.</em>, 2023</a>]</span>. This method is analogous to <a class="reference internal" href="gen_modules/torchdr.SinkhornAffinity.html#torchdr.SinkhornAffinity" title="torchdr.SinkhornAffinity"><code class="xref py py-class docutils literal notranslate"><span class="pre">SinkhornAffinity</span></code></a> where the Shannon entropy is replaced by the <span class="math notranslate nohighlight">\(\ell_2\)</span> norm to recover a sparse affinity.
It is available at <a class="reference internal" href="gen_modules/torchdr.DoublyStochasticQuadraticAffinity.html#torchdr.DoublyStochasticQuadraticAffinity" title="torchdr.DoublyStochasticQuadraticAffinity"><code class="xref py py-class docutils literal notranslate"><span class="pre">DoublyStochasticQuadraticAffinity</span></code></a>.</p>
</section>
</section>
<section id="scalability-backends">
<h2><a class="toc-backref" href="#id12" role="doc-backlink">Scalability & Backends</a><a class="headerlink" href="#scalability-backends" title="Link to this heading">#</a></h2>
<p>For large datasets, TorchDR provides several features to scale computations.</p>
<section id="handling-the-quadratic-cost-via-sparsity-or-symbolic-tensors">
<h3>Handling the quadratic cost via sparsity or symbolic tensors<a class="headerlink" href="#handling-the-quadratic-cost-via-sparsity-or-symbolic-tensors" title="Link to this heading">#</a></h3>
<p>Affinities naturally incur a quadratic memory cost, which can be particularly
problematic when dealing with large numbers of samples, especially when using GPUs.</p>
<p>Many affinity metrics only require computing distances to each point’s k nearest neighbors.
For large datasets (typically when <span class="math notranslate nohighlight">\(n > 10^4\)</span>) where the full pairwise-distance matrix won’t fit in GPU memory,
TorchDR can offload these computations to the GPU-compatible kNN library <a class="reference external" href="https://github.com/facebookresearch/faiss">Faiss</a>.
Simply set <code class="xref py py-attr docutils literal notranslate"><span class="pre">backend</span></code> to <code class="docutils literal notranslate"><span class="pre">faiss</span></code> to leverage Faiss’s efficient implementations.</p>
<p>Alternatively, for exact computations or affinities that can’t be limited to kNNs, you can use symbolic (lazy) tensors to avoid memory overflows.
TorchDR integrates with <code class="docutils literal notranslate"><span class="pre">pykeops</span></code><span id="id8">[<a class="reference internal" href="torchdr.bibliography.html#id16" title="Benjamin Charlier, Jean Feydy, Joan Alexis Glaunes, François-David Collin, and Ghislain Durif. Kernel operations on the gpu, with autodiff, without memory overflows. Journal of Machine Learning Research, 22(74):1–6, 2021.">Charlier <em>et al.</em>, 2021</a>]</span>, representing tensors as mathematical expressions evaluated directly on your data samples.
By computing on-the-fly formulas instead of storing full matrices, this approach removes memory constraints entirely.
However, for very large datasets (typically when <span class="math notranslate nohighlight">\(n > 10^5\)</span>), the computational cost becomes prohibitive.
Simply set <code class="xref py py-attr docutils literal notranslate"><span class="pre">backend</span></code> to <code class="docutils literal notranslate"><span class="pre">keops</span></code> in any module to enable it.</p>
<a class="reference internal image-reference" href="_images/symbolic_matrix.svg"><img alt="_images/symbolic_matrix.svg" class="align-center" src="_images/symbolic_matrix.svg" style="width: 800px;" />
</a>
<p>The above figure is taken from <a class="reference external" href="https://github.com/getkeops/keops/blob/main/doc/_static/symbolic_matrix.svg">here</a>.</p>
</section>
<section id="multi-gpu-distributed-training">
<h3>Multi-GPU distributed training<a class="headerlink" href="#multi-gpu-distributed-training" title="Link to this heading">#</a></h3>
<p>For very large datasets, TorchDR supports <strong>multi-GPU distributed training</strong> to parallelize computations across multiple GPUs. This is particularly useful when the affinity matrix is too large to compute on a single GPU or when faster computation is needed.</p>
<p><strong>How it works:</strong></p>
<ol class="arabic simple">
<li><p><strong>Data partitioning</strong>: The full dataset <span class="math notranslate nohighlight">\(\mathbf{X} \in \mathbb{R}^{N \times d}\)</span> is split into chunks across GPUs. Each GPU handles a subset of <span class="math notranslate nohighlight">\(n_i\)</span> samples.</p></li>
<li><p><strong>Shared neighbor index</strong>: A FAISS index is built on the full dataset and shared across all GPUs, enabling efficient k-nearest neighbor queries.</p></li>
<li><p><strong>Parallel affinity computation</strong>: Each GPU computes the affinity rows for its chunk by querying the shared index. GPU <span class="math notranslate nohighlight">\(i\)</span> computes rows <span class="math notranslate nohighlight">\([\text{start}_i, \text{end}_i)\)</span> of the affinity matrix.</p></li>
<li><p><strong>Distributed gradient computation</strong>: During optimization, each GPU:</p>
<ul class="simple">
<li><p>Computes gradients <span class="math notranslate nohighlight">\(\nabla \mathcal{L}_i\)</span> for its local chunk</p></li>
<li><p>Maintains a full copy of the embedding <span class="math notranslate nohighlight">\(\mathbf{Z} \in \mathbb{R}^{N \times q}\)</span></p></li>
<li><p>Synchronizes gradients via <strong>all-reduce</strong> across GPUs</p></li>
</ul>
</li>
<li><p><strong>Synchronized updates</strong>: After all-reduce, each GPU has identical aggregated gradients and applies the same optimizer step, keeping embeddings synchronized.</p></li>
</ol>
<a class="reference internal image-reference" href="_images/multi_gpu_dr_extended.png"><img alt="_images/multi_gpu_dr_extended.png" class="align-center" src="_images/multi_gpu_dr_extended.png" style="width: 800px;" />
</a>
<p><em>Multi-GPU dimensionality reduction pipeline. The dataset is split into chunks across GPUs. Each GPU performs neighbor search using a shared index, computes its affinity chunk, and generates local gradients. Gradients are synchronized via all-reduce and used to update the embedding copies maintained on each device.</em></p>
<p><strong>Usage:</strong></p>
<p>Multi-GPU mode is enabled automatically when launching with <code class="docutils literal notranslate"><span class="pre">torchrun</span></code> or the TorchDR CLI:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># Using the TorchDR CLI (recommended)</span>
torchdr<span class="w"> </span>--gpus<span class="w"> </span><span class="m">4</span><span class="w"> </span>your_script.py
<span class="c1"># Or using torchrun directly</span>
torchrun<span class="w"> </span>--nproc_per_node<span class="o">=</span><span class="m">4</span><span class="w"> </span>your_script.py
</pre></div>
</div>
<p>In your script, simply use TorchDR as usual:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">torchdr</span>
<span class="c1"># Distributed mode is auto-detected when launched with torchrun</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">torchdr</span><span class="o">.</span><span class="n">UMAP</span><span class="p">(</span><span class="n">n_neighbors</span><span class="o">=</span><span class="mi">15</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">embedding</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">fit_transform</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
</pre></div>
</div>
<p><strong>Requirements:</strong></p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">backend="faiss"</span></code> (default for most methods)</p></li>
<li><p>Multiple CUDA GPUs available</p></li>
<li><p>Launch with <code class="docutils literal notranslate"><span class="pre">torchdr</span></code> CLI or <code class="docutils literal notranslate"><span class="pre">torchrun</span></code></p></li>
</ul>
<p><strong>Supported methods:</strong></p>
<p>Currently, the following methods support multi-GPU:</p>
<ul class="simple">
<li><p><a class="reference internal" href="gen_modules/torchdr.PCA.html#torchdr.PCA" title="torchdr.PCA"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCA</span></code></a> (with <code class="docutils literal notranslate"><span class="pre">distributed="auto"</span></code>)</p></li>
<li><p><a class="reference internal" href="gen_modules/torchdr.ExactIncrementalPCA.html#torchdr.ExactIncrementalPCA" title="torchdr.ExactIncrementalPCA"><code class="xref py py-class docutils literal notranslate"><span class="pre">ExactIncrementalPCA</span></code></a> (with <code class="docutils literal notranslate"><span class="pre">distributed="auto"</span></code>)</p></li>
<li><p><a class="reference internal" href="gen_modules/torchdr.UMAP.html#torchdr.UMAP" title="torchdr.UMAP"><code class="xref py py-class docutils literal notranslate"><span class="pre">UMAP</span></code></a></p></li>
<li><p><a class="reference internal" href="gen_modules/torchdr.InfoTSNE.html#torchdr.InfoTSNE" title="torchdr.InfoTSNE"><code class="xref py py-class docutils literal notranslate"><span class="pre">InfoTSNE</span></code></a></p></li>
<li><p><a class="reference internal" href="gen_modules/torchdr.LargeVis.html#torchdr.LargeVis" title="torchdr.LargeVis"><code class="xref py py-class docutils literal notranslate"><span class="pre">LargeVis</span></code></a></p></li>
</ul>
</section>
<section id="dataloader-for-streaming-data">
<h3>DataLoader for Streaming Data<a class="headerlink" href="#dataloader-for-streaming-data" title="Link to this heading">#</a></h3>
<p>For datasets too large to fit in RAM, TorchDR supports passing a PyTorch <code class="docutils literal notranslate"><span class="pre">DataLoader</span></code> directly to <code class="docutils literal notranslate"><span class="pre">fit_transform</span></code> instead of a tensor. This enables streaming computation where data is processed batch-by-batch.</p>
<p><strong>How it works:</strong></p>
<ol class="arabic simple">
<li><p><strong>Incremental index building</strong>: The FAISS index is built incrementally from batches</p></li>
<li><p><strong>Batch-by-batch k-NN queries</strong>: Neighbor queries are performed as each batch arrives</p></li>
<li><p><strong>Automatic PCA initialization</strong>: When <code class="docutils literal notranslate"><span class="pre">init="pca"</span></code>, <a class="reference internal" href="gen_modules/torchdr.IncrementalPCA.html#torchdr.IncrementalPCA" title="torchdr.IncrementalPCA"><code class="xref py py-class docutils literal notranslate"><span class="pre">IncrementalPCA</span></code></a> is used automatically</p></li>
</ol>
<p><strong>Usage:</strong></p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">torch.utils.data</span><span class="w"> </span><span class="kn">import</span> <span class="n">DataLoader</span><span class="p">,</span> <span class="n">TensorDataset</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">torchdr</span>
<span class="n">dataset</span> <span class="o">=</span> <span class="n">TensorDataset</span><span class="p">(</span><span class="n">X</span><span class="p">)</span> <span class="c1"># X is your large tensor</span>
<span class="n">dataloader</span> <span class="o">=</span> <span class="n">DataLoader</span><span class="p">(</span><span class="n">dataset</span><span class="p">,</span> <span class="n">batch_size</span><span class="o">=</span><span class="mi">10000</span><span class="p">)</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">torchdr</span><span class="o">.</span><span class="n">UMAP</span><span class="p">(</span><span class="n">n_neighbors</span><span class="o">=</span><span class="mi">15</span><span class="p">,</span> <span class="n">backend</span><span class="o">=</span><span class="s2">"faiss"</span><span class="p">)</span>
<span class="n">embedding</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">fit_transform</span><span class="p">(</span><span class="n">dataloader</span><span class="p">)</span>
</pre></div>
</div>
<p><strong>Requirements:</strong></p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">backend="faiss"</span></code> must be used</p></li>
<li><p>The DataLoader must yield tensors (or tuples where the first element is the data tensor)</p></li>
</ul>
</section>
</section>
</section>
</article>
<footer class="prev-next-footer d-print-none">
<div class="prev-next-area">
<a class="left-prev"
href="index.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
<p class="prev-next-title">Torch Dimensionality Reduction</p>
</div>
</a>
<a class="right-next"
href="stubs/torchdr.DRModule.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
<p class="prev-next-title">DRModule</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
</div>
</footer>
</div>
<dialog id="pst-secondary-sidebar-modal"></dialog>
<div id="pst-secondary-sidebar" class="bd-sidebar-secondary bd-toc"><div class="sidebar-secondary-items sidebar-secondary__inner">
<div class="sidebar-secondary-item">
<div
id="pst-page-navigation-heading-2"
class="page-toc tocsection onthispage">
<i class="fa-solid fa-list"></i> On this page
</div>
<nav class="bd-toc-nav page-toc" aria-labelledby="pst-page-navigation-heading-2">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#overview">Overview</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#general-formulation-of-dimensionality-reduction">General Formulation of Dimensionality Reduction</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#gpu-support">GPU Support</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#dimensionality-reduction-methods">Dimensionality Reduction Methods</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#neighbor-embedding">Neighbor Embedding</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#spectral-methods">Spectral Methods</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#affinities">Affinities</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#base-structure">Base structure</a><ul class="nav section-nav flex-column">
</ul>
</li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#spotlight-on-affinities-based-on-entropic-projections">Spotlight on affinities based on entropic projections</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#examples-using-entropicaffinity">Examples using <code class="docutils literal notranslate"><span class="pre">EntropicAffinity</span></code>:</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#other-various-affinities">Other various affinities</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#scalability-backends">Scalability & Backends</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#handling-the-quadratic-cost-via-sparsity-or-symbolic-tensors">Handling the quadratic cost via sparsity or symbolic tensors</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#multi-gpu-distributed-training">Multi-GPU distributed training</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#dataloader-for-streaming-data">DataLoader for Streaming Data</a></li>
</ul>
</li>
</ul>
</nav></div>
<div class="sidebar-secondary-item">
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/torchdr.user_guide.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div></div>
</div></div>
</div>
<footer class="bd-footer-content">
</footer>
</main>
</div>
</div>
<!-- Scripts loaded after <body> so the DOM is not blocked -->
<script defer src="_static/scripts/bootstrap.js?digest=8878045cc6db502f8baf"></script>
<script defer src="_static/scripts/pydata-sphinx-theme.js?digest=8878045cc6db502f8baf"></script>
<footer class="bd-footer">
<div class="bd-footer__inner bd-page-width">
<div class="footer-items__start">
<div class="footer-item">
<p class="copyright">
© Copyright 2024, TorchDR team.
<br/>
</p>
</div>
<div class="footer-item">
<p class="sphinx-version">
Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 8.1.3.
<br/>
</p>
</div>
</div>
<div class="footer-items__end">
<div class="footer-item">
<p class="theme-version">
<!-- # L10n: Setting the PST URL as an argument as this does not need to be localized -->
Built with the <a href="https://pydata-sphinx-theme.readthedocs.io/en/stable/index.html">PyData Sphinx Theme</a> 0.16.1.
</p></div>
</div>
</div>
</footer>
</body>
</html>