Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions Doc/using/cmdline.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1113,6 +1113,14 @@ conflict.
and kill the process. Only enable this in environments where the
huge-page pool is properly sized and fork-safety is not a concern.

On Windows you need a special privilege. See the
`Windows documentation for large pages
<https://learn.microsoft.com/windows/win32/memory/large-page-support>`_
for details. Python will fail on startup if the required privilege
`SeLockMemoryPrivilege
<https://learn.microsoft.com/previous-versions/windows/it-pro/windows-10/security/threat-protection/security-policy-settings/lock-pages-in-memory>`_
is not held by the user.

.. versionadded:: 3.15


Expand Down
15 changes: 12 additions & 3 deletions Doc/using/configure.rst
Original file line number Diff line number Diff line change
Expand Up @@ -792,9 +792,18 @@ also be used to improve performance.

Even when compiled with this option, huge pages are **not** used at runtime
unless the :envvar:`PYTHON_PYMALLOC_HUGEPAGES` environment variable is set
to ``1``. This opt-in is required because huge pages carry risks on Linux:
if the huge-page pool is exhausted, page faults (including copy-on-write
faults after :func:`os.fork`) deliver ``SIGBUS`` and kill the process.
to ``1``. This opt-in is required because huge pages

* carry risks on Linux: if the huge-page pool is exhausted, page faults
(including copy-on-write faults after :func:`os.fork`) deliver ``SIGBUS``
and kill the process.

* need a special privilege on Windows. See the `Windows documentation for large pages
<https://learn.microsoft.com/windows/win32/memory/large-page-support>`_
for details. Python will fail on startup if the required privilege
`SeLockMemoryPrivilege
<https://learn.microsoft.com/previous-versions/windows/it-pro/windows-10/security/threat-protection/security-policy-settings/lock-pages-in-memory>`_
is not held by the user.

The configure script checks that the platform supports ``MAP_HUGETLB``
and emits a warning if it is not available.
Expand Down
46 changes: 46 additions & 0 deletions Python/pylifecycle.c
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,10 @@
#include "pycore_jit.h" // _PyJIT_Fini()
#endif

#if defined(PYMALLOC_USE_HUGEPAGES) && defined(MS_WINDOWS)
#include <Windows.h>
#endif

#include "opcode.h"

#include <locale.h> // setlocale()
Expand Down Expand Up @@ -485,6 +489,39 @@ pyinit_core_reconfigure(_PyRuntimeState *runtime,
return _PyStatus_OK();
}

#if defined(PYMALLOC_USE_HUGEPAGES) && defined(MS_WINDOWS)
static PyStatus
get_huge_pages_privilege(void)
{
HANDLE hToken;
if (!OpenProcessToken(GetCurrentProcess(), TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY, &hToken))
{
return _PyStatus_ERR("failed to open process token");
}
TOKEN_PRIVILEGES tp;
if (!LookupPrivilegeValue(NULL, SE_LOCK_MEMORY_NAME, &tp.Privileges[0].Luid))
{
CloseHandle(hToken);
return _PyStatus_ERR("failed to lookup SeLockMemoryPrivilege for huge pages");
}
tp.PrivilegeCount = 1;
tp.Privileges[0].Attributes = SE_PRIVILEGE_ENABLED;
// AdjustTokenPrivileges can return with nonzero status (i.e. success)
// but without having all privileges adjusted (ERROR_NOT_ALL_ASSIGNED).
BOOL status = AdjustTokenPrivileges(hToken, FALSE, &tp, sizeof(TOKEN_PRIVILEGES), NULL, NULL);
DWORD error = GetLastError();
if (!status || (error != ERROR_SUCCESS))
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I am not missing anything seems that the MSDN docs state that AdjustTokenPrivileges returns nonzero (success) even when not all privileges were adjusted: we correctly handle this via GetLastError(). However, the error message "failed to obtain SeLockMemoryPrivilege for huge pages" doesn’t distinguish between:

  • The token adjustment API failing entirely
  • The privilege simply not being held

The second case is the common, expected failure mode (user just doesn’t have the privilege). A more actionable error message for that case would help like perhaps "SeLockMemoryPrivilege not held; grant it via Local Security Policy or disable PYTHON_PYMALLOC_HUGEPAGES".

{
CloseHandle(hToken);
return _PyStatus_ERR("failed to obtain SeLockMemoryPrivilege for huge pages");
}
if (!CloseHandle(hToken))
{
return _PyStatus_ERR("failed to close process token handle");
}
return _PyStatus_OK();
}
#endif

static PyStatus
pycore_init_runtime(_PyRuntimeState *runtime,
Expand All @@ -499,6 +536,15 @@ pycore_init_runtime(_PyRuntimeState *runtime,
return status;
}

#if defined(PYMALLOC_USE_HUGEPAGES) && defined(MS_WINDOWS)
if (runtime->allocators.use_hugepages) {
status = get_huge_pages_privilege();
if (_PyStatus_EXCEPTION(status)) {
return status;
}
}
#endif

/* Py_Finalize leaves _Py_Finalizing set in order to help daemon
* threads behave a little more gracefully at interpreter shutdown.
* We clobber it here so the new interpreter can start with a clean
Expand Down
Loading