diff --git a/GRPC_INTERFACE.md b/GRPC_INTERFACE.md deleted file mode 100644 index cdcffead9..000000000 --- a/GRPC_INTERFACE.md +++ /dev/null @@ -1,392 +0,0 @@ -# gRPC Interface Architecture - -## Overview - -The cuOpt remote execution system uses gRPC for client-server communication. The interface -supports arbitrarily large optimization problems (multi-GB) through a chunked array transfer -protocol that uses only unary (request-response) RPCs — no bidirectional streaming. - -All client-server serialization uses protocol buffers generated by `protoc` and -`grpc_cpp_plugin`. The internal server-to-worker pipe uses protobuf for metadata -headers and raw byte transfer for bulk array data (see Security Notes). - -## Directory Layout - -All gRPC-related C++ source lives under a single tree: - -``` -cpp/src/grpc/ -├── cuopt_remote.proto # Base protobuf messages (job status, settings, etc.) -├── cuopt_remote_service.proto # Service definition + messages (SubmitJob, ChunkedUpload, Incumbent, etc.) -├── grpc_problem_mapper.{hpp,cpp} # CPU problem ↔ proto (incl. chunked header) -├── grpc_solution_mapper.{hpp,cpp} # LP/MIP solution ↔ proto (unary + chunked) -├── grpc_settings_mapper.{hpp,cpp} # PDLP/MIP settings ↔ proto -├── grpc_service_mapper.{hpp,cpp} # Request/response builders (status, cancel, stream logs, etc.) -├── client/ -│ ├── grpc_client.{hpp,cpp} # High-level client: connect, submit, poll, get result -│ └── solve_remote.cpp # solve_lp_remote / solve_mip_remote (uses grpc_client) -└── server/ - ├── grpc_server_main.cpp # main(), argument parsing, gRPC server setup - ├── grpc_service_impl.cpp # CuOptRemoteServiceImpl — all RPC handlers - ├── grpc_server_types.hpp # Shared types, globals, forward declarations - ├── grpc_field_element_size.hpp # ArrayFieldId → element byte size (codegen target) - ├── grpc_pipe_serialization.hpp # Pipe I/O: protobuf headers + raw byte arrays (request/result) - ├── grpc_incumbent_proto.hpp # Incumbent proto build/parse (codegen target) - ├── grpc_worker.cpp # worker_process(), incumbent callback, store_simple_result - ├── grpc_worker_infra.cpp # Pipes, spawn, wait_for_workers, mark_worker_jobs_failed - ├── grpc_server_threads.cpp # result_retrieval, incumbent_retrieval, session_reaper - └── grpc_job_management.cpp # Pipe I/O, submit_job_async, check_status, cancel, etc. -``` - -- **Protos**: Live in `cpp/src/grpc/`. CMake generates C++ in the build dir (`cuopt_remote.pb.h`, `cuopt_remote_service.pb.h`, `cuopt_remote_service.grpc.pb.h`). -- **Mappers**: Shared by client and server; convert between host C++ types and protobuf. Used for unary and chunked paths. -- **Client**: Solver-level utility (not public API). Used by `solve_lp_remote`/`solve_mip_remote` and tests. -- **Server**: Standalone executable `cuopt_grpc_server`. See `GRPC_SERVER_ARCHITECTURE.md` for process model and file roles. - -## Protocol Files - -| File | Purpose | -|------|---------| -| `cpp/src/grpc/cuopt_remote.proto` | Message definitions (problems, settings, solutions, field IDs) | -| `cpp/src/grpc/cuopt_remote_service.proto` | gRPC service definition (RPCs) | - -Generated code is placed in the CMake build directory (not checked into source). - -## Service Interface - -```protobuf -service CuOptRemoteService { - // Job submission (small problems, single message) - rpc SubmitJob(SubmitJobRequest) returns (SubmitJobResponse); - - // Chunked upload (large problems, multiple unary RPCs) - rpc StartChunkedUpload(StartChunkedUploadRequest) returns (StartChunkedUploadResponse); - rpc SendArrayChunk(SendArrayChunkRequest) returns (SendArrayChunkResponse); - rpc FinishChunkedUpload(FinishChunkedUploadRequest) returns (SubmitJobResponse); - - // Job management - rpc CheckStatus(StatusRequest) returns (StatusResponse); - rpc CancelJob(CancelRequest) returns (CancelResponse); - rpc DeleteResult(DeleteRequest) returns (DeleteResponse); - - // Result retrieval (small results, single message) - rpc GetResult(GetResultRequest) returns (ResultResponse); - - // Chunked download (large results, multiple unary RPCs) - rpc StartChunkedDownload(StartChunkedDownloadRequest) returns (StartChunkedDownloadResponse); - rpc GetResultChunk(GetResultChunkRequest) returns (GetResultChunkResponse); - rpc FinishChunkedDownload(FinishChunkedDownloadRequest) returns (FinishChunkedDownloadResponse); - - // Blocking wait (returns status only, use GetResult afterward) - rpc WaitForCompletion(WaitRequest) returns (WaitResponse); - - // Real-time streaming - rpc StreamLogs(StreamLogsRequest) returns (stream LogMessage); - rpc GetIncumbents(IncumbentRequest) returns (IncumbentResponse); -} -``` - -## Chunked Array Transfer Protocol - -### Why Chunking? - -gRPC has per-message size limits (configurable, default set to 256 MiB in cuOpt), and -protobuf has a hard 2 GB serialization limit. Optimization problems and their solutions -can exceed several gigabytes, so a chunked transfer mechanism is needed. - -The protocol uses only **unary RPCs** (no bidirectional streaming), which simplifies -error handling, load balancing, and proxy compatibility. - -### Upload Protocol (Large Problems) - -When the estimated serialized problem size exceeds 75% of `max_message_bytes`, the client -splits large arrays into chunks and sends them via multiple unary RPCs: - -``` -Client Server - | | - |-- StartChunkedUpload(header, settings) -----> | - |<-- upload_id, max_message_bytes -------------- | - | | - |-- SendArrayChunk(upload_id, field, data) ----> | - |<-- ok ---------------------------------------- | - | | - |-- SendArrayChunk(upload_id, field, data) ----> | - |<-- ok ---------------------------------------- | - | ... | - | | - |-- FinishChunkedUpload(upload_id) ------------> | - |<-- job_id ------------------------------------ | -``` - -**Key features:** -- `StartChunkedUpload` sends a `ChunkedProblemHeader` with all scalar fields and - array metadata (`ArrayDescriptor` for each large array: field ID, total elements, - element size) -- Each `SendArrayChunk` carries one chunk of one array, identified by `ArrayFieldId` - and `element_offset` -- The server reports `max_message_bytes` so the client can adapt chunk sizing -- `FinishChunkedUpload` triggers server-side reassembly and job submission - -### Download Protocol (Large Results) - -When the result exceeds the gRPC max message size, the client fetches it via -chunked unary RPCs (mirrors the upload pattern): - -``` -Client Server - | | - |-- StartChunkedDownload(job_id) --------------> | - |<-- download_id, ChunkedResultHeader ---------- | - | | - |-- GetResultChunk(download_id, field, off) ----> | - |<-- data bytes --------------------------------- | - | | - |-- GetResultChunk(download_id, field, off) ----> | - |<-- data bytes --------------------------------- | - | ... | - | | - |-- FinishChunkedDownload(download_id) ---------> | - |<-- ok ----------------------------------------- | -``` - -**Key features:** -- `ChunkedResultHeader` carries all scalar fields (termination status, objectives, - residuals, solve time, warm start scalars) plus `ResultArrayDescriptor` entries - for each array (solution vectors, warm start arrays) -- Each `GetResultChunk` fetches a slice of one array, identified by `ResultFieldId` - and `element_offset` -- `FinishChunkedDownload` releases the server-side download session state -- LP results include PDLP warm start data (9 arrays + 8 scalars) for subsequent - warm-started solves - -### Automatic Routing - -The client handles size-based routing transparently: - -1. **Upload**: Estimate serialized problem size - - Below 75% of `max_message_bytes` → unary `SubmitJob` - - Above threshold → `StartChunkedUpload` + `SendArrayChunk` + `FinishChunkedUpload` -2. **Download**: Check `result_size_bytes` from `CheckStatus` - - Below `max_message_bytes` → unary `GetResult` - - Above limit (or `RESOURCE_EXHAUSTED`) → chunked download RPCs - -## Error Handling - -### gRPC Status Codes - -| Code | Meaning | Client Action | -|------|---------|---------------| -| `OK` | Success | Process result | -| `NOT_FOUND` | Job ID not found | Check job ID | -| `RESOURCE_EXHAUSTED` | Message too large | Use chunked transfer | -| `CANCELLED` | Job was cancelled | Handle gracefully | -| `DEADLINE_EXCEEDED` | Timeout | Retry or increase timeout | -| `UNAVAILABLE` | Server not reachable | Retry with backoff | -| `INTERNAL` | Server error | Report to user | -| `INVALID_ARGUMENT` | Bad request | Fix request | - -### Connection Handling - -- Client detects `context->IsCancelled()` for graceful disconnect -- Server cleans up job state on client disconnect during upload -- Automatic reconnection is NOT built-in (caller should retry) - -## Completion Strategy - -The `solve_lp` and `solve_mip` methods poll `CheckStatus` every `poll_interval_ms` -until the job reaches a terminal state (COMPLETED/FAILED/CANCELLED) or `timeout_seconds` -is exceeded. During polling, MIP incumbent callbacks are invoked on the main thread. - -The `WaitForCompletion` RPC is available as a public async API primitive for callers -managing jobs directly, but it is not used by the convenience `solve_*` methods because -polling provides timeout protection and enables incumbent callbacks. - -## Client API (`grpc_client_t`) - -### Configuration - -```cpp -struct grpc_client_config_t { - std::string server_address = "localhost:8765"; - int poll_interval_ms = 1000; - int timeout_seconds = 3600; // Max wait for job completion (1 hour) - bool stream_logs = false; // Stream solver logs from server - - // Callbacks - std::function log_callback; - std::function debug_log_callback; // Internal client debug messages - std::function&)> incumbent_callback; - int incumbent_poll_interval_ms = 1000; - - // TLS configuration - bool enable_tls = false; - std::string tls_root_certs; // CA certificate (PEM) - std::string tls_client_cert; // Client certificate (mTLS) - std::string tls_client_key; // Client private key (mTLS) - - // Transfer configuration - int64_t max_message_bytes = 256 * 1024 * 1024; // 256 MiB - int64_t chunk_size_bytes = 16 * 1024 * 1024; // 16 MiB per chunk - // Chunked upload threshold is computed as 75% of max_message_bytes. - bool enable_transfer_hash = false; // FNV-1a hash logging -}; -``` - -### Synchronous Operations - -```cpp -// Blocking solve — handles chunked transfer automatically -auto result = client.solve_lp(problem, settings); -auto result = client.solve_mip(problem, settings, enable_incumbents); -``` - -### Asynchronous Operations - -```cpp -// Submit and get job ID -auto submit = client.submit_lp(problem, settings); -std::string job_id = submit.job_id; - -// Poll for status -auto status = client.check_status(job_id); - -// Get result when ready -auto result = client.get_lp_result(job_id); - -// Cancel or delete -client.cancel_job(job_id); -client.delete_job(job_id); -``` - -### Real-Time Streaming - -```cpp -// Log streaming (callback-based) -client.stream_logs(job_id, 0, [](const std::string& line, bool done) { - std::cout << line; - return true; // continue streaming -}); - -// Incumbent polling (during MIP solve) -config.incumbent_callback = [](int64_t idx, double obj, const auto& sol) { - std::cout << "Incumbent " << idx << ": " << obj << "\n"; - return true; // return false to cancel solve -}; -``` - -## Environment Variables - -| Variable | Default | Description | -|----------|---------|-------------| -| `CUOPT_REMOTE_HOST` | `localhost` | Server hostname for remote solves | -| `CUOPT_REMOTE_PORT` | `8765` | Server port for remote solves | -| `CUOPT_CHUNK_SIZE` | 16 MiB | Override `chunk_size_bytes` | -| `CUOPT_MAX_MESSAGE_BYTES` | 256 MiB | Override `max_message_bytes` | -| `CUOPT_GRPC_DEBUG` | `0` | Enable client debug/throughput logging (`0` or `1`) | -| `CUOPT_TLS_ENABLED` | `0` | Enable TLS for client connections (`0` or `1`) | -| `CUOPT_TLS_ROOT_CERT` | *(none)* | Path to PEM root CA file (server verification) | -| `CUOPT_TLS_CLIENT_CERT` | *(none)* | Path to PEM client certificate file (for mTLS) | -| `CUOPT_TLS_CLIENT_KEY` | *(none)* | Path to PEM client private key file (for mTLS) | - -## TLS Configuration - -### Server-Side TLS - -```bash -./cuopt_grpc_server --port 8765 \ - --tls \ - --tls-cert server.crt \ - --tls-key server.key -``` - -### Mutual TLS (mTLS) - -Server requires client certificate: - -```bash -./cuopt_grpc_server --port 8765 \ - --tls \ - --tls-cert server.crt \ - --tls-key server.key \ - --tls-root ca.crt \ - --require-client-cert -``` - -Client provides certificate via environment variables (applies to Python, `cuopt_cli`, and C API): - -```bash -export CUOPT_TLS_ENABLED=1 -export CUOPT_TLS_ROOT_CERT=ca.crt -export CUOPT_TLS_CLIENT_CERT=client.crt -export CUOPT_TLS_CLIENT_KEY=client.key -``` - -Or programmatically via `grpc_client_config_t`: - -```cpp -config.enable_tls = true; -config.tls_root_certs = read_file("ca.crt"); -config.tls_client_cert = read_file("client.crt"); -config.tls_client_key = read_file("client.key"); -``` - -## Message Size Limits - -| Configuration | Default | Notes | -|---------------|---------|-------| -| Server `--max-message-mb` | 256 MiB | Per-message limit (also `--max-message-bytes` for exact byte values) | -| Server clamping | [4 KiB, ~2 GiB] | Enforced at startup to stay within protobuf's serialization limit | -| Client `max_message_bytes` | 256 MiB | Clamped to [4 MiB, ~2 GiB] at construction | -| Chunk size | 16 MiB | Payload per `SendArrayChunk`/`GetResultChunk` | -| Chunked threshold | 75% of max_message_bytes | Problems above this use chunked upload (e.g. 192 MiB when max is 256 MiB) | - -Chunked transfer allows unlimited total payload size; only individual -chunks must fit within the per-message limit. Neither client nor server -allows "unlimited" message size — both clamp to the protobuf 2 GiB ceiling. - -## Security Notes - -1. **gRPC Layer**: All client-server message parsing uses protobuf-generated code -2. **Internal Pipe**: The server-to-worker pipe uses protobuf for metadata headers - and length-prefixed raw `read()`/`write()` for bulk array data. This pipe is - internal to the server process (main → forked worker) and not exposed to clients. -3. **Standard gRPC Security**: HTTP/2 framing, flow control, standard status codes -4. **TLS Support**: Optional encryption with mutual authentication -5. **Input Validation**: Server validates all incoming gRPC messages before processing - -## Data Flow Summary - -``` -┌─────────┐ ┌─────────────┐ -│ Client │ │ Server │ -│ │ SubmitJob (small) │ │ -│ problem ├───────────────────────────────────►│ deserialize │ -│ │ -or- Chunked Upload (large) │ ↓ │ -│ │ │ worker │ -│ │ │ process │ -│ │ GetResult (small) │ ↓ │ -│ solution│◄───────────────────────────────────┤ serialize │ -│ │ -or- Chunked Download (large) │ │ -└─────────┘ └─────────────┘ -``` - -See `GRPC_SERVER_ARCHITECTURE.md` for details on internal server architecture. - -## Code Generation - -The `cpp/codegen` directory (optional) generates conversion snippets from `field_registry.yaml`. Targets include: - -- **Settings**: PDLP/MIP settings ↔ proto (replacing hand-written blocks in the settings mapper). -- **Result header/scalars/arrays**: ChunkedResultHeader and array field handling. -- **Field element size**: `grpc_field_element_size.hpp` (ArrayFieldId → byte size). -- **Incumbent**: `grpc_incumbent_proto.hpp` (build/parse `Incumbent` messages). - -Adding or changing a proto field can be done via YAML and regenerate instead of editing mapper code by hand. - -## Build - -- **libcuopt**: Includes the mapper `.cpp` files, `grpc_client.cpp`, and `solve_remote.cpp`. Requires `CUOPT_ENABLE_GRPC`, gRPC, and protobuf. Proto generation is done by CMake custom commands that depend on the `.proto` files in `cpp/src/grpc/`. -- **cuopt_grpc_server**: Executable built from `cpp/src/grpc/server/*.cpp`; links libcuopt, gRPC, protobuf. - -Tests that use the client (e.g. `grpc_client_test.cpp`, `grpc_integration_test.cpp`) get `cpp/src/grpc` and `cpp/src/grpc/client` in their include path. diff --git a/GRPC_QUICK_START.md b/GRPC_QUICK_START.md index a3864c101..e9bcdc0da 100644 --- a/GRPC_QUICK_START.md +++ b/GRPC_QUICK_START.md @@ -18,13 +18,13 @@ same as a local solve — no API changes required. ### Basic (no TLS) ```bash -cuopt_grpc_server --port 8765 --workers 1 +cuopt_grpc_server --port 5001 --workers 1 ``` ### TLS (server authentication) ```bash -cuopt_grpc_server --port 8765 \ +cuopt_grpc_server --port 5001 \ --tls \ --tls-cert server.crt \ --tls-key server.key @@ -33,7 +33,7 @@ cuopt_grpc_server --port 8765 \ ### mTLS (mutual authentication) ```bash -cuopt_grpc_server --port 8765 \ +cuopt_grpc_server --port 5001 \ --tls \ --tls-cert server.crt \ --tls-key server.key \ @@ -123,7 +123,7 @@ openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \ **4. Start the server with your CA:** ```bash -cuopt_grpc_server --port 8765 \ +cuopt_grpc_server --port 5001 \ --tls \ --tls-cert server.crt \ --tls-key server.key \ @@ -135,7 +135,7 @@ cuopt_grpc_server --port 8765 \ ```bash export CUOPT_REMOTE_HOST=server.example.com -export CUOPT_REMOTE_PORT=8765 +export CUOPT_REMOTE_PORT=5001 export CUOPT_TLS_ENABLED=1 export CUOPT_TLS_ROOT_CERT=ca.crt # verifies the server export CUOPT_TLS_CLIENT_CERT=client.crt # proves client identity @@ -157,7 +157,7 @@ They apply identically to the Python API, `cuopt_cli`, and the C API. ```bash export CUOPT_REMOTE_HOST= -export CUOPT_REMOTE_PORT=8765 +export CUOPT_REMOTE_PORT=5001 ``` When both `CUOPT_REMOTE_HOST` and `CUOPT_REMOTE_PORT` are set, every diff --git a/GRPC_SERVER_ARCHITECTURE.md b/GRPC_SERVER_ARCHITECTURE.md index 2d6c2c324..81303e3bc 100644 --- a/GRPC_SERVER_ARCHITECTURE.md +++ b/GRPC_SERVER_ARCHITECTURE.md @@ -13,29 +13,29 @@ For gRPC protocol and client API, see `GRPC_INTERFACE.md`. Server source files l ## Process Model ```text -┌────────────────────────────────────────────────────────────────────┐ -│ Main Server Process │ -│ │ +┌─────────────────────────────────────────────────────────────────────┐ +│ Main Server Process │ +│ │ │ ┌─────────────┐ ┌──────────────┐ ┌─────────────────────────────┐ │ │ │ gRPC │ │ Job │ │ Background Threads │ │ │ │ Service │ │ Tracker │ │ - Result retrieval │ │ │ │ Handler │ │ (job status,│ │ - Incumbent retrieval │ │ │ │ │ │ results) │ │ - Worker monitor │ │ │ └─────────────┘ └──────────────┘ └─────────────────────────────┘ │ -│ │ ▲ │ -│ │ shared memory │ pipes │ -│ ▼ │ │ -│ ┌─────────────────────────────────────────────────────────────────┐│ -│ │ Shared Memory Queues ││ -│ │ ┌─────────────────┐ ┌─────────────────────┐ ││ -│ │ │ Job Queue │ │ Result Queue │ ││ -│ │ │ (MAX_JOBS=100) │ │ (MAX_RESULTS=100) │ ││ -│ │ └─────────────────┘ └─────────────────────┘ ││ -│ └─────────────────────────────────────────────────────────────────┘│ -└────────────────────────────────────────────────────────────────────┘ - │ ▲ - │ fork() │ - ▼ │ +│ │ ▲ │ +│ │ shared memory │ pipes │ +│ ▼ │ │ +│ ┌────────────────────────────────────────────────────────────────┐ │ +│ │ Shared Memory Queues │ │ +│ │ ┌─────────────────┐ ┌─────────────────────┐ │ │ +│ │ │ Job Queue │ │ Result Queue │ │ │ +│ │ │ (MAX_JOBS=100) │ │ (MAX_RESULTS=100) │ │ │ +│ │ └─────────────────┘ └─────────────────────┘ │ │ +│ └────────────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────┘ + │ ▲ + │ fork() │ + ▼ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ Worker 0 │ │ Worker 1 │ │ Worker N │ │ ┌───────────┐ │ │ ┌───────────┐ │ │ ┌───────────┐ │ @@ -86,7 +86,9 @@ All paths below are under `cpp/src/grpc/server/`. | `grpc_server_types.hpp` | Shared structs (e.g. `JobQueueEntry`, `ResultQueueEntry`, `ServerConfig`, `JobInfo`), enums, globals (atomics, mutexes, condition variables), and forward declarations used across server .cpp files. | | `grpc_field_element_size.hpp` | Maps `cuopt::remote::ArrayFieldId` to element byte size; used by pipe deserialization and chunked logic. | | `grpc_pipe_serialization.hpp` | Streaming pipe I/O: write/read individual length-prefixed protobuf messages (ChunkedProblemHeader, ChunkedResultHeader, ArrayChunk) directly to/from pipe fds. Avoids large intermediate buffers. Also serializes SubmitJobRequest for unary pipe transfer. | +| `grpc_pipe_io.cpp` | Low-level pipe read/write helpers: length-prefixed protobuf serialization, raw byte transfer with retry-on-EINTR, and pipe buffer sizing. | | `grpc_incumbent_proto.hpp` | Build `Incumbent` proto from (job_id, objective, assignment) and parse it back; used by worker when pushing incumbents and by main when reading from the incumbent pipe. | +| `grpc_server_logger.{hpp,cpp}` | Server-side logging utilities: log file management, console echo, and log message formatting for worker processes. | | `grpc_worker.cpp` | `worker_process(worker_index)`: loop over job queue, receive job data via pipe (unary or chunked), call solver, send result (and optionally incumbents) back. Contains `IncumbentPipeCallback` and `store_simple_result`. | | `grpc_worker_infra.cpp` | Pipe creation/teardown, `spawn_worker` / `spawn_workers`, `wait_for_workers`, `mark_worker_jobs_failed`, `cleanup_shared_memory`. | | `grpc_server_threads.cpp` | `worker_monitor_thread`, `result_retrieval_thread`, `incumbent_retrieval_thread`, `session_reaper_thread`. | @@ -115,7 +117,7 @@ Client Server Worker │─── SubmitJob ──────────►│ │ │ │ Create job entry │ │ │ Store problem data │ - │ │ job_queue[slot].ready=true│ + │ │ job_queue[slot].ready=true │ │◄── job_id ──────────────│ │ ``` @@ -132,8 +134,8 @@ Client Server Worker │ │ │ solve_lp/solve_mip │ │ │ Convert GPU→CPU │ │ │ - │ │ result_queue[slot].ready │◄────────────────── - │ │◄── result data via pipe ─│ + │ │ result_queue[slot].ready │ + │ │◄── result data via pipe ──│ ``` ### 3. Result Retrieval @@ -213,15 +215,15 @@ The `StreamLogs` RPC: ## Job States ```text -┌─────────┐ submit ┌───────────┐ claim ┌────────────┐ -│ QUEUED │──────────►│ PROCESSING│─────────►│ COMPLETED │ -└─────────┘ └───────────┘ └────────────┘ - │ │ - │ cancel │ error - ▼ ▼ -┌───────────┐ ┌─────────┐ -│ CANCELLED │ │ FAILED │ -└───────────┘ └─────────┘ +┌─────────┐ submit ┌────────────┐ claim ┌────────────┐ +│ QUEUED │──────────►│ PROCESSING │─────────►│ COMPLETED │ +└─────────┘ └────────────┘ └────────────┘ + │ │ + │ cancel │ error + ▼ ▼ +┌───────────┐ ┌─────────┐ +│ CANCELLED │ │ FAILED │ +└───────────┘ └─────────┘ ``` ## Configuration Options @@ -229,7 +231,7 @@ The `StreamLogs` RPC: ```bash cuopt_grpc_server [options] - -p, --port PORT gRPC listen port (default: 8765) + -p, --port PORT gRPC listen port (default: 5001) -w, --workers NUM Number of worker processes (default: 1) --max-message-mb N Max gRPC message size in MiB (default: 256; clamped to [4 KiB, ~2 GiB]) --max-message-bytes N Max gRPC message size in bytes (exact; min 4096) diff --git a/build.sh b/build.sh index 5f9ac4071..ff6850243 100755 --- a/build.sh +++ b/build.sh @@ -15,10 +15,11 @@ REPODIR=$(cd "$(dirname "$0")"; pwd) LIBCUOPT_BUILD_DIR=${LIBCUOPT_BUILD_DIR:=${REPODIR}/cpp/build} LIBMPS_PARSER_BUILD_DIR=${LIBMPS_PARSER_BUILD_DIR:=${REPODIR}/cpp/libmps_parser/build} -VALIDARGS="clean libcuopt cuopt_grpc_server libmps_parser cuopt_mps_parser cuopt cuopt_server cuopt_sh_client docs deb -a -b -g -fsanitize -tsan -msan -v -l= --verbose-pdlp --build-lp-only --no-fetch-rapids --skip-c-python-adapters --skip-tests-build --skip-routing-build --skip-fatbin-write --host-lineinfo [--cmake-args=\\\"\\\"] [--cache-tool=] -n --allgpuarch --ci-only-arch --show_depr_warn -h --help" +VALIDARGS="clean codegen libcuopt cuopt_grpc_server libmps_parser cuopt_mps_parser cuopt cuopt_server cuopt_sh_client docs deb -a -b -g -fsanitize -tsan -msan -v -l= --verbose-pdlp --build-lp-only --no-fetch-rapids --skip-c-python-adapters --skip-tests-build --skip-routing-build --skip-fatbin-write --host-lineinfo [--cmake-args=\\\"\\\"] [--cache-tool=] -n --allgpuarch --ci-only-arch --show_depr_warn -h --help" HELP="$0 [ ...] [ ...] where is: clean - remove all existing build artifacts and configuration (start over) + codegen - regenerate gRPC .inc files and proto from field_registry.yaml (requires pyyaml) libcuopt - build the cuopt C++ code cuopt_grpc_server - build only the gRPC server binary (configures + builds libcuopt as needed) libmps_parser - build the libmps_parser C++ code @@ -358,6 +359,18 @@ if buildAll || hasArg libmps_parser; then fi fi +################################################################################ +# Regenerate gRPC codegen .inc files from the field registry (explicit target only) +if hasArg codegen; then + echo "Regenerating codegen .inc files from field_registry.yaml..." + python "${REPODIR}"/cpp/codegen/generate_conversions.py \ + --registry "${REPODIR}"/cpp/codegen/field_registry.yaml \ + --output-dir "${REPODIR}"/cpp/codegen/generated + cp "${REPODIR}"/cpp/codegen/generated/cuopt_remote_data.proto \ + "${REPODIR}"/cpp/src/grpc/cuopt_remote_data.proto + echo "Done. Remember to commit the generated files." +fi + ################################################################################ # Configure and build libcuopt (and optionally just the gRPC server) if buildAll || hasArg libcuopt || hasArg cuopt_grpc_server; then diff --git a/ci/test_cpp.sh b/ci/test_cpp.sh index 653c44133..23be59fe6 100755 --- a/ci/test_cpp.sh +++ b/ci/test_cpp.sh @@ -1,6 +1,6 @@ #!/bin/bash -# SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2023-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 set -euo pipefail @@ -31,6 +31,9 @@ mkdir -p "${RAPIDS_TESTS_DIR}" rapids-print-env +rapids-logger "Verify codegen output matches committed files" +./ci/verify_codegen.sh + rapids-logger "Check GPU usage" nvidia-smi diff --git a/ci/verify_codegen.sh b/ci/verify_codegen.sh new file mode 100755 index 000000000..66f388f0e --- /dev/null +++ b/ci/verify_codegen.sh @@ -0,0 +1,57 @@ +#!/bin/bash +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Verify that committed codegen output matches what generate_conversions.py produces. +# Fails if a developer edited field_registry.yaml without re-running ./build.sh codegen. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +REPO_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" +CODEGEN_DIR="${REPO_DIR}/cpp/codegen" +GENERATED_DIR="${CODEGEN_DIR}/generated" +PROTO_DEST="${REPO_DIR}/cpp/src/grpc/cuopt_remote_data.proto" + +TMPDIR=$(mktemp -d) +trap 'rm -rf ${TMPDIR}' EXIT + +echo "Running code generator into temp directory..." +python "${CODEGEN_DIR}/generate_conversions.py" \ + --registry "${CODEGEN_DIR}/field_registry.yaml" \ + --output-dir "${TMPDIR}" + +echo "Comparing generated output with committed files..." + +FAILED=0 + +for f in "${TMPDIR}"/*; do + fname=$(basename "$f") + committed="${GENERATED_DIR}/${fname}" + if [ ! -f "${committed}" ]; then + echo "MISSING: ${committed} (new generated file not committed)" + FAILED=1 + continue + fi + if ! diff -q "$f" "${committed}" > /dev/null 2>&1; then + echo "MISMATCH: cpp/codegen/generated/${fname}" + diff -u "${committed}" "$f" | head -30 + FAILED=1 + fi +done + +if [ -f "${TMPDIR}/cuopt_remote_data.proto" ] && [ -f "${PROTO_DEST}" ]; then + if ! diff -q "${TMPDIR}/cuopt_remote_data.proto" "${PROTO_DEST}" > /dev/null 2>&1; then + echo "MISMATCH: cpp/src/grpc/cuopt_remote_data.proto (not copied from codegen)" + FAILED=1 + fi +fi + +if [ ${FAILED} -ne 0 ]; then + echo "" + echo "ERROR: Committed generated files are out of sync with field_registry.yaml." + echo "Run './build.sh codegen' and commit the results." + exit 1 +fi + +echo "OK: All generated files match field_registry.yaml." diff --git a/conda/environments/all_cuda-129_arch-aarch64.yaml b/conda/environments/all_cuda-129_arch-aarch64.yaml index 3f87fff34..8ff9d05ac 100644 --- a/conda/environments/all_cuda-129_arch-aarch64.yaml +++ b/conda/environments/all_cuda-129_arch-aarch64.yaml @@ -60,12 +60,14 @@ dependencies: - pytest-cov - pytest<9.0 - python>=3.11,<3.15 +- pyyaml - pyyaml>=6.0.0 - rapids-build-backend>=0.4.0,<0.5.0 - rapids-logger==0.2.*,>=0.0.0a0 - re2 - requests - rmm==26.4.*,>=0.0.0a0 +- ruamel.yaml>=0.18 - scikit-build-core>=0.11.0 - scipy>=1.14.1 - sphinx diff --git a/conda/environments/all_cuda-129_arch-x86_64.yaml b/conda/environments/all_cuda-129_arch-x86_64.yaml index 490e3798c..87be7d637 100644 --- a/conda/environments/all_cuda-129_arch-x86_64.yaml +++ b/conda/environments/all_cuda-129_arch-x86_64.yaml @@ -60,12 +60,14 @@ dependencies: - pytest-cov - pytest<9.0 - python>=3.11,<3.15 +- pyyaml - pyyaml>=6.0.0 - rapids-build-backend>=0.4.0,<0.5.0 - rapids-logger==0.2.*,>=0.0.0a0 - re2 - requests - rmm==26.4.*,>=0.0.0a0 +- ruamel.yaml>=0.18 - scikit-build-core>=0.11.0 - scipy>=1.14.1 - sphinx diff --git a/conda/environments/all_cuda-131_arch-aarch64.yaml b/conda/environments/all_cuda-131_arch-aarch64.yaml index bf7b0de73..d6270363c 100644 --- a/conda/environments/all_cuda-131_arch-aarch64.yaml +++ b/conda/environments/all_cuda-131_arch-aarch64.yaml @@ -60,12 +60,14 @@ dependencies: - pytest-cov - pytest<9.0 - python>=3.11,<3.15 +- pyyaml - pyyaml>=6.0.0 - rapids-build-backend>=0.4.0,<0.5.0 - rapids-logger==0.2.*,>=0.0.0a0 - re2 - requests - rmm==26.4.*,>=0.0.0a0 +- ruamel.yaml>=0.18 - scikit-build-core>=0.11.0 - scipy>=1.14.1 - sphinx diff --git a/conda/environments/all_cuda-131_arch-x86_64.yaml b/conda/environments/all_cuda-131_arch-x86_64.yaml index 6f554809b..992438297 100644 --- a/conda/environments/all_cuda-131_arch-x86_64.yaml +++ b/conda/environments/all_cuda-131_arch-x86_64.yaml @@ -60,12 +60,14 @@ dependencies: - pytest-cov - pytest<9.0 - python>=3.11,<3.15 +- pyyaml - pyyaml>=6.0.0 - rapids-build-backend>=0.4.0,<0.5.0 - rapids-logger==0.2.*,>=0.0.0a0 - re2 - requests - rmm==26.4.*,>=0.0.0a0 +- ruamel.yaml>=0.18 - scikit-build-core>=0.11.0 - scipy>=1.14.1 - sphinx diff --git a/conda/recipes/libcuopt/recipe.yaml b/conda/recipes/libcuopt/recipe.yaml index 789ab55c3..096fd1f63 100644 --- a/conda/recipes/libcuopt/recipe.yaml +++ b/conda/recipes/libcuopt/recipe.yaml @@ -74,6 +74,8 @@ cache: - make - ninja - git + - python + - pyyaml - tbb-devel - zlib - bzip2 diff --git a/cpp/CMakeLists.txt b/cpp/CMakeLists.txt index 0cf657530..20bea0858 100644 --- a/cpp/CMakeLists.txt +++ b/cpp/CMakeLists.txt @@ -305,8 +305,28 @@ else() endif() endif() -# Generate C++ code from cuopt_remote.proto (base message definitions) -set(PROTO_FILE "${CMAKE_CURRENT_SOURCE_DIR}/src/grpc/cuopt_remote.proto") +# Proto search paths: manual protos in src/grpc, generated data proto in codegen/generated +set(PROTO_PATH_MANUAL "${CMAKE_CURRENT_SOURCE_DIR}/src/grpc") +set(PROTO_PATH_GEN "${CMAKE_CURRENT_SOURCE_DIR}/codegen/generated") + +# Generate C++ code from cuopt_remote_data.proto (auto-generated data definitions) +set(DATA_PROTO_FILE "${PROTO_PATH_GEN}/cuopt_remote_data.proto") +set(DATA_PROTO_SRCS "${CMAKE_CURRENT_BINARY_DIR}/cuopt_remote_data.pb.cc") +set(DATA_PROTO_HDRS "${CMAKE_CURRENT_BINARY_DIR}/cuopt_remote_data.pb.h") + +add_custom_command( + OUTPUT "${DATA_PROTO_SRCS}" "${DATA_PROTO_HDRS}" + COMMAND ${_PROTOBUF_PROTOC} + ARGS --cpp_out ${CMAKE_CURRENT_BINARY_DIR} + --proto_path ${PROTO_PATH_GEN} + ${DATA_PROTO_FILE} + DEPENDS ${DATA_PROTO_FILE} + COMMENT "Generating C++ code from cuopt_remote_data.proto" + VERBATIM +) + +# Generate C++ code from cuopt_remote.proto (control/protocol messages, imports data proto) +set(PROTO_FILE "${PROTO_PATH_MANUAL}/cuopt_remote.proto") set(PROTO_SRCS "${CMAKE_CURRENT_BINARY_DIR}/cuopt_remote.pb.cc") set(PROTO_HDRS "${CMAKE_CURRENT_BINARY_DIR}/cuopt_remote.pb.h") @@ -314,15 +334,16 @@ add_custom_command( OUTPUT "${PROTO_SRCS}" "${PROTO_HDRS}" COMMAND ${_PROTOBUF_PROTOC} ARGS --cpp_out ${CMAKE_CURRENT_BINARY_DIR} - --proto_path ${CMAKE_CURRENT_SOURCE_DIR}/src/grpc + --proto_path ${PROTO_PATH_MANUAL} + --proto_path ${PROTO_PATH_GEN} ${PROTO_FILE} - DEPENDS ${PROTO_FILE} + DEPENDS ${PROTO_FILE} ${DATA_PROTO_FILE} COMMENT "Generating C++ code from cuopt_remote.proto" VERBATIM ) # Generate gRPC service code from cuopt_remote_service.proto -set(GRPC_PROTO_FILE "${CMAKE_CURRENT_SOURCE_DIR}/src/grpc/cuopt_remote_service.proto") +set(GRPC_PROTO_FILE "${PROTO_PATH_MANUAL}/cuopt_remote_service.proto") set(GRPC_PROTO_SRCS "${CMAKE_CURRENT_BINARY_DIR}/cuopt_remote_service.pb.cc") set(GRPC_PROTO_HDRS "${CMAKE_CURRENT_BINARY_DIR}/cuopt_remote_service.pb.h") set(GRPC_SERVICE_SRCS "${CMAKE_CURRENT_BINARY_DIR}/cuopt_remote_service.grpc.pb.cc") @@ -334,9 +355,10 @@ add_custom_command( ARGS --cpp_out ${CMAKE_CURRENT_BINARY_DIR} --grpc_out ${CMAKE_CURRENT_BINARY_DIR} --plugin=protoc-gen-grpc=${_GRPC_CPP_PLUGIN_EXECUTABLE} - --proto_path ${CMAKE_CURRENT_SOURCE_DIR}/src/grpc + --proto_path ${PROTO_PATH_MANUAL} + --proto_path ${PROTO_PATH_GEN} ${GRPC_PROTO_FILE} - DEPENDS ${GRPC_PROTO_FILE} ${PROTO_FILE} + DEPENDS ${GRPC_PROTO_FILE} ${PROTO_FILE} ${DATA_PROTO_FILE} COMMENT "Generating gRPC C++ code from cuopt_remote_service.proto" VERBATIM ) @@ -363,6 +385,7 @@ endif() # Add gRPC mapper files and generated protobuf sources list(APPEND CUOPT_SRC_FILES + ${DATA_PROTO_SRCS} ${PROTO_SRCS} ${GRPC_PROTO_SRCS} ${GRPC_SERVICE_SRCS} @@ -427,6 +450,7 @@ target_include_directories(cuopt "${CMAKE_CURRENT_SOURCE_DIR}/src" "${CMAKE_CURRENT_SOURCE_DIR}/src/grpc" "${CMAKE_CURRENT_SOURCE_DIR}/src/grpc/client" + "${CMAKE_CURRENT_SOURCE_DIR}/codegen/generated" "${CMAKE_CURRENT_BINARY_DIR}" "${CUDSS_INCLUDE}" PUBLIC @@ -751,6 +775,7 @@ target_include_directories(cuopt_grpc_server "${CMAKE_CURRENT_SOURCE_DIR}/src" "${CMAKE_CURRENT_SOURCE_DIR}/src/grpc" "${CMAKE_CURRENT_SOURCE_DIR}/src/grpc/server" + "${CMAKE_CURRENT_SOURCE_DIR}/codegen/generated" "${CMAKE_CURRENT_SOURCE_DIR}/include" "${CMAKE_CURRENT_SOURCE_DIR}/libmps_parser/include" "${CMAKE_CURRENT_BINARY_DIR}" diff --git a/cpp/codegen/field_registry.yaml b/cpp/codegen/field_registry.yaml new file mode 100644 index 000000000..c5806a1ea --- /dev/null +++ b/cpp/codegen/field_registry.yaml @@ -0,0 +1,637 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Field registry for auto-generating proto definitions and C++ conversion code. +# +# Convention-over-configuration: +# - Bare field name = double type, getter = get_() +# - "field: type" = specified type +# - "field: { type: T, ... }" = full override +# +# Scalar getters default to get_(). +# Array getters default to get__host() (for optimization_problem) or +# get__host() for solution arrays. +# Proto field name = registry field name (always). +# Enum conventions: see the "Enums" section below for full defaults. +# +# Field numbers: +# field_num – proto message field number (auto-assigned if missing) +# array_id – enum value for ArrayFieldId or ResultFieldId (auto-assigned) +# +# Numbering is contiguous within each proto message (no artificial caps). +# The only hard ranges are for solution scalars that share ChunkedResultHeader: +# LP: 1000-1999 +# MIP: 2000-2999 +# WS: 3000-3999 +# +# Attributes (all optional): +# +# Per-field: +# type – proto wire type (default: double for scalars, repeated double for arrays) +# field_num – proto message field number (auto-assigned if missing) +# array_id – enum value for ArrayFieldId or ResultFieldId (auto-assigned) +# setter_getter_root – C++ getter/setter root when different from field name +# getter – explicit C++ getter expression (overrides setter_getter_root) +# setter – explicit C++ setter name +# member – C++ member name or access path (when different from field name) +# setter_group – name of multi-argument setter group +# conditional – true => guard serialization with emptiness check +# skip_conversion – true => include in proto but not in conversion code +# proto_only – true => include in proto header but not a constructor arg +# sentinel – sentinel value handling for fields with "unset" semantics +# to_proto_cast – explicit cast type for C++ -> proto direction +# from_proto_cast – explicit cast type for proto -> C++ direction +# +# Per-section: +# cpp_type – C++ type this section maps to +# constructor_args – mapping of constructor parameter names to field sources +# presence_check – predicate expression to test if optional data is present +# getter – (per-section) expression to access the section's C++ data +# +# See codegen/README.md for the full specification and examples. + +# ───────────────────────────────────────────────────────────────────────────── +# Enums +# ───────────────────────────────────────────────────────────────────────────── +# +# Convention-based defaults (override any field explicitly when needed): +# cpp_type — {key}_t +# proto_type — PascalCase from key (acronyms PDLP/MIP/LP/QP/VRP/PDP uppercased) +# to_proto_fn — to_proto_{key}() +# from_proto — from_proto_{key}() +# default — first value (proto3 zero-value) +# values — bare names auto-number from 0; {Name: N} resets counter to N +# proto_prefix — "" (no prefix); set when proto values need disambiguation +# +# Proto value names: proto_prefix + UPPER_SNAKE(CppName), or just CppName if +# no prefix. +# +enums: + pdlp_termination_status: + domain: solution + proto_prefix: PDLP + values: + - NoTermination + - NumericalError + - Optimal + - PrimalInfeasible + - DualInfeasible + - IterationLimit + - TimeLimit + - ConcurrentLimit + - PrimalFeasible + + mip_termination_status: + domain: solution + proto_prefix: MIP + values: + - NoTermination + - Optimal + - FeasibleFound + - Infeasible + - Unbounded + - TimeLimit + - WorkLimit + + pdlp_solver_mode: + domain: settings + default: Stable3 + values: + - Stable1 + - Stable2 + - Methodical1 + - Fast1 + - Stable3 + + lp_method: + domain: settings + cpp_type: method_t + values: + - Concurrent + - PDLP + - DualSimplex + - Barrier + + variable_type: + domain: problem + cpp_type: var_t + values: + - CONTINUOUS + - INTEGER + + problem_category: + domain: problem + values: + - LP + - MIP + +# ───────────────────────────────────────────────────────────────────────────── +# LP Solution +# ───────────────────────────────────────────────────────────────────────────── +lp_solution: + cpp_type: "cpu_lp_solution_t" + + scalars: + - lp_termination_status: + field_num: 1000 + type: pdlp_termination_status + getter: get_termination_status() + - error_message: + field_num: 1001 + type: string + getter: "get_error_status().what()" + proto_only: true + - l2_primal_residual: + field_num: 1002 + - l2_dual_residual: + field_num: 1003 + - primal_objective: + field_num: 1004 + getter: get_objective_value() + - dual_objective: + field_num: 1005 + getter: get_dual_objective_value() + - gap: + field_num: 1006 + - nb_iterations: + field_num: 1007 + type: int32 + getter: get_num_iterations() + - solve_time: + field_num: 1008 + - solved_by_pdlp: + field_num: 1009 + type: bool + getter: is_solved_by_pdlp() + + arrays: + - primal_solution: + field_num: 1 + array_id: 0 + - dual_solution: + field_num: 2 + array_id: 1 + - reduced_cost: + field_num: 3 + array_id: 2 + + constructor_args: + scalars: + - lp_termination_status + - primal_objective + - dual_objective + - solve_time + - l2_primal_residual + - l2_dual_residual + - gap + - nb_iterations + - solved_by_pdlp + + warm_start: + presence_check: has_warm_start_data() + getter: get_cpu_pdlp_warm_start_data() + + scalars: + - initial_primal_weight_: + field_num: 3000 + - initial_step_size_: + field_num: 3001 + - total_pdlp_iterations_: + field_num: 3002 + type: int32 + - total_pdhg_iterations_: + field_num: 3003 + type: int32 + - last_candidate_kkt_score_: + field_num: 3004 + - last_restart_kkt_score_: + field_num: 3005 + - sum_solution_weight_: + field_num: 3006 + - iterations_since_last_restart_: + field_num: 3007 + type: int32 + + arrays: + - current_primal_solution_: + field_num: 1 + array_id: 3 + - current_dual_solution_: + field_num: 2 + array_id: 4 + - initial_primal_average_: + field_num: 3 + array_id: 5 + - initial_dual_average_: + field_num: 4 + array_id: 6 + - current_ATY_: + field_num: 5 + array_id: 7 + - sum_primal_solutions_: + field_num: 6 + array_id: 8 + - sum_dual_solutions_: + field_num: 7 + array_id: 9 + - last_restart_duality_gap_primal_solution_: + field_num: 8 + array_id: 10 + - last_restart_duality_gap_dual_solution_: + field_num: 9 + array_id: 11 + +# ───────────────────────────────────────────────────────────────────────────── +# MIP Solution +# ───────────────────────────────────────────────────────────────────────────── +mip_solution: + cpp_type: "cpu_mip_solution_t" + + scalars: + - mip_termination_status: + field_num: 2000 + type: mip_termination_status + getter: get_termination_status() + - mip_error_message: + field_num: 2001 + type: string + getter: "get_error_status().what()" + proto_only: true + - mip_objective: + field_num: 2002 + getter: get_objective_value() + - mip_gap: + field_num: 2003 + - solution_bound: + field_num: 2004 + - total_solve_time: + field_num: 2005 + getter: get_solve_time() + - presolve_time: + field_num: 2006 + - max_constraint_violation: + field_num: 2007 + - max_int_violation: + field_num: 2008 + - max_variable_bound_violation: + field_num: 2009 + - nodes: + field_num: 2010 + type: int32 + getter: get_num_nodes() + - simplex_iterations: + field_num: 2011 + type: int32 + getter: get_num_simplex_iterations() + + arrays: + - mip_solution: + field_num: 1 + array_id: 12 + getter: get_solution_host() + + constructor_args: + scalars: + - mip_termination_status + - mip_objective + - mip_gap + - solution_bound + - total_solve_time + - presolve_time + - max_constraint_violation + - max_int_violation + - max_variable_bound_violation + - nodes + - simplex_iterations + +# ───────────────────────────────────────────────────────────────────────────── +# PDLP Solver Settings +# ───────────────────────────────────────────────────────────────────────────── +# +# Settings use nested YAML to mirror C++ struct hierarchy. +# A list-valued entry is a sub-struct group (e.g. tolerances). +# +pdlp_settings: + cpp_type: "pdlp_solver_settings_t" + proto_type: "cuopt::remote::PDLPSolverSettings" + + fields: + # Termination tolerances (nested: settings.tolerances.) + - tolerances: + - absolute_gap_tolerance: + field_num: 1 + - relative_gap_tolerance: + field_num: 2 + - primal_infeasible_tolerance: + field_num: 3 + - dual_infeasible_tolerance: + field_num: 4 + - absolute_dual_tolerance: + field_num: 5 + - relative_dual_tolerance: + field_num: 6 + - absolute_primal_tolerance: + field_num: 7 + - relative_primal_tolerance: + field_num: 8 + + # Limits + - time_limit: + field_num: 9 + - iteration_limit: + field_num: 10 + type: int64 + sentinel: + to_proto: "std::numeric_limits::max()" + proto_value: -1 + from_proto_guard: ">= 0" + from_proto_cast: "i_t" + + # Solver configuration + - log_to_console: + field_num: 11 + type: bool + - detect_infeasibility: + field_num: 12 + type: bool + - strict_infeasibility: + field_num: 13 + type: bool + - pdlp_solver_mode: + field_num: 14 + type: pdlp_solver_mode + - method: + field_num: 15 + type: lp_method + - presolver: + field_num: 16 + type: int32 + to_proto_cast: "int32_t" + from_proto_cast: "presolver_t" + - dual_postsolve: + field_num: 17 + type: bool + - crossover: + field_num: 18 + type: bool + - num_gpus: + field_num: 19 + type: int32 + + - per_constraint_residual: + field_num: 20 + type: bool + - cudss_deterministic: + field_num: 21 + type: bool + - folding: + field_num: 22 + type: int32 + - augmented: + field_num: 23 + type: int32 + - dualize: + field_num: 24 + type: int32 + - ordering: + field_num: 25 + type: int32 + - barrier_dual_initial_point: + field_num: 26 + type: int32 + - eliminate_dense_columns: + field_num: 27 + type: bool + - save_best_primal_so_far: + field_num: 28 + type: bool + - first_primal_feasible: + field_num: 29 + type: bool + - pdlp_precision: + field_num: 30 + type: int32 + to_proto_cast: "int32_t" + from_proto_cast: "pdlp_precision_t" + +# ───────────────────────────────────────────────────────────────────────────── +# MIP Solver Settings +# ───────────────────────────────────────────────────────────────────────────── +mip_settings: + cpp_type: "mip_solver_settings_t" + proto_type: "cuopt::remote::MIPSolverSettings" + + fields: + # Limits + - time_limit: + field_num: 1 + + # Tolerances (nested: settings.tolerances.) + - tolerances: + - relative_mip_gap: + field_num: 2 + - absolute_mip_gap: + field_num: 3 + - integrality_tolerance: + field_num: 4 + - absolute_tolerance: + field_num: 5 + - relative_tolerance: + field_num: 6 + - presolve_absolute_tolerance: + field_num: 7 + + # Solver configuration + - log_to_console: + field_num: 8 + type: bool + - heuristics_only: + field_num: 9 + type: bool + - num_cpu_threads: + field_num: 10 + type: int32 + - num_gpus: + field_num: 11 + type: int32 + - presolver: + field_num: 12 + type: int32 + to_proto_cast: "int32_t" + from_proto_cast: "presolver_t" + - mip_scaling: + field_num: 13 + type: bool + + # Additional limits + - work_limit: + field_num: 14 + - node_limit: + field_num: 15 + type: int32 + sentinel: + to_proto: "std::numeric_limits::max()" + proto_value: -1 + from_proto_guard: ">= 0" + from_proto_cast: "i_t" + + # Branching + - reliability_branching: + field_num: 16 + type: int32 + - mip_batch_pdlp_strong_branching: + field_num: 17 + type: int32 + + # Cut configuration + - max_cut_passes: + field_num: 18 + type: int32 + - mir_cuts: + field_num: 19 + type: int32 + - mixed_integer_gomory_cuts: + field_num: 20 + type: int32 + - knapsack_cuts: + field_num: 21 + type: int32 + - clique_cuts: + field_num: 22 + type: int32 + - strong_chvatal_gomory_cuts: + field_num: 23 + type: int32 + - reduced_cost_strengthening: + field_num: 24 + type: int32 + - cut_change_threshold: + field_num: 25 + - cut_min_orthogonality: + field_num: 26 + + # Determinism and reproducibility + - determinism_mode: + field_num: 27 + type: int32 + - seed: + field_num: 28 + type: int32 + +# ───────────────────────────────────────────────────────────────────────────── +# Optimization Problem (cpu_optimization_problem_t) +# ───────────────────────────────────────────────────────────────────────────── +optimization_problem: + cpp_type: "cpu_optimization_problem_t" + proto_message: OptimizationProblem + + scalars: + - problem_name: + field_num: 1 + type: string + - objective_name: + field_num: 2 + type: string + - maximize: + field_num: 3 + type: bool + getter: get_sense() + - objective_scaling_factor: + field_num: 4 + - objective_offset: + field_num: 5 + - problem_category: + field_num: 6 + type: problem_category + + arrays: + - variable_names: + field_num: 7 + array_id: 0 + type: repeated string + - row_names: + field_num: 8 + array_id: 1 + type: repeated string + - A_values: + field_num: 9 + array_id: 2 + setter_getter_root: constraint_matrix_values + setter_group: csr_constraint_matrix + - A_indices: + field_num: 10 + array_id: 3 + setter_getter_root: constraint_matrix_indices + type: repeated int32 + setter_group: csr_constraint_matrix + - A_offsets: + field_num: 11 + array_id: 4 + setter_getter_root: constraint_matrix_offsets + type: repeated int32 + setter_group: csr_constraint_matrix + - c: + field_num: 12 + array_id: 5 + setter_getter_root: objective_coefficients + - b: + field_num: 13 + array_id: 6 + setter_getter_root: constraint_bounds + conditional: true + - variable_lower_bounds: + field_num: 14 + array_id: 7 + - variable_upper_bounds: + field_num: 15 + array_id: 8 + - constraint_lower_bounds: + field_num: 16 + array_id: 9 + conditional: true + - constraint_upper_bounds: + field_num: 17 + array_id: 10 + conditional: true + - row_types: + field_num: 18 + array_id: 11 + type: bytes + conditional: true + - variable_types: + field_num: 19 + array_id: 12 + type: repeated variable_type + - initial_primal_solution: + field_num: 20 + array_id: 13 + skip_conversion: true + - initial_dual_solution: + field_num: 21 + array_id: 14 + skip_conversion: true + - Q_values: + field_num: 22 + array_id: 15 + setter_getter_root: quadratic_objective_values + setter_group: quadratic_objective + - Q_indices: + field_num: 23 + array_id: 16 + setter_getter_root: quadratic_objective_indices + type: repeated int32 + setter_group: quadratic_objective + - Q_offsets: + field_num: 24 + array_id: 17 + setter_getter_root: quadratic_objective_offsets + type: repeated int32 + setter_group: quadratic_objective + + setter_groups: + csr_constraint_matrix: + setter: set_csr_constraint_matrix + fields: [A_values, A_indices, A_offsets] + quadratic_objective: + setter: set_quadratic_objective_matrix + fields: [Q_values, Q_indices, Q_offsets] diff --git a/cpp/codegen/generate_conversions.py b/cpp/codegen/generate_conversions.py new file mode 100644 index 000000000..4350dd9db --- /dev/null +++ b/cpp/codegen/generate_conversions.py @@ -0,0 +1,2446 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. +# SPDX-License-Identifier: Apache-2.0 +# All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +""" +Code generator for proto definitions, enum converters, and settings, problem, solution conversions. + +Reads field_registry.yaml and generates: + 1. cuopt_remote_data.proto — full proto: enums, OptimizationProblem, settings, solutions, + ChunkedResultHeader + 2. Enum converter C++ switch functions (.inc) + 3. Settings conversion C++ function bodies (.inc) + 4. Solution conversion C++ function bodies (.inc) — all 8 solution mapper functions + 5. Problem conversion C++ function bodies (.inc) + +Usage: + python generate_conversions.py [--registry field_registry.yaml] [--output-dir .] [--auto-number] +""" + +import argparse +import os +import re +import sys +from pathlib import Path + +import yaml + +# ============================================================================ +# Utilities +# ============================================================================ + +HEADER = """\ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ +""" + + +def write_file(path, content): + os.makedirs(os.path.dirname(path) or ".", exist_ok=True) + with open(path, "w") as f: + f.write(content) + print(f" wrote {path}") + + +def parse_field(entry): + """Parse a field entry from the YAML, which may be a bare string or a dict. + + Does NOT set a default 'type' — callers supply context-appropriate defaults + (e.g. "double" for scalars, "repeated double" for arrays). + """ + if isinstance(entry, str): + return {"name": entry} + if isinstance(entry, dict): + assert len(entry) == 1, f"Expected single-key dict, got {entry}" + name = list(entry.keys())[0] + val = entry[name] + if isinstance(val, str): + return {"name": name, "type": val} + if isinstance(val, dict): + result = {"name": name} + result.update(val) + return result + if val is None: + return {"name": name} + raise ValueError(f"Unexpected field format: {entry}") + + +def parse_settings_fields(entries, prefix=""): + """Parse settings fields, handling nested sub-structs (e.g. tolerances).""" + for entry in entries: + assert isinstance(entry, dict) and len(entry) == 1 + name = list(entry.keys())[0] + val = entry[name] + if isinstance(val, list): + yield from parse_settings_fields(val, prefix=f"{prefix}{name}.") + else: + f = parse_field(entry) + if prefix or "member" not in f: + f["member"] = f"{prefix}{name}" if prefix else name + yield f + + +def parse_enum_entry(entry, index=0): + """Parse a single enum value entry. + + Supports three forms: + - bare string 'CppName' → (name, index) + - {CppName: null} → (name, index) + - {CppName: number} → (name, number) + + The caller is responsible for tracking the running counter; explicit values + reset it (C-style enum semantics). + """ + if isinstance(entry, str): + return entry, index + assert isinstance(entry, dict) and len(entry) == 1 + name = list(entry.keys())[0] + num = entry[name] + return name, num if num is not None else index + + +def parse_enum_values(values): + """Parse a full enum values list with C-style auto-numbering. + + Bare names get the next sequential number; explicit {Name: N} overrides + reset the counter so the next bare name gets N+1. + """ + counter = 0 + result = [] + for entry in values: + name, num = parse_enum_entry(entry, index=counter) + result.append((name, num)) + counter = num + 1 + return result + + +def camel_to_upper_snake(name): + """Convert CamelCase to UPPER_SNAKE_CASE: PrimalInfeasible -> PRIMAL_INFEASIBLE.""" + s = re.sub(r"([a-z0-9])([A-Z])", r"\1_\2", name) + s = re.sub(r"([A-Z]+)([A-Z][a-z])", r"\1_\2", s) + return s.upper() + + +def proto_type(t): + """Map YAML type to proto type.""" + return { + "double": "double", + "float": "float", + "int32": "int32", + "int64": "int64", + "bool": "bool", + "string": "string", + "bytes": "bytes", + }.get(t, t) + + +def _from_proto_cast(ftype): + """Return C++ cast for reading a proto field back into template type.""" + if ftype in ("double", "float"): + return "f_t" + if ftype in ("int32", "int64"): + return "i_t" + return None + + +def _array_element_cast(ftype): + """Cast for serializing array elements to proto (to_proto direction).""" + if "double" in ftype: + return "double" + if "int32" in ftype: + return "int32_t" + return None + + +def _array_cpp_type(ftype): + """C++ vector element type for proto-to-C++ direction.""" + if "double" in ftype: + return "f_t" + if "int32" in ftype: + return "i_t" + return None + + +def _default_element_size(f): + """Default element byte size for a field.""" + if "element_size" in f: + return f["element_size"] + ftype = f.get("type", "repeated double") + if "int32" in ftype: + return 4 + if ftype in ("bytes", "repeated string"): + return 1 + if ftype.startswith("repeated ") and ftype.split()[-1] not in ( + "double", + "int32", + "string", + ): + return 4 + return 8 + + +# ============================================================================ +# Enum helpers — convention-based derivation +# ============================================================================ + +KNOWN_ACRONYMS = {"pdlp", "mip", "lp", "qp", "vrp", "pdp", "tsp"} + + +def _snake_to_pascal(key): + """Convert snake_case key to PascalCase, uppercasing known acronyms. + + pdlp_termination_status -> PDLPTerminationStatus + lp_method -> LPMethod + variable_type -> VariableType + """ + return "".join( + part.upper() if part in KNOWN_ACRONYMS else part.capitalize() + for part in key.split("_") + ) + + +def _enum_proto_type(key, edef): + """Derive proto enum type name: explicit or PascalCase from key.""" + return edef.get("proto_type", _snake_to_pascal(key)) + + +def _enum_cpp_type(key, edef): + """Derive C++ type: explicit or {key}_t.""" + return edef.get("cpp_type", f"{key}_t") + + +def _enum_default(key, edef): + """Derive default value: explicit or first value name (proto3 zero-value).""" + if "default" in edef: + return edef["default"] + first = edef["values"][0] + name, _ = parse_enum_entry(first) + return name + + +def _enum_to_proto_fn(key, _edef): + """Derive to_proto function name: to_proto_{key}.""" + return f"to_proto_{key}" + + +def _enum_from_proto_fn(key, _edef): + """Derive from_proto function name: from_proto_{key}.""" + return f"from_proto_{key}" + + +def _proto_enum_value_name(cpp_name, prefix): + """Derive proto enum value name from C++ name and prefix.""" + if not prefix: + return cpp_name + return f"{prefix}_{camel_to_upper_snake(cpp_name)}" + + +def _lookup_enum(registry, type_name): + """Find an enum definition in the registry by key name.""" + return registry.get("enums", {}).get(type_name) + + +def _is_enum_type(registry, type_name): + """Check if a type name refers to a registered enum.""" + return type_name in registry.get("enums", {}) + + +def _is_repeated_enum(registry, ftype): + """Check if a type is 'repeated ' and return enum key if so.""" + if ftype.startswith("repeated "): + inner = ftype.split(None, 1)[1] + if _is_enum_type(registry, inner): + return inner + return None + + +def _proto_cpp_name(name): + """Protobuf lowercases all field names for C++ accessor methods.""" + return name.lower() + + +# ============================================================================ +# Problem field helpers +# ============================================================================ + + +def _problem_getter_root(f): + """Get the getter root for a problem field (setter_getter_root or name).""" + return f.get("setter_getter_root", f["name"]) + + +def _default_problem_getter(f, is_scalar): + """Default getter for an optimization_problem field.""" + if "getter" in f: + return f["getter"] + root = _problem_getter_root(f) + if is_scalar: + return f"get_{root}()" + ftype = f.get("type", "repeated double") + if ftype == "repeated string": + return f"get_{root}()" + return f"get_{root}_host()" + + +def _default_problem_setter(f): + """Default setter name for an optimization_problem field.""" + if "setter" in f: + return f["setter"] + root = _problem_getter_root(f) + return f"set_{root}" + + +def _find_problem_field(obj, name): + """Find a field by name in the optimization_problem arrays list.""" + for entry in obj.get("arrays", []): + f = parse_field(entry) + if f["name"] == name: + return f + return None + + +def _field_array_id_name(f): + """Derive ArrayFieldId enum name from field name: FIELD_{UPPER_SNAKE(name)}.""" + return f"FIELD_{f['name'].upper()}" + + +def _field_result_id_name(f): + """Derive ResultFieldId enum name from field name: RESULT_{UPPER_SNAKE(name)}.""" + return f"RESULT_{f['name'].upper()}" + + +def _problem_field_proto_type(registry, ftype): + """Map optimization_problem field type to proto type.""" + if not ftype: + return "double" + enum_key = _is_repeated_enum(registry, ftype) + if enum_key: + edef = _lookup_enum(registry, enum_key) + return f"repeated {_enum_proto_type(enum_key, edef)}" + if ftype in ( + "repeated double", + "repeated int32", + "repeated string", + "bytes", + ): + return ftype + edef = _lookup_enum(registry, ftype) + if edef: + return _enum_proto_type(ftype, edef) + return proto_type(ftype) + + +def _settings_field_proto_type(registry, f): + """Map settings field type to proto type.""" + ftype = f.get("type", "double") + edef = _lookup_enum(registry, ftype) + if edef: + return _enum_proto_type(ftype, edef) + return proto_type(ftype) + + +def _solution_scalar_proto_type(registry, f): + """Map solution scalar type to proto type.""" + ftype = f.get("type", "double") + edef = _lookup_enum(registry, ftype) + if edef: + return _enum_proto_type(ftype, edef) + return proto_type(ftype) + + +# ============================================================================ +# Enum generation +# ============================================================================ + + +def generate_enum_proto_from_registry(registry): + """Generate all enum definitions for the proto file.""" + enums = registry.get("enums", {}) + blocks = [] + for key, edef in enums.items(): + proto_type_name = _enum_proto_type(key, edef) + prefix = edef.get("proto_prefix", "") + lines = [f"enum {proto_type_name} {{"] + for cpp_name, num in parse_enum_values(edef["values"]): + pname = _proto_enum_value_name(cpp_name, prefix) + lines.append(f" {pname} = {num};") + lines.append("}") + blocks.append("\n".join(lines)) + return "\n\n".join(blocks) + + +def generate_enum_converters_inc(registry, domain=None): + """Generate C++ to_proto/from_proto switch functions for enums. + + If domain is given, only emit converters for enums with that domain tag. + If domain is None, emit all converters (backward compat). + """ + enums = registry.get("enums", {}) + funcs = [] + for key, edef in enums.items(): + if domain is not None and edef.get("domain") != domain: + continue + cpp_type = _enum_cpp_type(key, edef) + proto_type_name = _enum_proto_type(key, edef) + to_fn = _enum_to_proto_fn(key, edef) + from_fn = _enum_from_proto_fn(key, edef) + prefix = edef.get("proto_prefix", "") + default_cpp = _enum_default(key, edef) + default_proto = _proto_enum_value_name(default_cpp, prefix) + + # to_proto + lines = [ + f"cuopt::remote::{proto_type_name} {to_fn}({cpp_type} v)", + "{", + " switch (v) {", + ] + for cpp_name, _ in parse_enum_values(edef["values"]): + pname = _proto_enum_value_name(cpp_name, prefix) + lines.append( + f" case {cpp_type}::{cpp_name}: return cuopt::remote::{pname};" + ) + lines.append(f" default: return cuopt::remote::{default_proto};") + lines.extend([" }", "}"]) + funcs.append("\n".join(lines)) + + # from_proto + lines = [ + f"{cpp_type} {from_fn}(cuopt::remote::{proto_type_name} v)", + "{", + " switch (v) {", + ] + for cpp_name, _ in parse_enum_values(edef["values"]): + pname = _proto_enum_value_name(cpp_name, prefix) + lines.append( + f" case cuopt::remote::{pname}: return {cpp_type}::{cpp_name};" + ) + lines.append(f" default: return {cpp_type}::{default_cpp};") + lines.extend([" }", "}"]) + funcs.append("\n".join(lines)) + + return "\n\n".join(funcs) + + +# ============================================================================ +# Proto file generation — ResultFieldId, ArrayFieldId, messages +# ============================================================================ + + +def generate_proto_result_enums(registry): + """Generate ResultFieldId enum entries (sorted by array_id).""" + entries = [] + for obj_name in ["lp_solution", "mip_solution"]: + obj = registry.get(obj_name, {}) + for entry in obj.get("arrays", []): + f = parse_field(entry) + enum_id = _field_result_id_name(f) + aid = f.get("array_id") + if aid is not None: + entries.append((aid, f" {enum_id} = {aid};")) + ws = obj.get("warm_start", {}) + for entry in ws.get("arrays", []): + f = parse_field(entry) + enum_id = _field_result_id_name(f) + aid = f.get("array_id") + if aid is not None: + entries.append((aid, f" {enum_id} = {aid};")) + entries.sort(key=lambda x: x[0]) + seen, lines = set(), [] + for _, line in entries: + eid = line.split(" = ")[0].strip() + if eid not in seen: + lines.append(line) + seen.add(eid) + return "\n".join(lines) + + +def generate_array_field_id_enum(registry): + """Generate ArrayFieldId enum for the proto file.""" + obj = registry.get("optimization_problem", {}) + entries = [] + for entry in obj.get("arrays", []): + f = parse_field(entry) + afid = _field_array_id_name(f) + afnum = f.get("array_id") + if afnum is not None: + entries.append((afnum, f" {afid} = {afnum};")) + entries.sort(key=lambda x: x[0]) + return "\n".join(e[1] for e in entries) + + +def generate_array_field_element_size_inc(registry): + """Generate body of array_field_element_size() switch function.""" + obj = registry.get("optimization_problem", {}) + cases_by_size = {} + for entry in obj.get("arrays", []): + f = parse_field(entry) + afid = _field_array_id_name(f) + size = _default_element_size(f) + cases_by_size.setdefault(size, []).append(afid) + lines = [" switch (field_id) {"] + for size in sorted(cases_by_size.keys()): + if size == 8: + continue + for afid in cases_by_size[size]: + lines.append(f" case cuopt::remote::{afid}:") + lines.append(f" return {size};") + lines.append(" default: return 8;") + lines.append(" }") + return "\n".join(lines) + + +# ============================================================================ +# Settings proto + conversion generation +# ============================================================================ + + +def generate_settings_message_proto(registry, message_name, obj): + lines = [] + for f in parse_settings_fields(obj.get("fields", [])): + num = f.get("field_num") + if num is None: + continue + ptype = _settings_field_proto_type(registry, f) + lines.append((num, f" {ptype} {f['name']} = {num};")) + if message_name == "PDLPSolverSettings": + lines.append((50, " PDLPWarmStartData warm_start_data = 50;")) + lines.sort(key=lambda x: x[0]) + return "\n".join(item[1] for item in lines) + + +def generate_settings_to_proto_body(registry, obj_name, obj, indent=" "): + lines, ind = [], indent + for f in parse_settings_fields(obj.get("fields", [])): + name, ftype = f["name"], f.get("type", "double") + pname = _proto_cpp_name(name) + cpp_member = f.get("member", f["name"]) + sentinel = f.get("sentinel") + to_proto_cast = f.get("to_proto_cast") + + edef = _lookup_enum(registry, ftype) + to_fn = _enum_to_proto_fn(ftype, edef) if edef else None + + if sentinel: + sv, pv = sentinel["to_proto"], sentinel["proto_value"] + lines.append(f"{ind}if (settings.{cpp_member} == {sv}) {{") + lines.append(f"{ind} pb_settings->set_{pname}({pv});") + lines.append(f"{ind}}} else {{") + cast = to_proto_cast or ("int64_t" if ftype == "int64" else None) + expr = ( + f"static_cast<{cast}>(settings.{cpp_member})" + if cast + else f"settings.{cpp_member}" + ) + lines.append(f"{ind} pb_settings->set_{pname}({expr});") + lines.append(f"{ind}}}") + elif to_fn: + lines.append( + f"{ind}pb_settings->set_{pname}({to_fn}(settings.{cpp_member}));" + ) + elif to_proto_cast: + lines.append( + f"{ind}pb_settings->set_{pname}(static_cast<{to_proto_cast}>(settings.{cpp_member}));" + ) + else: + lines.append( + f"{ind}pb_settings->set_{pname}(settings.{cpp_member});" + ) + return "\n".join(lines) + + +def generate_proto_to_settings_body(registry, obj_name, obj, indent=" "): + lines, ind = [], indent + for f in parse_settings_fields(obj.get("fields", [])): + name, ftype = f["name"], f.get("type", "double") + pname = _proto_cpp_name(name) + cpp_member = f.get("member", f["name"]) + from_proto_cast = f.get("from_proto_cast") + sentinel = f.get("sentinel") + + edef = _lookup_enum(registry, ftype) + from_fn = _enum_from_proto_fn(ftype, edef) if edef else None + + if sentinel: + guard = sentinel["from_proto_guard"] + cast = sentinel.get("from_proto_cast", from_proto_cast) + lines.append(f"{ind}if (pb_settings.{pname}() {guard}) {{") + expr = ( + f"static_cast<{cast}>(pb_settings.{pname}())" + if cast + else f"pb_settings.{pname}()" + ) + lines.append(f"{ind} settings.{cpp_member} = {expr};") + lines.append(f"{ind}}}") + elif from_fn: + lines.append( + f"{ind}settings.{cpp_member} = {from_fn}(pb_settings.{pname}());" + ) + elif from_proto_cast: + lines.append( + f"{ind}settings.{cpp_member} = static_cast<{from_proto_cast}>(pb_settings.{pname}());" + ) + else: + lines.append( + f"{ind}settings.{cpp_member} = pb_settings.{pname}();" + ) + return "\n".join(lines) + + +# ============================================================================ +# Proto message generation — WarmStart, LPSolution, MIPSolution +# ============================================================================ + + +def generate_warm_start_message_proto(registry): + obj = registry.get("lp_solution", {}).get("warm_start", {}) + lines = [] + for entry in obj.get("arrays", []): + f = parse_field(entry) + num = f.get("field_num") + if num is not None: + lines.append((num, f" repeated double {f['name']} = {num};")) + for entry in obj.get("scalars", []): + f = parse_field(entry) + num = f.get("field_num") + if num is not None: + lines.append( + ( + num, + f" {proto_type(f.get('type', 'double'))} {f['name']} = {num};", + ) + ) + lines.sort(key=lambda x: x[0]) + return "\n".join(item[1] for item in lines) + + +def generate_lp_solution_message_proto(registry): + obj = registry.get("lp_solution", {}) + lines = [] + for entry in obj.get("arrays", []): + f = parse_field(entry) + num = f.get("field_num") + if num is not None: + lines.append((num, f" repeated double {f['name']} = {num};")) + lines.append((4, " PDLPWarmStartData warm_start_data = 4;")) + for entry in obj.get("scalars", []): + f = parse_field(entry) + num = f.get("field_num") + if num is not None: + ptype = _solution_scalar_proto_type(registry, f) + lines.append((num, f" {ptype} {f['name']} = {num};")) + lines.sort(key=lambda x: x[0]) + return "\n".join(item[1] for item in lines) + + +def generate_mip_solution_message_proto(registry): + obj = registry.get("mip_solution", {}) + lines = [] + for entry in obj.get("arrays", []): + f = parse_field(entry) + num = f.get("field_num") + if num is not None: + lines.append((num, f" repeated double {f['name']} = {num};")) + for entry in obj.get("scalars", []): + f = parse_field(entry) + num = f.get("field_num") + if num is not None: + ptype = _solution_scalar_proto_type(registry, f) + lines.append((num, f" {ptype} {f['name']} = {num};")) + lines.sort(key=lambda x: x[0]) + return "\n".join(item[1] for item in lines) + + +def generate_optimization_problem_proto(registry): + obj = registry.get("optimization_problem", {}) + if not obj: + return "" + lines = [] + for entry in obj.get("scalars", []): + f = parse_field(entry) + num = f.get("field_num") + if num is not None: + ptype = _problem_field_proto_type( + registry, f.get("type", "double") + ) + lines.append((num, f" {ptype} {f['name']} = {num};")) + for entry in obj.get("arrays", []): + f = parse_field(entry) + num = f.get("field_num") + if num is not None: + ptype = _problem_field_proto_type( + registry, f.get("type", "repeated double") + ) + lines.append((num, f" {ptype} {f['name']} = {num};")) + lines.sort(key=lambda x: x[0]) + return "\n".join(item[1] for item in lines) + + +def generate_chunked_result_header_proto(registry): + pc_enum = _enum_proto_type( + "problem_category", registry["enums"]["problem_category"] + ) + lines = [(1, f" {pc_enum} problem_category = 1;")] + for obj_name in ["lp_solution", "mip_solution"]: + obj = registry.get(obj_name, {}) + for entry in obj.get("scalars", []): + f = parse_field(entry) + num = f.get("field_num") + if num is not None: + ptype = _solution_scalar_proto_type(registry, f) + lines.append((num, f" {ptype} {f['name']} = {num};")) + ws = obj.get("warm_start", {}) + for entry in ws.get("scalars", []): + f = parse_field(entry) + num = f.get("field_num") + if num is not None: + lines.append( + ( + num, + f" {proto_type(f.get('type', 'double'))} {f['name']} = {num};", + ) + ) + lines.append((50, " repeated ResultArrayDescriptor arrays = 50;")) + lines.sort(key=lambda x: x[0]) + return "\n".join(item[1] for item in lines) + + +# ============================================================================ +# Full data proto assembly +# ============================================================================ + + +def generate_data_proto(registry): + enums = generate_enum_proto_from_registry(registry) + rfid = ( + "enum ResultFieldId {\n" + + generate_proto_result_enums(registry) + + "\n}" + ) + opt_prob = "" + if "optimization_problem" in registry: + opt_prob = ( + "message OptimizationProblem {\n" + + generate_optimization_problem_proto(registry) + + "\n}\n" + ) + pdlp_s = ( + "message PDLPSolverSettings {\n" + + generate_settings_message_proto( + registry, "PDLPSolverSettings", registry["pdlp_settings"] + ) + + "\n}" + ) + mip_s = ( + "message MIPSolverSettings {\n" + + generate_settings_message_proto( + registry, "MIPSolverSettings", registry["mip_settings"] + ) + + "\n}" + ) + ws = ( + "message PDLPWarmStartData {\n" + + generate_warm_start_message_proto(registry) + + "\n}" + ) + lp = ( + "message LPSolution {\n" + + generate_lp_solution_message_proto(registry) + + "\n}" + ) + mip = ( + "message MIPSolution {\n" + + generate_mip_solution_message_proto(registry) + + "\n}" + ) + rad = ( + "message ResultArrayDescriptor {\n" + " ResultFieldId field_id = 1;\n" + " int64 total_elements = 2;\n" + " int64 element_size_bytes = 3;\n" + "}" + ) + ch = ( + "message ChunkedResultHeader {\n" + + generate_chunked_result_header_proto(registry) + + "\n}" + ) + afid = "" + if "optimization_problem" in registry: + afid_body = generate_array_field_id_enum(registry) + if afid_body: + afid = "enum ArrayFieldId {\n" + afid_body + "\n}\n" + parts = [ + "// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml", + "// DO NOT EDIT — regenerate with: python cpp/codegen/generate_conversions.py", + "", + 'syntax = "proto3";', + "", + "package cuopt.remote;", + "", + enums, + "", + rfid, + "", + afid, + opt_prob, + pdlp_s, + "", + mip_s, + "", + ws, + "", + lp, + "", + mip, + "", + rad, + "", + ch, + "", + ] + return "\n".join(parts) + + +# ============================================================================ +# Solution conversion code generation +# ============================================================================ + + +def _gen_solution_to_proto(registry, obj_name, obj, indent=" "): + """Generate body of map_{lp,mip}_solution_to_proto().""" + ind = indent + lines = [] + + for entry in obj.get("scalars", []): + f = parse_field(entry) + name = f["name"] + pname = _proto_cpp_name(name) + getter = f.get("getter", f"get_{name}()") + edef = _lookup_enum(registry, f.get("type", "double")) + if edef: + to_fn = _enum_to_proto_fn(f["type"], edef) + lines.append( + f"{ind}pb_solution->set_{pname}({to_fn}(solution.{getter}));" + ) + else: + lines.append(f"{ind}pb_solution->set_{pname}(solution.{getter});") + + lines.append("") + + for entry in obj.get("arrays", []): + f = parse_field(entry) + pname = _proto_cpp_name(f["name"]) + getter = f.get("getter", f"get_{f['name']}_host()") + var = f"_{f['name']}" + lines.append(f"{ind}const auto& {var} = solution.{getter};") + lines.append( + f"{ind}for (const auto& v : {var}) pb_solution->add_{pname}(static_cast(v));" + ) + + # Warm start + ws = obj.get("warm_start") + if ws: + check = ws.get("presence_check", "has_warm_start_data()") + ws_getter = ws.get("getter", "get_cpu_pdlp_warm_start_data()") + lines.append("") + lines.append(f"{ind}if (solution.{check}) {{") + lines.append( + f"{ind} auto* pb_ws = pb_solution->mutable_warm_start_data();" + ) + lines.append(f"{ind} const auto& ws = solution.{ws_getter};") + for entry in ws.get("arrays", []): + f = parse_field(entry) + pname = _proto_cpp_name(f["name"]) + member = f.get("member", f["name"]) + lines.append( + f"{ind} for (const auto& v : ws.{member}) pb_ws->add_{pname}(static_cast(v));" + ) + for entry in ws.get("scalars", []): + f = parse_field(entry) + pname = _proto_cpp_name(f["name"]) + member = f.get("member", f["name"]) + ftype = f.get("type", "double") + cast = ( + "double" + if ftype == "double" + else ("int32_t" if ftype == "int32" else "double") + ) + lines.append( + f"{ind} pb_ws->set_{pname}(static_cast<{cast}>(ws.{member}));" + ) + lines.append(f"{ind}}}") + + return "\n".join(lines) + + +def _gen_proto_to_solution(registry, obj_name, obj, indent=" "): + """Generate body of map_proto_to_{lp,mip}_solution().""" + ind = indent + lines = [] + cpp_type = obj["cpp_type"] + constructor_args = obj.get("constructor_args", {}) + ws = obj.get("warm_start") + + for entry in obj.get("arrays", []): + f = parse_field(entry) + pname = _proto_cpp_name(f["name"]) + lines.append( + f"{ind}std::vector {f['name']}(pb_solution.{pname}().begin(), pb_solution.{pname}().end());" + ) + + lines.append("") + + scalar_vars = {} + for entry in obj.get("scalars", []): + f = parse_field(entry) + name = f["name"] + pname = _proto_cpp_name(name) + if f.get("proto_only"): + continue + ftype = f.get("type", "double") + edef = _lookup_enum(registry, ftype) + if edef: + from_fn = _enum_from_proto_fn(ftype, edef) + lines.append( + f"{ind}auto _{name} = {from_fn}(pb_solution.{pname}());" + ) + else: + cast = _from_proto_cast(ftype) + if cast: + lines.append( + f"{ind}auto _{name} = static_cast<{cast}>(pb_solution.{pname}());" + ) + else: + lines.append(f"{ind}auto _{name} = pb_solution.{pname}();") + scalar_vars[name] = f"_{name}" + + array_names = [parse_field(e)["name"] for e in obj.get("arrays", [])] + arg_scalars = constructor_args.get("scalars", []) + args = [f"std::move({n})" for n in array_names] + args += [scalar_vars.get(s, f"_{s}") for s in arg_scalars] + + # Warm start + if ws: + accessor_getter = "warm_start_data" + lines.append("") + lines.append(f"{ind}if (pb_solution.has_{accessor_getter}()) {{") + lines.append( + f"{ind} const auto& pb_ws = pb_solution.{accessor_getter}();" + ) + lines.append(f"{ind} cpu_pdlp_warm_start_data_t ws;") + + for entry in ws.get("arrays", []): + f = parse_field(entry) + pname = _proto_cpp_name(f["name"]) + member = f.get("member", f["name"]) + lines.append( + f"{ind} ws.{member}.assign(pb_ws.{pname}().begin(), pb_ws.{pname}().end());" + ) + + for entry in ws.get("scalars", []): + f = parse_field(entry) + pname = _proto_cpp_name(f["name"]) + member = f.get("member", f["name"]) + ftype = f.get("type", "double") + cast = _from_proto_cast(ftype) + if cast: + lines.append( + f"{ind} ws.{member} = static_cast<{cast}>(pb_ws.{pname}());" + ) + else: + lines.append(f"{ind} ws.{member} = pb_ws.{pname}();") + + ws_args = args + ["std::move(ws)"] + lines.append(f"{ind} return {cpp_type}({', '.join(ws_args)});") + lines.append(f"{ind}}}") + lines.append("") + + lines.append(f"{ind}return {cpp_type}({', '.join(args)});") + return "\n".join(lines) + + +def _gen_chunked_header(registry, obj_name, obj, indent=" "): + """Generate body of populate_chunked_result_header_{lp,mip}().""" + ind = indent + lines = [] + category = "MIP" if obj_name == "mip_solution" else "LP" + lines.append( + f"{ind}header->set_problem_category(cuopt::remote::{category});" + ) + + for entry in obj.get("scalars", []): + f = parse_field(entry) + name = f["name"] + pname = _proto_cpp_name(name) + getter = f.get("getter", f"get_{name}()") + edef = _lookup_enum(registry, f.get("type", "double")) + if edef: + to_fn = _enum_to_proto_fn(f["type"], edef) + lines.append( + f"{ind}header->set_{pname}({to_fn}(solution.{getter}));" + ) + else: + lines.append(f"{ind}header->set_{pname}(solution.{getter});") + + lines.append("") + + for entry in obj.get("arrays", []): + f = parse_field(entry) + getter = f.get("getter", f"get_{f['name']}_host()") + eid = _field_result_id_name(f) + lines.append( + f"{ind}add_result_array_descriptor(header, cuopt::remote::{eid}, solution.{getter}.size(), sizeof(double));" + ) + + # Warm start + ws = obj.get("warm_start") + if ws: + check = ws.get("presence_check", "has_warm_start_data()") + ws_getter = ws.get("getter", "get_cpu_pdlp_warm_start_data()") + lines.append("") + lines.append(f"{ind}if (solution.{check}) {{") + lines.append(f"{ind} const auto& ws = solution.{ws_getter};") + for entry in ws.get("scalars", []): + f = parse_field(entry) + pname = _proto_cpp_name(f["name"]) + member = f.get("member", f["name"]) + ftype = f.get("type", "double") + cast = ( + "double" + if ftype == "double" + else ("int32_t" if ftype == "int32" else "double") + ) + lines.append( + f"{ind} header->set_{pname}(static_cast<{cast}>(ws.{member}));" + ) + lines.append("") + for entry in ws.get("arrays", []): + f = parse_field(entry) + member = f.get("member", f["name"]) + eid = _field_result_id_name(f) + lines.append( + f"{ind} add_result_array_descriptor(header, cuopt::remote::{eid}, ws.{member}.size(), sizeof(double));" + ) + lines.append(f"{ind}}}") + + return "\n".join(lines) + + +def _gen_collect_arrays(registry, obj_name, obj, indent=" "): + """Generate body of collect_{lp,mip}_solution_arrays().""" + ind = indent + lines = [f"{ind}std::map> arrays;"] + + for entry in obj.get("arrays", []): + f = parse_field(entry) + getter = f.get("getter", f"get_{f['name']}_host()") + eid = _field_result_id_name(f) + var = f"_{f['name']}" + lines.append(f"{ind}const auto& {var} = solution.{getter};") + lines.append( + f"{ind}if (!{var}.empty()) {{ arrays[cuopt::remote::{eid}] = doubles_to_bytes({var}); }}" + ) + + ws = obj.get("warm_start") + if ws: + check = ws.get("presence_check", "has_warm_start_data()") + ws_getter = ws.get("getter", "get_cpu_pdlp_warm_start_data()") + lines.append(f"{ind}if (solution.{check}) {{") + lines.append(f"{ind} const auto& ws = solution.{ws_getter};") + for entry in ws.get("arrays", []): + f = parse_field(entry) + member = f.get("member", f["name"]) + eid = _field_result_id_name(f) + lines.append( + f"{ind} if (!ws.{member}.empty()) {{ arrays[cuopt::remote::{eid}] = doubles_to_bytes(ws.{member}); }}" + ) + lines.append(f"{ind}}}") + + lines.append(f"{ind}return arrays;") + return "\n".join(lines) + + +def _gen_chunked_to_solution(registry, obj_name, obj, indent=" "): + """Generate body of chunked_result_to_{lp,mip}_solution().""" + ind = indent + lines = [] + constructor_args = obj.get("constructor_args", {}) + ws = obj.get("warm_start") + + for entry in obj.get("arrays", []): + f = parse_field(entry) + eid = _field_result_id_name(f) + lines.append( + f"{ind}auto {f['name']} = bytes_to_typed(arrays, cuopt::remote::{eid});" + ) + + lines.append("") + + scalar_vars = {} + for entry in obj.get("scalars", []): + f = parse_field(entry) + name = f["name"] + pname = _proto_cpp_name(name) + if f.get("proto_only"): + continue + ftype = f.get("type", "double") + edef = _lookup_enum(registry, ftype) + if edef: + from_fn = _enum_from_proto_fn(ftype, edef) + lines.append(f"{ind}auto _{name} = {from_fn}(h.{pname}());") + else: + cast = _from_proto_cast(ftype) + if cast: + lines.append( + f"{ind}auto _{name} = static_cast<{cast}>(h.{pname}());" + ) + else: + lines.append(f"{ind}auto _{name} = h.{pname}();") + scalar_vars[name] = f"_{name}" + + array_names = [parse_field(e)["name"] for e in obj.get("arrays", [])] + arg_scalars = constructor_args.get("scalars", []) + args = [f"std::move({n})" for n in array_names] + args += [scalar_vars.get(s, f"_{s}") for s in arg_scalars] + + # Warm start + if ws: + ws_arrays = ws.get("arrays", []) + first_array = parse_field(ws_arrays[0]) if ws_arrays else None + detect_eid = ( + _field_result_id_name(first_array) if first_array else None + ) + lines.append("") + lines.append( + f"{ind}auto _ws_detect = bytes_to_typed(arrays, cuopt::remote::{detect_eid});" + ) + lines.append(f"{ind}if (!_ws_detect.empty()) {{") + lines.append(f"{ind} cpu_pdlp_warm_start_data_t ws;") + + first = True + for entry in ws_arrays: + f = parse_field(entry) + member = f.get("member", f["name"]) + eid = _field_result_id_name(f) + if first: + lines.append(f"{ind} ws.{member} = std::move(_ws_detect);") + first = False + else: + lines.append( + f"{ind} ws.{member} = bytes_to_typed(arrays, cuopt::remote::{eid});" + ) + + for entry in ws.get("scalars", []): + f = parse_field(entry) + pname = _proto_cpp_name(f["name"]) + member = f.get("member", f["name"]) + ftype = f.get("type", "double") + cast = _from_proto_cast(ftype) + if cast: + lines.append( + f"{ind} ws.{member} = static_cast<{cast}>(h.{pname}());" + ) + else: + lines.append(f"{ind} ws.{member} = h.{pname}();") + + cpp_type = obj["cpp_type"] + ws_args = args + ["std::move(ws)"] + lines.append(f"{ind} return {cpp_type}({', '.join(ws_args)});") + lines.append(f"{ind}}}") + lines.append("") + + cpp_type = obj["cpp_type"] + lines.append(f"{ind}return {cpp_type}({', '.join(args)});") + return "\n".join(lines) + + +def _gen_estimate_size(registry, obj_name, obj, indent=" "): + """Generate body of estimate_{lp,mip}_solution_proto_size().""" + ind = indent + lines = [f"{ind}size_t est = 0;"] + + for entry in obj.get("arrays", []): + f = parse_field(entry) + getter = f.get("getter", f"get_{f['name']}_host()") + size_getter = getter.replace("_host()", "_size()") + lines.append( + f"{ind}est += static_cast(solution.{size_getter}) * sizeof(double);" + ) + + ws = obj.get("warm_start") + if ws: + check = ws.get("presence_check", "has_warm_start_data()") + ws_getter = ws.get("getter", "get_cpu_pdlp_warm_start_data()") + lines.append(f"{ind}if (solution.{check}) {{") + lines.append(f"{ind} const auto& ws = solution.{ws_getter};") + for entry in ws.get("arrays", []): + f = parse_field(entry) + member = f.get("member", f["name"]) + lines.append(f"{ind} est += ws.{member}.size() * sizeof(double);") + lines.append(f"{ind}}}") + + overhead = 512 if ws else 256 + lines.append(f"{ind}est += {overhead};") + lines.append(f"{ind}return est;") + return "\n".join(lines) + + +# ============================================================================ +# Problem conversion code generation +# ============================================================================ + + +def _gen_problem_to_proto(registry, indent=" "): + """Generate body of map_problem_to_proto().""" + obj = registry.get("optimization_problem", {}) + ind = indent + lines = [] + setter_groups = obj.get("setter_groups", {}) + grouped_fields = set() + for gdef in setter_groups.values(): + for fname in gdef.get("fields", []): + grouped_fields.add(fname) + + # Scalars + for entry in obj.get("scalars", []): + f = parse_field(entry) + pname = _proto_cpp_name(f["name"]) + getter = _default_problem_getter(f, is_scalar=True) + ftype = f.get("type", "double") + edef = _lookup_enum(registry, ftype) + if edef: + to_fn = _enum_to_proto_fn(ftype, edef) + lines.append( + f"{ind}pb_problem->set_{pname}({to_fn}(cpu_problem.{getter}));" + ) + else: + lines.append( + f"{ind}pb_problem->set_{pname}(cpu_problem.{getter});" + ) + + lines.append("") + + # Non-grouped arrays + for entry in obj.get("arrays", []): + f = parse_field(entry) + name = f["name"] + pname = _proto_cpp_name(name) + if name in grouped_fields or f.get("skip_conversion"): + continue + ftype = f.get("type", "repeated double") + getter = _default_problem_getter(f, is_scalar=False) + + enum_key = _is_repeated_enum(registry, ftype) + if enum_key: + edef = _lookup_enum(registry, enum_key) + to_fn = _enum_to_proto_fn(enum_key, edef) + lines.append(f"{ind}{{") + lines.append(f"{ind} auto _{name} = cpu_problem.{getter};") + lines.append( + f"{ind} for (const auto& v : _{name}) pb_problem->add_{pname}({to_fn}(v));" + ) + lines.append(f"{ind}}}") + elif ftype == "repeated string": + lines.append( + f"{ind}for (const auto& s : cpu_problem.{getter}) pb_problem->add_{pname}(s);" + ) + elif ftype == "bytes": + lines.append(f"{ind}{{") + lines.append(f"{ind} auto _{name} = cpu_problem.{getter};") + lines.append(f"{ind} if (!_{name}.empty()) {{") + lines.append( + f"{ind} pb_problem->set_{pname}(std::string(_{name}.begin(), _{name}.end()));" + ) + lines.append(f"{ind} }}") + lines.append(f"{ind}}}") + elif ftype.startswith("repeated"): + cast = _array_element_cast(ftype) + var = f"_{name}" + lines.append(f"{ind}{{") + lines.append(f"{ind} auto {var} = cpu_problem.{getter};") + lines.append( + f"{ind} for (const auto& v : {var}) pb_problem->add_{pname}(static_cast<{cast}>(v));" + ) + lines.append(f"{ind}}}") + + # Setter groups + for gname, gdef in setter_groups.items(): + fields = [ + _find_problem_field(obj, fn) for fn in gdef.get("fields", []) + ] + for f in fields: + if f is None: + continue + pname = _proto_cpp_name(f["name"]) + getter = _default_problem_getter(f, is_scalar=False) + ftype = f.get("type", "repeated double") + cast = _array_element_cast(ftype) + lines.append(f"{ind}{{") + lines.append(f"{ind} auto _{f['name']} = cpu_problem.{getter};") + lines.append( + f"{ind} for (const auto& v : _{f['name']}) pb_problem->add_{pname}(static_cast<{cast}>(v));" + ) + lines.append(f"{ind}}}") + + return "\n".join(lines) + + +def _gen_proto_to_problem(registry, indent=" "): + """Generate body of map_proto_to_problem().""" + obj = registry.get("optimization_problem", {}) + ind = indent + lines = [] + setter_groups = obj.get("setter_groups", {}) + grouped_fields = set() + for gdef in setter_groups.values(): + for fname in gdef.get("fields", []): + grouped_fields.add(fname) + # Scalars + for entry in obj.get("scalars", []): + f = parse_field(entry) + name = f["name"] + pname = _proto_cpp_name(name) + setter = _default_problem_setter(f) + ftype = f.get("type", "double") + edef = _lookup_enum(registry, ftype) + if edef: + from_fn = _enum_from_proto_fn(ftype, edef) + lines.append( + f"{ind}cpu_problem.{setter}({from_fn}(pb_problem.{pname}()));" + ) + else: + lines.append(f"{ind}cpu_problem.{setter}(pb_problem.{pname}());") + + lines.append("") + + # Setter groups — guard on first field having data + for gname, gdef in setter_groups.items(): + setter_name = gdef["setter"] + fields = [ + _find_problem_field(obj, fn) for fn in gdef.get("fields", []) + ] + first = next((f for f in fields if f is not None), None) + if first is None: + continue + first_pname = _proto_cpp_name(first["name"]) + lines.append(f"{ind}if (pb_problem.{first_pname}_size() > 0) {{") + ii = ind + " " + + for f in fields: + if f is None: + continue + pname = _proto_cpp_name(f["name"]) + ftype = f.get("type", "repeated double") + cpp_t = _array_cpp_type(ftype) + lines.append( + f"{ii}std::vector<{cpp_t}> {f['name']}(pb_problem.{pname}().begin(), pb_problem.{pname}().end());" + ) + + args = [] + for f in fields: + if f is None: + continue + args.append(f"{f['name']}.data()") + args.append(f"static_cast({f['name']}.size())") + lines.append(f"{ii}cpu_problem.{setter_name}({', '.join(args)});") + lines.append(f"{ind}}}") + lines.append("") + + # Non-grouped arrays + for entry in obj.get("arrays", []): + f = parse_field(entry) + name = f["name"] + pname = _proto_cpp_name(name) + if name in grouped_fields or f.get("skip_conversion"): + continue + ftype = f.get("type", "repeated double") + setter = _default_problem_setter(f) + + enum_key = _is_repeated_enum(registry, ftype) + if enum_key: + edef = _lookup_enum(registry, enum_key) + from_fn = _enum_from_proto_fn(enum_key, edef) + cpp_type = _enum_cpp_type(enum_key, edef) + lines.append(f"{ind}if (pb_problem.{pname}_size() > 0) {{") + lines.append(f"{ind} std::vector<{cpp_type}> {name};") + lines.append(f"{ind} {name}.reserve(pb_problem.{pname}_size());") + proto_type = _enum_proto_type(enum_key, edef) + lines.append( + f"{ind} for (const auto& v : pb_problem.{pname}()) {{" + ) + lines.append( + f"{ind} {name}.push_back({from_fn}(static_cast(v)));" + ) + lines.append(f"{ind} }}") + lines.append( + f"{ind} cpu_problem.{setter}({name}.data(), static_cast({name}.size()));" + ) + lines.append(f"{ind}}}") + elif ftype == "repeated string": + lines.append(f"{ind}if (pb_problem.{pname}_size() > 0) {{") + lines.append( + f"{ind} std::vector {name}(pb_problem.{pname}().begin(), pb_problem.{pname}().end());" + ) + lines.append(f"{ind} cpu_problem.{setter}({name});") + lines.append(f"{ind}}}") + elif ftype == "bytes": + lines.append(f"{ind}if (!pb_problem.{pname}().empty()) {{") + lines.append( + f"{ind} const std::string& {name}_str = pb_problem.{pname}();" + ) + lines.append( + f"{ind} cpu_problem.{setter}({name}_str.data(), static_cast({name}_str.size()));" + ) + lines.append(f"{ind}}}") + elif ftype.startswith("repeated"): + cpp_t = _array_cpp_type(ftype) + guard = f.get("conditional") + open_brace = ( + f"if (pb_problem.{pname}_size() > 0) {{" if guard else "{" + ) + lines.append(f"{ind}{open_brace}") + lines.append( + f"{ind} std::vector<{cpp_t}> {name}(pb_problem.{pname}().begin(), pb_problem.{pname}().end());" + ) + lines.append( + f"{ind} cpu_problem.{setter}({name}.data(), static_cast({name}.size()));" + ) + lines.append(f"{ind}}}") + + return "\n".join(lines) + + +def _gen_estimate_problem_size(registry, indent=" "): + """Generate body of estimate_problem_proto_size().""" + obj = registry.get("optimization_problem", {}) + ind = indent + lines = [f"{ind}size_t est = 0;"] + + for entry in obj.get("arrays", []): + f = parse_field(entry) + if f.get("skip_conversion"): + continue + ftype = f.get("type", "repeated double") + getter = _default_problem_getter(f, is_scalar=False) + if ftype == "repeated string": + lines.append( + f"{ind}for (const auto& s : cpu_problem.{getter}) est += s.size() + 2;" + ) + elif ftype == "bytes": + lines.append(f"{ind}est += cpu_problem.{getter}.size();") + elif "int32" in ftype: + lines.append(f"{ind}est += cpu_problem.{getter}.size() * 5;") + elif ftype.startswith("repeated ") and _is_repeated_enum( + registry, ftype + ): + lines.append(f"{ind}est += cpu_problem.{getter}.size() * 4;") + else: + lines.append( + f"{ind}est += cpu_problem.{getter}.size() * sizeof(double);" + ) + + lines.append(f"{ind}est += 512;") + lines.append(f"{ind}return est;") + return "\n".join(lines) + + +def _gen_populate_chunked_header(registry, solver_type, indent=" "): + """Generate body of populate_chunked_header_{lp,mip}().""" + obj = registry.get("optimization_problem", {}) + ind = indent + lines = [] + + lines.append(f"{ind}auto* rh = header->mutable_header();") + lines.append(f"{ind}rh->set_version(1);") + if solver_type == "lp": + lines.append(f"{ind}rh->set_problem_category(cuopt::remote::LP);") + else: + lines.append(f"{ind}rh->set_problem_category(cuopt::remote::MIP);") + + lines.append("") + + for entry in obj.get("scalars", []): + f = parse_field(entry) + pname = _proto_cpp_name(f["name"]) + getter = _default_problem_getter(f, is_scalar=True) + ftype = f.get("type", "double") + edef = _lookup_enum(registry, ftype) + if edef: + to_fn = _enum_to_proto_fn(ftype, edef) + lines.append( + f"{ind}header->set_{pname}({to_fn}(cpu_problem.{getter}));" + ) + else: + lines.append(f"{ind}header->set_{pname}(cpu_problem.{getter});") + + lines.append("") + + if solver_type == "lp": + lines.append( + f"{ind}map_pdlp_settings_to_proto(settings, header->mutable_lp_settings());" + ) + else: + lines.append( + f"{ind}map_mip_settings_to_proto(settings, header->mutable_mip_settings());" + ) + lines.append(f"{ind}header->set_enable_incumbents(enable_incumbents);") + + return "\n".join(lines) + + +def _gen_chunked_header_to_problem(registry, indent=" "): + """Generate body of map_chunked_header_to_problem().""" + obj = registry.get("optimization_problem", {}) + ind = indent + lines = [] + + for entry in obj.get("scalars", []): + f = parse_field(entry) + name = f["name"] + pname = _proto_cpp_name(name) + setter = _default_problem_setter(f) + ftype = f.get("type", "double") + edef = _lookup_enum(registry, ftype) + if edef: + from_fn = _enum_from_proto_fn(ftype, edef) + lines.append( + f"{ind}cpu_problem.{setter}({from_fn}(header.{pname}()));" + ) + else: + lines.append(f"{ind}cpu_problem.{setter}(header.{pname}());") + + lines.append("") + + for entry in obj.get("arrays", []): + f = parse_field(entry) + if f.get("type") != "repeated string": + continue + name = f["name"] + pname = _proto_cpp_name(name) + setter = _default_problem_setter(f) + lines.append(f"{ind}if (header.{pname}_size() > 0) {{") + lines.append( + f"{ind} std::vector {name}(header.{pname}().begin(), header.{pname}().end());" + ) + lines.append(f"{ind} cpu_problem.{setter}({name});") + lines.append(f"{ind}}}") + + return "\n".join(lines) + + +def _gen_chunked_arrays_to_problem(registry, indent=" "): + """Generate body of map_chunked_arrays_to_problem().""" + obj = registry.get("optimization_problem", {}) + ind = indent + lines = [] + setter_groups = obj.get("setter_groups", {}) + grouped_fields = set() + for gdef in setter_groups.values(): + for fname in gdef.get("fields", []): + grouped_fields.add(fname) + lines.append(f"{ind}map_chunked_header_to_problem(header, cpu_problem);") + lines.append("") + + # Lambda helpers + lines.append( + f"{ind}auto get_doubles = [&](int32_t field_id) -> std::vector {{" + ) + lines.append(f"{ind} auto it = arrays.find(field_id);") + lines.append( + f"{ind} if (it == arrays.end() || it->second.empty()) return {{}};" + ) + lines.append( + f"{ind} if (it->second.size() % sizeof(double) != 0) return {{}};" + ) + lines.append(f"{ind} size_t n = it->second.size() / sizeof(double);") + lines.append(f"{ind} if constexpr (std::is_same_v) {{") + lines.append(f"{ind} std::vector v(n);") + lines.append( + f"{ind} std::memcpy(v.data(), it->second.data(), n * sizeof(double));" + ) + lines.append(f"{ind} return v;") + lines.append(f"{ind} }} else {{") + lines.append(f"{ind} std::vector tmp(n);") + lines.append( + f"{ind} std::memcpy(tmp.data(), it->second.data(), n * sizeof(double));" + ) + lines.append(f"{ind} return std::vector(tmp.begin(), tmp.end());") + lines.append(f"{ind} }}") + lines.append(f"{ind}}};") + lines.append("") + + lines.append( + f"{ind}auto get_ints = [&](int32_t field_id) -> std::vector {{" + ) + lines.append(f"{ind} auto it = arrays.find(field_id);") + lines.append( + f"{ind} if (it == arrays.end() || it->second.empty()) return {{}};" + ) + lines.append( + f"{ind} if (it->second.size() % sizeof(int32_t) != 0) return {{}};" + ) + lines.append(f"{ind} size_t n = it->second.size() / sizeof(int32_t);") + lines.append(f"{ind} if constexpr (std::is_same_v) {{") + lines.append(f"{ind} std::vector v(n);") + lines.append( + f"{ind} std::memcpy(v.data(), it->second.data(), n * sizeof(int32_t));" + ) + lines.append(f"{ind} return v;") + lines.append(f"{ind} }} else {{") + lines.append(f"{ind} std::vector tmp(n);") + lines.append( + f"{ind} std::memcpy(tmp.data(), it->second.data(), n * sizeof(int32_t));" + ) + lines.append(f"{ind} return std::vector(tmp.begin(), tmp.end());") + lines.append(f"{ind} }}") + lines.append(f"{ind}}};") + lines.append("") + + lines.append( + f"{ind}auto get_bytes = [&](int32_t field_id) -> std::string {{" + ) + lines.append(f"{ind} auto it = arrays.find(field_id);") + lines.append( + f"{ind} if (it == arrays.end() || it->second.empty()) return {{}};" + ) + lines.append( + f"{ind} return std::string(reinterpret_cast(it->second.data()), it->second.size());" + ) + lines.append(f"{ind}}};") + lines.append("") + + lines.append( + f"{ind}auto get_string_list = [&](int32_t field_id) -> std::vector {{" + ) + lines.append(f"{ind} auto it = arrays.find(field_id);") + lines.append( + f"{ind} if (it == arrays.end() || it->second.empty()) return {{}};" + ) + lines.append(f"{ind} std::vector names;") + lines.append( + f"{ind} const char* s = reinterpret_cast(it->second.data());" + ) + lines.append(f"{ind} const char* s_end = s + it->second.size();") + lines.append(f"{ind} while (s < s_end) {{") + lines.append( + f"{ind} const char* nul = static_cast(std::memchr(s, '\\0', s_end - s));" + ) + lines.append(f"{ind} if (!nul) nul = s_end;") + lines.append(f"{ind} names.emplace_back(s, nul);") + lines.append(f"{ind} if (nul == s_end) break;") + lines.append(f"{ind} s = nul + 1;") + lines.append(f"{ind} }}") + lines.append(f"{ind} return names;") + lines.append(f"{ind}}};") + lines.append("") + + # Setter groups + for gname, gdef in setter_groups.items(): + setter_name = gdef["setter"] + fields = [ + _find_problem_field(obj, fn) for fn in gdef.get("fields", []) + ] + + for f in fields: + if f is None: + continue + ftype = f.get("type", "repeated double") + afid = _field_array_id_name(f) + extract_fn = "get_doubles" if "double" in ftype else "get_ints" + lines.append( + f"{ind}auto {f['name']} = {extract_fn}(cuopt::remote::{afid});" + ) + + guard_parts = [f"!{f['name']}.empty()" for f in fields if f] + lines.append(f"{ind}if ({' && '.join(guard_parts)}) {{") + + args = [] + for f in fields: + if f is None: + continue + args.append(f"{f['name']}.data()") + args.append(f"static_cast({f['name']}.size())") + lines.append(f"{ind} cpu_problem.{setter_name}({', '.join(args)});") + lines.append(f"{ind}}}") + lines.append("") + + # Non-grouped arrays + for entry in obj.get("arrays", []): + f = parse_field(entry) + name = f["name"] + if name in grouped_fields or f.get("skip_conversion"): + continue + ftype = f.get("type", "repeated double") + afid = _field_array_id_name(f) + setter = _default_problem_setter(f) + + enum_key = _is_repeated_enum(registry, ftype) + if enum_key: + edef = _lookup_enum(registry, enum_key) + from_fn = _enum_from_proto_fn(enum_key, edef) + cpp_type = _enum_cpp_type(enum_key, edef) + lines.append( + f"{ind}auto {name}_ints = get_ints(cuopt::remote::{afid});" + ) + lines.append(f"{ind}if (!{name}_ints.empty()) {{") + lines.append(f"{ind} std::vector<{cpp_type}> {name};") + lines.append(f"{ind} {name}.reserve({name}_ints.size());") + lines.append(f"{ind} for (auto v : {name}_ints) {{") + lines.append( + f"{ind} {name}.push_back({from_fn}(static_cast(v)));" + ) + lines.append(f"{ind} }}") + lines.append( + f"{ind} cpu_problem.{setter}({name}.data(), static_cast({name}.size()));" + ) + lines.append(f"{ind}}}") + elif ftype == "repeated string": + lines.append( + f"{ind}auto {name} = get_string_list(cuopt::remote::{afid});" + ) + lines.append( + f"{ind}if (!{name}.empty()) {{ cpu_problem.{setter}({name}); }}" + ) + elif ftype == "bytes": + lines.append( + f"{ind}auto {name}_str = get_bytes(cuopt::remote::{afid});" + ) + lines.append(f"{ind}if (!{name}_str.empty()) {{") + lines.append( + f"{ind} cpu_problem.set_{name}({name}_str.data(), static_cast({name}_str.size()));" + ) + lines.append(f"{ind}}}") + elif ftype.startswith("repeated"): + extract_fn = "get_doubles" if "double" in ftype else "get_ints" + lines.append( + f"{ind}auto {name} = {extract_fn}(cuopt::remote::{afid});" + ) + lines.append(f"{ind}if (!{name}.empty()) {{") + lines.append( + f"{ind} cpu_problem.{setter}({name}.data(), static_cast({name}.size()));" + ) + lines.append(f"{ind}}}") + lines.append("") + + return "\n".join(lines).rstrip() + + +def _gen_build_array_chunk_requests(registry, indent=" "): + """Generate body of build_array_chunk_requests().""" + obj = registry.get("optimization_problem", {}) + ind = indent + lines = [ + f"{ind}std::vector requests;" + ] + lines.append("") + setter_groups = obj.get("setter_groups", {}) + grouped_fields = set() + for gdef in setter_groups.values(): + for fname in gdef.get("fields", []): + grouped_fields.add(fname) + + def _emit_chunk_call(f, getter_expr, var_name, ii): + ftype = f.get("type", "repeated double") + afid = _field_array_id_name(f) + + if ftype == "repeated string": + lines.append(f"{ii}{{") + lines.append(f"{ii} auto _blob = names_to_blob({getter_expr});") + lines.append( + f"{ii} chunk_byte_blob(requests, cuopt::remote::{afid}, _blob, upload_id, chunk_size_bytes);" + ) + lines.append(f"{ii}}}") + elif ftype == "bytes": + lines.append(f"{ii}if (!{var_name}.empty()) {{") + lines.append( + f"{ii} std::vector _bytes({var_name}.begin(), {var_name}.end());" + ) + lines.append( + f"{ii} chunk_byte_blob(requests, cuopt::remote::{afid}, _bytes, upload_id, chunk_size_bytes);" + ) + lines.append(f"{ii}}}") + elif "int32" in ftype: + lines.append( + f"{ii}chunk_typed_array(requests, cuopt::remote::{afid}, {var_name}, upload_id, chunk_size_bytes);" + ) + elif _is_repeated_enum(registry, ftype): + lines.append( + f"{ii}chunk_typed_array(requests, cuopt::remote::{afid}, {var_name}, upload_id, chunk_size_bytes);" + ) + else: + lines.append( + f"{ii}chunk_typed_array(requests, cuopt::remote::{afid}, {var_name}, upload_id, chunk_size_bytes);" + ) + + # Non-grouped arrays + for entry in obj.get("arrays", []): + f = parse_field(entry) + name = f["name"] + if name in grouped_fields or f.get("skip_conversion"): + continue + ftype = f.get("type", "repeated double") + getter = _default_problem_getter(f, is_scalar=False) + + enum_key = _is_repeated_enum(registry, ftype) + if enum_key: + edef = _lookup_enum(registry, enum_key) + to_fn = _enum_to_proto_fn(enum_key, edef) + afid = _field_array_id_name(f) + lines.append(f"{ind}{{") + lines.append(f"{ind} auto _{name} = problem.{getter};") + lines.append(f"{ind} if (!_{name}.empty()) {{") + lines.append(f"{ind} std::vector _{name}_ints;") + lines.append(f"{ind} _{name}_ints.reserve(_{name}.size());") + lines.append( + f"{ind} for (const auto& v : _{name}) _{name}_ints.push_back(static_cast({to_fn}(v)));" + ) + lines.append( + f"{ind} chunk_typed_array(requests, cuopt::remote::{afid}, _{name}_ints, upload_id, chunk_size_bytes);" + ) + lines.append(f"{ind} }}") + lines.append(f"{ind}}}") + elif ftype == "repeated string": + _emit_chunk_call(f, f"problem.{getter}", None, ind) + elif ftype == "bytes": + lines.append(f"{ind}{{") + lines.append(f"{ind} auto _{name} = problem.{getter};") + _emit_chunk_call(f, None, f"_{name}", ind + " ") + lines.append(f"{ind}}}") + else: + lines.append(f"{ind}{{") + lines.append(f"{ind} auto _{name} = problem.{getter};") + _emit_chunk_call(f, None, f"_{name}", ind + " ") + lines.append(f"{ind}}}") + + # Setter groups + for gname, gdef in setter_groups.items(): + fields = [ + _find_problem_field(obj, fn) for fn in gdef.get("fields", []) + ] + for f in fields: + if f is None: + continue + getter = _default_problem_getter(f, is_scalar=False) + afid = _field_array_id_name(f) + lines.append(f"{ind}{{") + lines.append( + f"{ind} const auto& _{f['name']} = problem.{getter};" + ) + lines.append( + f"{ind} chunk_typed_array(requests, cuopt::remote::{afid}, _{f['name']}, upload_id, chunk_size_bytes);" + ) + lines.append(f"{ind}}}") + + lines.append("") + lines.append(f"{ind}return requests;") + return "\n".join(lines) + + +# ============================================================================ +# Auto-numbering (ruamel.yaml round-trip) +# ============================================================================ + +# Ranges for auto-numbering. hi=None means no upper bound (own message). +# Only solution scalars need hard caps because they share ChunkedResultHeader. +# Solution array_ids share a global ResultFieldId pool (handled separately). +FIELD_NUM_RANGES = { + "optimization_problem.field_num": (1, None), + "optimization_problem.array_id": (0, None), + "lp_solution.scalars": (1000, 1999), + "lp_solution.arrays.field_num": (1, None), + "mip_solution.scalars": (2000, 2999), + "mip_solution.arrays.field_num": (1, None), + "lp_solution.warm_start.scalars": (3000, 3999), + "lp_solution.warm_start.arrays.field_num": (1, None), + "pdlp_settings": (1, None), + "mip_settings": (1, None), +} + + +def _collect_field_nums(entries, key_name="field_num"): + """Collect all existing field_num or array_id values from a list of field entries.""" + nums = set() + for entry in entries: + if isinstance(entry, dict) and len(entry) == 1: + name = list(entry.keys())[0] + val = entry[name] + if isinstance(val, dict) and key_name in val: + nums.add(val[key_name]) + return nums + + +def _collect_settings_field_nums(entries): + """Collect field_num values from settings fields (handles nesting).""" + nums = set() + for entry in entries: + if isinstance(entry, dict) and len(entry) == 1: + name = list(entry.keys())[0] + val = entry[name] + if isinstance(val, list): + nums |= _collect_settings_field_nums(val) + elif isinstance(val, dict) and "field_num" in val: + nums.add(val["field_num"]) + return nums + + +def _ruamel_insert(mapping, key, value): + """Add a key to a CommentedMap at its conventional position. + + field_num goes first (position 0), array_id goes right after field_num, + everything else appends.""" + if not hasattr(mapping, "insert"): + mapping[key] = value + return + if key == "field_num": + mapping.insert(0, key, value) + elif key == "array_id": + pos = 1 if "field_num" in mapping else 0 + mapping.insert(pos, key, value) + else: + mapping[key] = value + + +def _assign_to_field_list(entries, key_name, lo, hi, existing, label): + """Assign missing key_name values to fields in a list. Mutates entries in-place. + hi=None means no upper bound.""" + next_num = max(existing) + 1 if existing else lo + assigned = 0 + for idx, entry in enumerate(entries): + if isinstance(entry, str): + continue + if not isinstance(entry, dict) or len(entry) != 1: + continue + name = list(entry.keys())[0] + val = entry[name] + if val is None: + try: + from ruamel.yaml.comments import CommentedMap + + entry[name] = CommentedMap([(key_name, next_num)]) + except ImportError: + entry[name] = {key_name: next_num} + existing.add(next_num) + next_num += 1 + assigned += 1 + elif isinstance(val, dict) and key_name not in val: + _ruamel_insert(val, key_name, next_num) + existing.add(next_num) + next_num += 1 + assigned += 1 + if hi is not None and next_num - 1 > hi and assigned > 0: + raise ValueError( + f"Range overflow in {label}: assigned {next_num - 1} exceeds max {hi}" + ) + return assigned + + +def _assign_to_settings_fields(entries, lo, hi, existing, label): + """Assign missing field_num to settings fields (handles nesting). Mutates in-place. + hi=None means no upper bound.""" + try: + from ruamel.yaml.comments import CommentedMap as _CMap + except ImportError: + _CMap = None + next_num = max(existing) + 1 if existing else lo + assigned = 0 + for entry in entries: + if not isinstance(entry, dict) or len(entry) != 1: + continue + name = list(entry.keys())[0] + val = entry[name] + if isinstance(val, list): + sub_assigned = _assign_to_settings_fields( + val, lo, hi, existing, f"{label}.{name}" + ) + assigned += sub_assigned + next_num = max(existing) + 1 if existing else next_num + elif val is None: + if _CMap is not None: + entry[name] = _CMap([("field_num", next_num)]) + else: + entry[name] = {"field_num": next_num} + existing.add(next_num) + next_num += 1 + assigned += 1 + elif isinstance(val, dict) and "field_num" not in val: + _ruamel_insert(val, "field_num", next_num) + existing.add(next_num) + next_num += 1 + assigned += 1 + if hi is not None and next_num - 1 > hi and assigned > 0: + raise ValueError( + f"Range overflow in {label}: assigned {next_num - 1} exceeds max {hi}" + ) + return assigned + + +def _normalize_bare_strings(entries): + """Convert bare string entries in a field list to single-key dicts. + This lets the auto-numbering logic treat them uniformly. + + Avoids modifying CommentedSeq comment attributes (ca.items) to prevent + corrupting ruamel.yaml's internal comment tracking on dump.""" + try: + from ruamel.yaml.comments import CommentedMap + + mk_inner = CommentedMap + + def mk_outer(name, inner): + return CommentedMap([(name, inner)]) + + except ImportError: + mk_inner = dict + + def mk_outer(name, inner): + return {name: inner} + + for idx, entry in enumerate(entries): + if isinstance(entry, str): + entries[idx] = mk_outer(entry, mk_inner()) + + +_STRIP_RE = re.compile(r"^ +(?:field_num|array_id): +\d+ *\n", re.MULTILINE) + + +def strip_field_numbers_text(path): + """Remove field_num/array_id lines from the registry via text substitution. + + Pure text operation — guaranteed to preserve all comments and formatting. + Returns the number of lines removed.""" + with open(path) as f: + original = f.read() + stripped, count = _STRIP_RE.subn("", original) + if count > 0: + with open(path, "w") as f: + f.write(stripped) + return count + + +def _dump_and_verify(ryaml, data, path): + """Dump YAML data to a string buffer, verify comments survived, then write. + + Never truncates the original file until the output is validated.""" + from io import StringIO + + with open(path) as f: + original = f.read() + original_comments = [ + line for line in original.splitlines() if line.lstrip().startswith("#") + ] + + buf = StringIO() + ryaml.dump(data, buf) + written = buf.getvalue() + written_comments = [ + line for line in written.splitlines() if line.lstrip().startswith("#") + ] + + if len(written_comments) < len(original_comments): + print( + f"ERROR: comment loss detected in {path} " + f"({len(original_comments)} → {len(written_comments)} comment lines). " + f"File not modified.", + file=sys.stderr, + ) + sys.exit(1) + + with open(path, "w") as f: + f.write(written) + + +def _registry_has_field_numbers(registry): + """Check if the registry has at least one field_num or array_id assigned.""" + for section_key in list(registry.keys()): + section = registry.get(section_key) + if not isinstance(section, dict): + continue + for list_key in ["scalars", "arrays"]: + for nums in [ + _collect_field_nums(section.get(list_key, []), "field_num"), + _collect_field_nums(section.get(list_key, []), "array_id"), + ]: + if nums: + return True + if "fields" in section: + if _collect_settings_field_nums(section["fields"]): + return True + sub = section.get("warm_start") + if isinstance(sub, dict): + for list_key in ["scalars", "arrays"]: + for key in ["field_num", "array_id"]: + if _collect_field_nums(sub.get(list_key, []), key): + return True + return False + + +def auto_assign_field_numbers(data): + """Walk the registry and assign missing field_num / array_id values. + + Operates on ruamel.yaml CommentedMap data in-place. + Returns the total number of assignments made. + """ + # Normalize bare string entries to dicts before processing + for section_key in list(data.keys()): + section = data.get(section_key) + if not isinstance(section, dict): + continue + for list_key in ["scalars", "arrays", "fields"]: + entries = section.get(list_key) + if entries: + _normalize_bare_strings(entries) + for sub_key in ["warm_start"]: + sub = section.get(sub_key) + if isinstance(sub, dict): + for list_key in ["scalars", "arrays"]: + entries = sub.get(list_key) + if entries: + _normalize_bare_strings(entries) + + total = 0 + + for section_key in ["optimization_problem"]: + section = data.get(section_key) + if not section: + continue + + # field_num — single contiguous pool across scalars and arrays + scalars = section.get("scalars", []) + arrays = section.get("arrays", []) + lo, hi = FIELD_NUM_RANGES[f"{section_key}.field_num"] + existing_fn = _collect_field_nums( + scalars, "field_num" + ) | _collect_field_nums(arrays, "field_num") + total += _assign_to_field_list( + scalars, + "field_num", + lo, + hi, + existing_fn, + f"{section_key}.scalars.field_num", + ) + total += _assign_to_field_list( + arrays, + "field_num", + lo, + hi, + existing_fn, + f"{section_key}.arrays.field_num", + ) + + # array_id — separate namespace + lo, hi = FIELD_NUM_RANGES[f"{section_key}.array_id"] + existing_aid = _collect_field_nums(arrays, "array_id") + total += _assign_to_field_list( + arrays, + "array_id", + lo, + hi, + existing_aid, + f"{section_key}.arrays.array_id", + ) + + # Collect all solution array_ids into a shared pool (ResultFieldId is global). + shared_result_ids = set() + all_solution_array_lists = [] + for section_key in ["lp_solution", "mip_solution"]: + section = data.get(section_key) + if not section: + continue + arr = section.get("arrays", []) + shared_result_ids |= _collect_field_nums(arr, "array_id") + all_solution_array_lists.append((section_key, arr)) + ws = section.get("warm_start") + if ws: + ws_arr = ws.get("arrays", []) + shared_result_ids |= _collect_field_nums(ws_arr, "array_id") + all_solution_array_lists.append( + (f"{section_key}.warm_start", ws_arr) + ) + + for section_key in ["lp_solution", "mip_solution"]: + section = data.get(section_key) + if not section: + continue + + # Scalars — field_num (range-bound for ChunkedResultHeader) + scalars = section.get("scalars", []) + lo, hi = FIELD_NUM_RANGES[f"{section_key}.scalars"] + existing = _collect_field_nums(scalars, "field_num") + total += _assign_to_field_list( + scalars, "field_num", lo, hi, existing, f"{section_key}.scalars" + ) + + # Arrays — field_num (per-message, no cap) + arrays = section.get("arrays", []) + arr_fn_range = FIELD_NUM_RANGES.get( + f"{section_key}.arrays.field_num", (1, None) + ) + existing_fn = _collect_field_nums(arrays, "field_num") + total += _assign_to_field_list( + arrays, + "field_num", + arr_fn_range[0], + arr_fn_range[1], + existing_fn, + f"{section_key}.arrays.field_num", + ) + + # Arrays — array_id (shared ResultFieldId pool, no cap) + total += _assign_to_field_list( + arrays, + "array_id", + 0, + None, + shared_result_ids, + f"{section_key}.arrays.array_id", + ) + + # Warm start + ws = section.get("warm_start") + if ws: + ws_scalars = ws.get("scalars", []) + lo, hi = FIELD_NUM_RANGES.get( + f"{section_key}.warm_start.scalars", (3000, 3999) + ) + existing = _collect_field_nums(ws_scalars, "field_num") + total += _assign_to_field_list( + ws_scalars, + "field_num", + lo, + hi, + existing, + f"{section_key}.warm_start.scalars", + ) + + ws_arrays = ws.get("arrays", []) + ws_fn_range = FIELD_NUM_RANGES.get( + f"{section_key}.warm_start.arrays.field_num", (1, None) + ) + existing_fn = _collect_field_nums(ws_arrays, "field_num") + total += _assign_to_field_list( + ws_arrays, + "field_num", + ws_fn_range[0], + ws_fn_range[1], + existing_fn, + f"{section_key}.warm_start.arrays.field_num", + ) + + # WS arrays — array_id (shared ResultFieldId pool) + total += _assign_to_field_list( + ws_arrays, + "array_id", + 0, + None, + shared_result_ids, + f"{section_key}.warm_start.arrays.array_id", + ) + + for section_key in ["pdlp_settings", "mip_settings"]: + section = data.get(section_key) + if not section: + continue + fields = section.get("fields", []) + lo, hi = FIELD_NUM_RANGES[section_key] + existing = _collect_settings_field_nums(fields) + total += _assign_to_settings_fields( + fields, lo, hi, existing, section_key + ) + + return total + + +# ============================================================================ +# main() +# ============================================================================ + + +def main(): + parser = argparse.ArgumentParser( + description="Generate conversion code from field registry" + ) + parser.add_argument( + "--registry", + default=str(Path(__file__).parent / "field_registry.yaml"), + ) + parser.add_argument( + "--output-dir", default=str(Path(__file__).parent / "generated") + ) + parser.add_argument( + "--auto-number", + action="store_true", + help="Auto-assign missing field_num and array_id values, writing " + "them back to the registry YAML (preserves comments/formatting).", + ) + parser.add_argument( + "--strip", + action="store_true", + help="Remove all field_num and array_id values from the registry YAML " + "and exit. Useful before committing a minimal registry.", + ) + args = parser.parse_args() + + if args.strip: + n = strip_field_numbers_text(args.registry) + if n > 0: + print(f" stripped {n} field number(s) from {args.registry}") + else: + print(" no field numbers to strip") + sys.exit(0) + + if args.auto_number: + try: + from ruamel.yaml import YAML + except ImportError: + print( + "ERROR: --auto-number requires ruamel.yaml. " + "Install with: pip install ruamel.yaml", + file=sys.stderr, + ) + sys.exit(1) + + ryaml = YAML(typ="rt") + ryaml.preserve_quotes = True + with open(args.registry) as f: + rt_data = ryaml.load(f) + + n = auto_assign_field_numbers(rt_data) + if n > 0: + _dump_and_verify(ryaml, rt_data, args.registry) + print(f" auto-numbered {n} field(s) in {args.registry}") + else: + print(" all field numbers already assigned") + + with open(args.registry) as f: + registry = yaml.safe_load(f) + + if not _registry_has_field_numbers(registry): + print( + "ERROR: Registry has no field_num or array_id values assigned.\n" + "Run with --auto-number to assign them before generating.", + file=sys.stderr, + ) + sys.exit(1) + + outdir = args.output_dir + + # Proto reference .inc files + write_file( + os.path.join(outdir, "generated_result_enums.proto.inc"), + HEADER + + "// ResultFieldId enum entries\n" + + generate_proto_result_enums(registry) + + "\n", + ) + + # Settings conversion .inc + for key, label in [("pdlp_settings", "pdlp"), ("mip_settings", "mip")]: + if key in registry: + obj = registry[key] + write_file( + os.path.join( + outdir, f"generated_{label}_settings_to_proto.inc" + ), + HEADER + + generate_settings_to_proto_body(registry, key, obj) + + "\n", + ) + write_file( + os.path.join( + outdir, f"generated_proto_to_{label}_settings.inc" + ), + HEADER + + generate_proto_to_settings_body(registry, key, obj) + + "\n", + ) + + # Full data proto + write_file( + os.path.join(outdir, "cuopt_remote_data.proto"), + generate_data_proto(registry), + ) + + # Per-domain enum converters + for domain in ("settings", "solution", "problem"): + write_file( + os.path.join(outdir, f"generated_enum_converters_{domain}.inc"), + HEADER + generate_enum_converters_inc(registry, domain) + "\n", + ) + + # Solution conversion .inc files + for key, label in [("lp_solution", "lp"), ("mip_solution", "mip")]: + obj = registry.get(key) + if not obj: + continue + write_file( + os.path.join(outdir, f"generated_{label}_solution_to_proto.inc"), + HEADER + _gen_solution_to_proto(registry, key, obj) + "\n", + ) + write_file( + os.path.join(outdir, f"generated_proto_to_{label}_solution.inc"), + HEADER + _gen_proto_to_solution(registry, key, obj) + "\n", + ) + write_file( + os.path.join(outdir, f"generated_{label}_chunked_header.inc"), + HEADER + _gen_chunked_header(registry, key, obj) + "\n", + ) + write_file( + os.path.join(outdir, f"generated_collect_{label}_arrays.inc"), + HEADER + _gen_collect_arrays(registry, key, obj) + "\n", + ) + write_file( + os.path.join(outdir, f"generated_chunked_to_{label}_solution.inc"), + HEADER + _gen_chunked_to_solution(registry, key, obj) + "\n", + ) + write_file( + os.path.join(outdir, f"generated_estimate_{label}_size.inc"), + HEADER + _gen_estimate_size(registry, key, obj) + "\n", + ) + + # Problem conversion .inc files + if "optimization_problem" in registry: + write_file( + os.path.join(outdir, "generated_problem_to_proto.inc"), + HEADER + _gen_problem_to_proto(registry) + "\n", + ) + write_file( + os.path.join(outdir, "generated_proto_to_problem.inc"), + HEADER + _gen_proto_to_problem(registry) + "\n", + ) + write_file( + os.path.join(outdir, "generated_estimate_problem_size.inc"), + HEADER + _gen_estimate_problem_size(registry) + "\n", + ) + write_file( + os.path.join(outdir, "generated_populate_chunked_header_lp.inc"), + HEADER + _gen_populate_chunked_header(registry, "lp") + "\n", + ) + write_file( + os.path.join(outdir, "generated_populate_chunked_header_mip.inc"), + HEADER + _gen_populate_chunked_header(registry, "mip") + "\n", + ) + write_file( + os.path.join(outdir, "generated_chunked_header_to_problem.inc"), + HEADER + _gen_chunked_header_to_problem(registry) + "\n", + ) + write_file( + os.path.join(outdir, "generated_chunked_arrays_to_problem.inc"), + HEADER + _gen_chunked_arrays_to_problem(registry) + "\n", + ) + write_file( + os.path.join(outdir, "generated_build_array_chunks.inc"), + HEADER + _gen_build_array_chunk_requests(registry) + "\n", + ) + write_file( + os.path.join(outdir, "generated_array_field_element_size.inc"), + HEADER + generate_array_field_element_size_inc(registry) + "\n", + ) + + print(f"\nDone! Generated {len(os.listdir(outdir))} files in: {outdir}") + + +if __name__ == "__main__": + main() diff --git a/cpp/codegen/generated/cuopt_remote_data.proto b/cpp/codegen/generated/cuopt_remote_data.proto new file mode 100644 index 000000000..70d5f92d9 --- /dev/null +++ b/cpp/codegen/generated/cuopt_remote_data.proto @@ -0,0 +1,276 @@ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT — regenerate with: python cpp/codegen/generate_conversions.py + +syntax = "proto3"; + +package cuopt.remote; + +enum PDLPTerminationStatus { + PDLP_NO_TERMINATION = 0; + PDLP_NUMERICAL_ERROR = 1; + PDLP_OPTIMAL = 2; + PDLP_PRIMAL_INFEASIBLE = 3; + PDLP_DUAL_INFEASIBLE = 4; + PDLP_ITERATION_LIMIT = 5; + PDLP_TIME_LIMIT = 6; + PDLP_CONCURRENT_LIMIT = 7; + PDLP_PRIMAL_FEASIBLE = 8; +} + +enum MIPTerminationStatus { + MIP_NO_TERMINATION = 0; + MIP_OPTIMAL = 1; + MIP_FEASIBLE_FOUND = 2; + MIP_INFEASIBLE = 3; + MIP_UNBOUNDED = 4; + MIP_TIME_LIMIT = 5; + MIP_WORK_LIMIT = 6; +} + +enum PDLPSolverMode { + Stable1 = 0; + Stable2 = 1; + Methodical1 = 2; + Fast1 = 3; + Stable3 = 4; +} + +enum LPMethod { + Concurrent = 0; + PDLP = 1; + DualSimplex = 2; + Barrier = 3; +} + +enum VariableType { + CONTINUOUS = 0; + INTEGER = 1; +} + +enum ProblemCategory { + LP = 0; + MIP = 1; +} + +enum ResultFieldId { + RESULT_PRIMAL_SOLUTION = 0; + RESULT_DUAL_SOLUTION = 1; + RESULT_REDUCED_COST = 2; + RESULT_CURRENT_PRIMAL_SOLUTION_ = 3; + RESULT_CURRENT_DUAL_SOLUTION_ = 4; + RESULT_INITIAL_PRIMAL_AVERAGE_ = 5; + RESULT_INITIAL_DUAL_AVERAGE_ = 6; + RESULT_CURRENT_ATY_ = 7; + RESULT_SUM_PRIMAL_SOLUTIONS_ = 8; + RESULT_SUM_DUAL_SOLUTIONS_ = 9; + RESULT_LAST_RESTART_DUALITY_GAP_PRIMAL_SOLUTION_ = 10; + RESULT_LAST_RESTART_DUALITY_GAP_DUAL_SOLUTION_ = 11; + RESULT_MIP_SOLUTION = 12; +} + +enum ArrayFieldId { + FIELD_VARIABLE_NAMES = 0; + FIELD_ROW_NAMES = 1; + FIELD_A_VALUES = 2; + FIELD_A_INDICES = 3; + FIELD_A_OFFSETS = 4; + FIELD_C = 5; + FIELD_B = 6; + FIELD_VARIABLE_LOWER_BOUNDS = 7; + FIELD_VARIABLE_UPPER_BOUNDS = 8; + FIELD_CONSTRAINT_LOWER_BOUNDS = 9; + FIELD_CONSTRAINT_UPPER_BOUNDS = 10; + FIELD_ROW_TYPES = 11; + FIELD_VARIABLE_TYPES = 12; + FIELD_INITIAL_PRIMAL_SOLUTION = 13; + FIELD_INITIAL_DUAL_SOLUTION = 14; + FIELD_Q_VALUES = 15; + FIELD_Q_INDICES = 16; + FIELD_Q_OFFSETS = 17; +} + +message OptimizationProblem { + string problem_name = 1; + string objective_name = 2; + bool maximize = 3; + double objective_scaling_factor = 4; + double objective_offset = 5; + ProblemCategory problem_category = 6; + repeated string variable_names = 7; + repeated string row_names = 8; + repeated double A_values = 9; + repeated int32 A_indices = 10; + repeated int32 A_offsets = 11; + repeated double c = 12; + repeated double b = 13; + repeated double variable_lower_bounds = 14; + repeated double variable_upper_bounds = 15; + repeated double constraint_lower_bounds = 16; + repeated double constraint_upper_bounds = 17; + bytes row_types = 18; + repeated VariableType variable_types = 19; + repeated double initial_primal_solution = 20; + repeated double initial_dual_solution = 21; + repeated double Q_values = 22; + repeated int32 Q_indices = 23; + repeated int32 Q_offsets = 24; +} + +message PDLPSolverSettings { + double absolute_gap_tolerance = 1; + double relative_gap_tolerance = 2; + double primal_infeasible_tolerance = 3; + double dual_infeasible_tolerance = 4; + double absolute_dual_tolerance = 5; + double relative_dual_tolerance = 6; + double absolute_primal_tolerance = 7; + double relative_primal_tolerance = 8; + double time_limit = 9; + int64 iteration_limit = 10; + bool log_to_console = 11; + bool detect_infeasibility = 12; + bool strict_infeasibility = 13; + PDLPSolverMode pdlp_solver_mode = 14; + LPMethod method = 15; + int32 presolver = 16; + bool dual_postsolve = 17; + bool crossover = 18; + int32 num_gpus = 19; + bool per_constraint_residual = 20; + bool cudss_deterministic = 21; + int32 folding = 22; + int32 augmented = 23; + int32 dualize = 24; + int32 ordering = 25; + int32 barrier_dual_initial_point = 26; + bool eliminate_dense_columns = 27; + bool save_best_primal_so_far = 28; + bool first_primal_feasible = 29; + int32 pdlp_precision = 30; + PDLPWarmStartData warm_start_data = 50; +} + +message MIPSolverSettings { + double time_limit = 1; + double relative_mip_gap = 2; + double absolute_mip_gap = 3; + double integrality_tolerance = 4; + double absolute_tolerance = 5; + double relative_tolerance = 6; + double presolve_absolute_tolerance = 7; + bool log_to_console = 8; + bool heuristics_only = 9; + int32 num_cpu_threads = 10; + int32 num_gpus = 11; + int32 presolver = 12; + bool mip_scaling = 13; + double work_limit = 14; + int32 node_limit = 15; + int32 reliability_branching = 16; + int32 mip_batch_pdlp_strong_branching = 17; + int32 max_cut_passes = 18; + int32 mir_cuts = 19; + int32 mixed_integer_gomory_cuts = 20; + int32 knapsack_cuts = 21; + int32 clique_cuts = 22; + int32 strong_chvatal_gomory_cuts = 23; + int32 reduced_cost_strengthening = 24; + double cut_change_threshold = 25; + double cut_min_orthogonality = 26; + int32 determinism_mode = 27; + int32 seed = 28; +} + +message PDLPWarmStartData { + repeated double current_primal_solution_ = 1; + repeated double current_dual_solution_ = 2; + repeated double initial_primal_average_ = 3; + repeated double initial_dual_average_ = 4; + repeated double current_ATY_ = 5; + repeated double sum_primal_solutions_ = 6; + repeated double sum_dual_solutions_ = 7; + repeated double last_restart_duality_gap_primal_solution_ = 8; + repeated double last_restart_duality_gap_dual_solution_ = 9; + double initial_primal_weight_ = 3000; + double initial_step_size_ = 3001; + int32 total_pdlp_iterations_ = 3002; + int32 total_pdhg_iterations_ = 3003; + double last_candidate_kkt_score_ = 3004; + double last_restart_kkt_score_ = 3005; + double sum_solution_weight_ = 3006; + int32 iterations_since_last_restart_ = 3007; +} + +message LPSolution { + repeated double primal_solution = 1; + repeated double dual_solution = 2; + repeated double reduced_cost = 3; + PDLPWarmStartData warm_start_data = 4; + PDLPTerminationStatus lp_termination_status = 1000; + string error_message = 1001; + double l2_primal_residual = 1002; + double l2_dual_residual = 1003; + double primal_objective = 1004; + double dual_objective = 1005; + double gap = 1006; + int32 nb_iterations = 1007; + double solve_time = 1008; + bool solved_by_pdlp = 1009; +} + +message MIPSolution { + repeated double mip_solution = 1; + MIPTerminationStatus mip_termination_status = 2000; + string mip_error_message = 2001; + double mip_objective = 2002; + double mip_gap = 2003; + double solution_bound = 2004; + double total_solve_time = 2005; + double presolve_time = 2006; + double max_constraint_violation = 2007; + double max_int_violation = 2008; + double max_variable_bound_violation = 2009; + int32 nodes = 2010; + int32 simplex_iterations = 2011; +} + +message ResultArrayDescriptor { + ResultFieldId field_id = 1; + int64 total_elements = 2; + int64 element_size_bytes = 3; +} + +message ChunkedResultHeader { + ProblemCategory problem_category = 1; + repeated ResultArrayDescriptor arrays = 50; + PDLPTerminationStatus lp_termination_status = 1000; + string error_message = 1001; + double l2_primal_residual = 1002; + double l2_dual_residual = 1003; + double primal_objective = 1004; + double dual_objective = 1005; + double gap = 1006; + int32 nb_iterations = 1007; + double solve_time = 1008; + bool solved_by_pdlp = 1009; + MIPTerminationStatus mip_termination_status = 2000; + string mip_error_message = 2001; + double mip_objective = 2002; + double mip_gap = 2003; + double solution_bound = 2004; + double total_solve_time = 2005; + double presolve_time = 2006; + double max_constraint_violation = 2007; + double max_int_violation = 2008; + double max_variable_bound_violation = 2009; + int32 nodes = 2010; + int32 simplex_iterations = 2011; + double initial_primal_weight_ = 3000; + double initial_step_size_ = 3001; + int32 total_pdlp_iterations_ = 3002; + int32 total_pdhg_iterations_ = 3003; + double last_candidate_kkt_score_ = 3004; + double last_restart_kkt_score_ = 3005; + double sum_solution_weight_ = 3006; + int32 iterations_since_last_restart_ = 3007; +} diff --git a/cpp/codegen/generated/generated_array_field_element_size.inc b/cpp/codegen/generated/generated_array_field_element_size.inc new file mode 100644 index 000000000..9687d2ccb --- /dev/null +++ b/cpp/codegen/generated/generated_array_field_element_size.inc @@ -0,0 +1,17 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + switch (field_id) { + case cuopt::remote::FIELD_VARIABLE_NAMES: + case cuopt::remote::FIELD_ROW_NAMES: + case cuopt::remote::FIELD_ROW_TYPES: + return 1; + case cuopt::remote::FIELD_A_INDICES: + case cuopt::remote::FIELD_A_OFFSETS: + case cuopt::remote::FIELD_VARIABLE_TYPES: + case cuopt::remote::FIELD_Q_INDICES: + case cuopt::remote::FIELD_Q_OFFSETS: + return 4; + default: return 8; + } diff --git a/cpp/codegen/generated/generated_build_array_chunks.inc b/cpp/codegen/generated/generated_build_array_chunks.inc new file mode 100644 index 000000000..6f086bb58 --- /dev/null +++ b/cpp/codegen/generated/generated_build_array_chunks.inc @@ -0,0 +1,80 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + std::vector requests; + + { + auto _blob = names_to_blob(problem.get_variable_names()); + chunk_byte_blob(requests, cuopt::remote::FIELD_VARIABLE_NAMES, _blob, upload_id, chunk_size_bytes); + } + { + auto _blob = names_to_blob(problem.get_row_names()); + chunk_byte_blob(requests, cuopt::remote::FIELD_ROW_NAMES, _blob, upload_id, chunk_size_bytes); + } + { + auto _c = problem.get_objective_coefficients_host(); + chunk_typed_array(requests, cuopt::remote::FIELD_C, _c, upload_id, chunk_size_bytes); + } + { + auto _b = problem.get_constraint_bounds_host(); + chunk_typed_array(requests, cuopt::remote::FIELD_B, _b, upload_id, chunk_size_bytes); + } + { + auto _variable_lower_bounds = problem.get_variable_lower_bounds_host(); + chunk_typed_array(requests, cuopt::remote::FIELD_VARIABLE_LOWER_BOUNDS, _variable_lower_bounds, upload_id, chunk_size_bytes); + } + { + auto _variable_upper_bounds = problem.get_variable_upper_bounds_host(); + chunk_typed_array(requests, cuopt::remote::FIELD_VARIABLE_UPPER_BOUNDS, _variable_upper_bounds, upload_id, chunk_size_bytes); + } + { + auto _constraint_lower_bounds = problem.get_constraint_lower_bounds_host(); + chunk_typed_array(requests, cuopt::remote::FIELD_CONSTRAINT_LOWER_BOUNDS, _constraint_lower_bounds, upload_id, chunk_size_bytes); + } + { + auto _constraint_upper_bounds = problem.get_constraint_upper_bounds_host(); + chunk_typed_array(requests, cuopt::remote::FIELD_CONSTRAINT_UPPER_BOUNDS, _constraint_upper_bounds, upload_id, chunk_size_bytes); + } + { + auto _row_types = problem.get_row_types_host(); + if (!_row_types.empty()) { + std::vector _bytes(_row_types.begin(), _row_types.end()); + chunk_byte_blob(requests, cuopt::remote::FIELD_ROW_TYPES, _bytes, upload_id, chunk_size_bytes); + } + } + { + auto _variable_types = problem.get_variable_types_host(); + if (!_variable_types.empty()) { + std::vector _variable_types_ints; + _variable_types_ints.reserve(_variable_types.size()); + for (const auto& v : _variable_types) _variable_types_ints.push_back(static_cast(to_proto_variable_type(v))); + chunk_typed_array(requests, cuopt::remote::FIELD_VARIABLE_TYPES, _variable_types_ints, upload_id, chunk_size_bytes); + } + } + { + const auto& _A_values = problem.get_constraint_matrix_values_host(); + chunk_typed_array(requests, cuopt::remote::FIELD_A_VALUES, _A_values, upload_id, chunk_size_bytes); + } + { + const auto& _A_indices = problem.get_constraint_matrix_indices_host(); + chunk_typed_array(requests, cuopt::remote::FIELD_A_INDICES, _A_indices, upload_id, chunk_size_bytes); + } + { + const auto& _A_offsets = problem.get_constraint_matrix_offsets_host(); + chunk_typed_array(requests, cuopt::remote::FIELD_A_OFFSETS, _A_offsets, upload_id, chunk_size_bytes); + } + { + const auto& _Q_values = problem.get_quadratic_objective_values_host(); + chunk_typed_array(requests, cuopt::remote::FIELD_Q_VALUES, _Q_values, upload_id, chunk_size_bytes); + } + { + const auto& _Q_indices = problem.get_quadratic_objective_indices_host(); + chunk_typed_array(requests, cuopt::remote::FIELD_Q_INDICES, _Q_indices, upload_id, chunk_size_bytes); + } + { + const auto& _Q_offsets = problem.get_quadratic_objective_offsets_host(); + chunk_typed_array(requests, cuopt::remote::FIELD_Q_OFFSETS, _Q_offsets, upload_id, chunk_size_bytes); + } + + return requests; diff --git a/cpp/codegen/generated/generated_chunked_arrays_to_problem.inc b/cpp/codegen/generated/generated_chunked_arrays_to_problem.inc new file mode 100644 index 000000000..a1de7631a --- /dev/null +++ b/cpp/codegen/generated/generated_chunked_arrays_to_problem.inc @@ -0,0 +1,124 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + map_chunked_header_to_problem(header, cpu_problem); + + auto get_doubles = [&](int32_t field_id) -> std::vector { + auto it = arrays.find(field_id); + if (it == arrays.end() || it->second.empty()) return {}; + if (it->second.size() % sizeof(double) != 0) return {}; + size_t n = it->second.size() / sizeof(double); + if constexpr (std::is_same_v) { + std::vector v(n); + std::memcpy(v.data(), it->second.data(), n * sizeof(double)); + return v; + } else { + std::vector tmp(n); + std::memcpy(tmp.data(), it->second.data(), n * sizeof(double)); + return std::vector(tmp.begin(), tmp.end()); + } + }; + + auto get_ints = [&](int32_t field_id) -> std::vector { + auto it = arrays.find(field_id); + if (it == arrays.end() || it->second.empty()) return {}; + if (it->second.size() % sizeof(int32_t) != 0) return {}; + size_t n = it->second.size() / sizeof(int32_t); + if constexpr (std::is_same_v) { + std::vector v(n); + std::memcpy(v.data(), it->second.data(), n * sizeof(int32_t)); + return v; + } else { + std::vector tmp(n); + std::memcpy(tmp.data(), it->second.data(), n * sizeof(int32_t)); + return std::vector(tmp.begin(), tmp.end()); + } + }; + + auto get_bytes = [&](int32_t field_id) -> std::string { + auto it = arrays.find(field_id); + if (it == arrays.end() || it->second.empty()) return {}; + return std::string(reinterpret_cast(it->second.data()), it->second.size()); + }; + + auto get_string_list = [&](int32_t field_id) -> std::vector { + auto it = arrays.find(field_id); + if (it == arrays.end() || it->second.empty()) return {}; + std::vector names; + const char* s = reinterpret_cast(it->second.data()); + const char* s_end = s + it->second.size(); + while (s < s_end) { + const char* nul = static_cast(std::memchr(s, '\0', s_end - s)); + if (!nul) nul = s_end; + names.emplace_back(s, nul); + if (nul == s_end) break; + s = nul + 1; + } + return names; + }; + + auto A_values = get_doubles(cuopt::remote::FIELD_A_VALUES); + auto A_indices = get_ints(cuopt::remote::FIELD_A_INDICES); + auto A_offsets = get_ints(cuopt::remote::FIELD_A_OFFSETS); + if (!A_values.empty() && !A_indices.empty() && !A_offsets.empty()) { + cpu_problem.set_csr_constraint_matrix(A_values.data(), static_cast(A_values.size()), A_indices.data(), static_cast(A_indices.size()), A_offsets.data(), static_cast(A_offsets.size())); + } + + auto Q_values = get_doubles(cuopt::remote::FIELD_Q_VALUES); + auto Q_indices = get_ints(cuopt::remote::FIELD_Q_INDICES); + auto Q_offsets = get_ints(cuopt::remote::FIELD_Q_OFFSETS); + if (!Q_values.empty() && !Q_indices.empty() && !Q_offsets.empty()) { + cpu_problem.set_quadratic_objective_matrix(Q_values.data(), static_cast(Q_values.size()), Q_indices.data(), static_cast(Q_indices.size()), Q_offsets.data(), static_cast(Q_offsets.size())); + } + + auto variable_names = get_string_list(cuopt::remote::FIELD_VARIABLE_NAMES); + if (!variable_names.empty()) { cpu_problem.set_variable_names(variable_names); } + + auto row_names = get_string_list(cuopt::remote::FIELD_ROW_NAMES); + if (!row_names.empty()) { cpu_problem.set_row_names(row_names); } + + auto c = get_doubles(cuopt::remote::FIELD_C); + if (!c.empty()) { + cpu_problem.set_objective_coefficients(c.data(), static_cast(c.size())); + } + + auto b = get_doubles(cuopt::remote::FIELD_B); + if (!b.empty()) { + cpu_problem.set_constraint_bounds(b.data(), static_cast(b.size())); + } + + auto variable_lower_bounds = get_doubles(cuopt::remote::FIELD_VARIABLE_LOWER_BOUNDS); + if (!variable_lower_bounds.empty()) { + cpu_problem.set_variable_lower_bounds(variable_lower_bounds.data(), static_cast(variable_lower_bounds.size())); + } + + auto variable_upper_bounds = get_doubles(cuopt::remote::FIELD_VARIABLE_UPPER_BOUNDS); + if (!variable_upper_bounds.empty()) { + cpu_problem.set_variable_upper_bounds(variable_upper_bounds.data(), static_cast(variable_upper_bounds.size())); + } + + auto constraint_lower_bounds = get_doubles(cuopt::remote::FIELD_CONSTRAINT_LOWER_BOUNDS); + if (!constraint_lower_bounds.empty()) { + cpu_problem.set_constraint_lower_bounds(constraint_lower_bounds.data(), static_cast(constraint_lower_bounds.size())); + } + + auto constraint_upper_bounds = get_doubles(cuopt::remote::FIELD_CONSTRAINT_UPPER_BOUNDS); + if (!constraint_upper_bounds.empty()) { + cpu_problem.set_constraint_upper_bounds(constraint_upper_bounds.data(), static_cast(constraint_upper_bounds.size())); + } + + auto row_types_str = get_bytes(cuopt::remote::FIELD_ROW_TYPES); + if (!row_types_str.empty()) { + cpu_problem.set_row_types(row_types_str.data(), static_cast(row_types_str.size())); + } + + auto variable_types_ints = get_ints(cuopt::remote::FIELD_VARIABLE_TYPES); + if (!variable_types_ints.empty()) { + std::vector variable_types; + variable_types.reserve(variable_types_ints.size()); + for (auto v : variable_types_ints) { + variable_types.push_back(from_proto_variable_type(static_cast(v))); + } + cpu_problem.set_variable_types(variable_types.data(), static_cast(variable_types.size())); + } diff --git a/cpp/codegen/generated/generated_chunked_header_to_problem.inc b/cpp/codegen/generated/generated_chunked_header_to_problem.inc new file mode 100644 index 000000000..349e8d4e7 --- /dev/null +++ b/cpp/codegen/generated/generated_chunked_header_to_problem.inc @@ -0,0 +1,19 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + cpu_problem.set_problem_name(header.problem_name()); + cpu_problem.set_objective_name(header.objective_name()); + cpu_problem.set_maximize(header.maximize()); + cpu_problem.set_objective_scaling_factor(header.objective_scaling_factor()); + cpu_problem.set_objective_offset(header.objective_offset()); + cpu_problem.set_problem_category(from_proto_problem_category(header.problem_category())); + + if (header.variable_names_size() > 0) { + std::vector variable_names(header.variable_names().begin(), header.variable_names().end()); + cpu_problem.set_variable_names(variable_names); + } + if (header.row_names_size() > 0) { + std::vector row_names(header.row_names().begin(), header.row_names().end()); + cpu_problem.set_row_names(row_names); + } diff --git a/cpp/codegen/generated/generated_chunked_to_lp_solution.inc b/cpp/codegen/generated/generated_chunked_to_lp_solution.inc new file mode 100644 index 000000000..6b819b805 --- /dev/null +++ b/cpp/codegen/generated/generated_chunked_to_lp_solution.inc @@ -0,0 +1,42 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + auto primal_solution = bytes_to_typed(arrays, cuopt::remote::RESULT_PRIMAL_SOLUTION); + auto dual_solution = bytes_to_typed(arrays, cuopt::remote::RESULT_DUAL_SOLUTION); + auto reduced_cost = bytes_to_typed(arrays, cuopt::remote::RESULT_REDUCED_COST); + + auto _lp_termination_status = from_proto_pdlp_termination_status(h.lp_termination_status()); + auto _l2_primal_residual = static_cast(h.l2_primal_residual()); + auto _l2_dual_residual = static_cast(h.l2_dual_residual()); + auto _primal_objective = static_cast(h.primal_objective()); + auto _dual_objective = static_cast(h.dual_objective()); + auto _gap = static_cast(h.gap()); + auto _nb_iterations = static_cast(h.nb_iterations()); + auto _solve_time = static_cast(h.solve_time()); + auto _solved_by_pdlp = h.solved_by_pdlp(); + + auto _ws_detect = bytes_to_typed(arrays, cuopt::remote::RESULT_CURRENT_PRIMAL_SOLUTION_); + if (!_ws_detect.empty()) { + cpu_pdlp_warm_start_data_t ws; + ws.current_primal_solution_ = std::move(_ws_detect); + ws.current_dual_solution_ = bytes_to_typed(arrays, cuopt::remote::RESULT_CURRENT_DUAL_SOLUTION_); + ws.initial_primal_average_ = bytes_to_typed(arrays, cuopt::remote::RESULT_INITIAL_PRIMAL_AVERAGE_); + ws.initial_dual_average_ = bytes_to_typed(arrays, cuopt::remote::RESULT_INITIAL_DUAL_AVERAGE_); + ws.current_ATY_ = bytes_to_typed(arrays, cuopt::remote::RESULT_CURRENT_ATY_); + ws.sum_primal_solutions_ = bytes_to_typed(arrays, cuopt::remote::RESULT_SUM_PRIMAL_SOLUTIONS_); + ws.sum_dual_solutions_ = bytes_to_typed(arrays, cuopt::remote::RESULT_SUM_DUAL_SOLUTIONS_); + ws.last_restart_duality_gap_primal_solution_ = bytes_to_typed(arrays, cuopt::remote::RESULT_LAST_RESTART_DUALITY_GAP_PRIMAL_SOLUTION_); + ws.last_restart_duality_gap_dual_solution_ = bytes_to_typed(arrays, cuopt::remote::RESULT_LAST_RESTART_DUALITY_GAP_DUAL_SOLUTION_); + ws.initial_primal_weight_ = static_cast(h.initial_primal_weight_()); + ws.initial_step_size_ = static_cast(h.initial_step_size_()); + ws.total_pdlp_iterations_ = static_cast(h.total_pdlp_iterations_()); + ws.total_pdhg_iterations_ = static_cast(h.total_pdhg_iterations_()); + ws.last_candidate_kkt_score_ = static_cast(h.last_candidate_kkt_score_()); + ws.last_restart_kkt_score_ = static_cast(h.last_restart_kkt_score_()); + ws.sum_solution_weight_ = static_cast(h.sum_solution_weight_()); + ws.iterations_since_last_restart_ = static_cast(h.iterations_since_last_restart_()); + return cpu_lp_solution_t(std::move(primal_solution), std::move(dual_solution), std::move(reduced_cost), _lp_termination_status, _primal_objective, _dual_objective, _solve_time, _l2_primal_residual, _l2_dual_residual, _gap, _nb_iterations, _solved_by_pdlp, std::move(ws)); + } + + return cpu_lp_solution_t(std::move(primal_solution), std::move(dual_solution), std::move(reduced_cost), _lp_termination_status, _primal_objective, _dual_objective, _solve_time, _l2_primal_residual, _l2_dual_residual, _gap, _nb_iterations, _solved_by_pdlp); diff --git a/cpp/codegen/generated/generated_chunked_to_mip_solution.inc b/cpp/codegen/generated/generated_chunked_to_mip_solution.inc new file mode 100644 index 000000000..f09f0b94b --- /dev/null +++ b/cpp/codegen/generated/generated_chunked_to_mip_solution.inc @@ -0,0 +1,18 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + auto mip_solution = bytes_to_typed(arrays, cuopt::remote::RESULT_MIP_SOLUTION); + + auto _mip_termination_status = from_proto_mip_termination_status(h.mip_termination_status()); + auto _mip_objective = static_cast(h.mip_objective()); + auto _mip_gap = static_cast(h.mip_gap()); + auto _solution_bound = static_cast(h.solution_bound()); + auto _total_solve_time = static_cast(h.total_solve_time()); + auto _presolve_time = static_cast(h.presolve_time()); + auto _max_constraint_violation = static_cast(h.max_constraint_violation()); + auto _max_int_violation = static_cast(h.max_int_violation()); + auto _max_variable_bound_violation = static_cast(h.max_variable_bound_violation()); + auto _nodes = static_cast(h.nodes()); + auto _simplex_iterations = static_cast(h.simplex_iterations()); + return cpu_mip_solution_t(std::move(mip_solution), _mip_termination_status, _mip_objective, _mip_gap, _solution_bound, _total_solve_time, _presolve_time, _max_constraint_violation, _max_int_violation, _max_variable_bound_violation, _nodes, _simplex_iterations); diff --git a/cpp/codegen/generated/generated_collect_lp_arrays.inc b/cpp/codegen/generated/generated_collect_lp_arrays.inc new file mode 100644 index 000000000..d9e11bfb9 --- /dev/null +++ b/cpp/codegen/generated/generated_collect_lp_arrays.inc @@ -0,0 +1,24 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + std::map> arrays; + const auto& _primal_solution = solution.get_primal_solution_host(); + if (!_primal_solution.empty()) { arrays[cuopt::remote::RESULT_PRIMAL_SOLUTION] = doubles_to_bytes(_primal_solution); } + const auto& _dual_solution = solution.get_dual_solution_host(); + if (!_dual_solution.empty()) { arrays[cuopt::remote::RESULT_DUAL_SOLUTION] = doubles_to_bytes(_dual_solution); } + const auto& _reduced_cost = solution.get_reduced_cost_host(); + if (!_reduced_cost.empty()) { arrays[cuopt::remote::RESULT_REDUCED_COST] = doubles_to_bytes(_reduced_cost); } + if (solution.has_warm_start_data()) { + const auto& ws = solution.get_cpu_pdlp_warm_start_data(); + if (!ws.current_primal_solution_.empty()) { arrays[cuopt::remote::RESULT_CURRENT_PRIMAL_SOLUTION_] = doubles_to_bytes(ws.current_primal_solution_); } + if (!ws.current_dual_solution_.empty()) { arrays[cuopt::remote::RESULT_CURRENT_DUAL_SOLUTION_] = doubles_to_bytes(ws.current_dual_solution_); } + if (!ws.initial_primal_average_.empty()) { arrays[cuopt::remote::RESULT_INITIAL_PRIMAL_AVERAGE_] = doubles_to_bytes(ws.initial_primal_average_); } + if (!ws.initial_dual_average_.empty()) { arrays[cuopt::remote::RESULT_INITIAL_DUAL_AVERAGE_] = doubles_to_bytes(ws.initial_dual_average_); } + if (!ws.current_ATY_.empty()) { arrays[cuopt::remote::RESULT_CURRENT_ATY_] = doubles_to_bytes(ws.current_ATY_); } + if (!ws.sum_primal_solutions_.empty()) { arrays[cuopt::remote::RESULT_SUM_PRIMAL_SOLUTIONS_] = doubles_to_bytes(ws.sum_primal_solutions_); } + if (!ws.sum_dual_solutions_.empty()) { arrays[cuopt::remote::RESULT_SUM_DUAL_SOLUTIONS_] = doubles_to_bytes(ws.sum_dual_solutions_); } + if (!ws.last_restart_duality_gap_primal_solution_.empty()) { arrays[cuopt::remote::RESULT_LAST_RESTART_DUALITY_GAP_PRIMAL_SOLUTION_] = doubles_to_bytes(ws.last_restart_duality_gap_primal_solution_); } + if (!ws.last_restart_duality_gap_dual_solution_.empty()) { arrays[cuopt::remote::RESULT_LAST_RESTART_DUALITY_GAP_DUAL_SOLUTION_] = doubles_to_bytes(ws.last_restart_duality_gap_dual_solution_); } + } + return arrays; diff --git a/cpp/codegen/generated/generated_collect_mip_arrays.inc b/cpp/codegen/generated/generated_collect_mip_arrays.inc new file mode 100644 index 000000000..640a9a913 --- /dev/null +++ b/cpp/codegen/generated/generated_collect_mip_arrays.inc @@ -0,0 +1,8 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + std::map> arrays; + const auto& _mip_solution = solution.get_solution_host(); + if (!_mip_solution.empty()) { arrays[cuopt::remote::RESULT_MIP_SOLUTION] = doubles_to_bytes(_mip_solution); } + return arrays; diff --git a/cpp/codegen/generated/generated_enum_converters_problem.inc b/cpp/codegen/generated/generated_enum_converters_problem.inc new file mode 100644 index 000000000..bb4aa50e0 --- /dev/null +++ b/cpp/codegen/generated/generated_enum_converters_problem.inc @@ -0,0 +1,39 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ +cuopt::remote::VariableType to_proto_variable_type(var_t v) +{ + switch (v) { + case var_t::CONTINUOUS: return cuopt::remote::CONTINUOUS; + case var_t::INTEGER: return cuopt::remote::INTEGER; + default: return cuopt::remote::CONTINUOUS; + } +} + +var_t from_proto_variable_type(cuopt::remote::VariableType v) +{ + switch (v) { + case cuopt::remote::CONTINUOUS: return var_t::CONTINUOUS; + case cuopt::remote::INTEGER: return var_t::INTEGER; + default: return var_t::CONTINUOUS; + } +} + +cuopt::remote::ProblemCategory to_proto_problem_category(problem_category_t v) +{ + switch (v) { + case problem_category_t::LP: return cuopt::remote::LP; + case problem_category_t::MIP: return cuopt::remote::MIP; + default: return cuopt::remote::LP; + } +} + +problem_category_t from_proto_problem_category(cuopt::remote::ProblemCategory v) +{ + switch (v) { + case cuopt::remote::LP: return problem_category_t::LP; + case cuopt::remote::MIP: return problem_category_t::MIP; + default: return problem_category_t::LP; + } +} diff --git a/cpp/codegen/generated/generated_enum_converters_settings.inc b/cpp/codegen/generated/generated_enum_converters_settings.inc new file mode 100644 index 000000000..b8cd20f34 --- /dev/null +++ b/cpp/codegen/generated/generated_enum_converters_settings.inc @@ -0,0 +1,49 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ +cuopt::remote::PDLPSolverMode to_proto_pdlp_solver_mode(pdlp_solver_mode_t v) +{ + switch (v) { + case pdlp_solver_mode_t::Stable1: return cuopt::remote::Stable1; + case pdlp_solver_mode_t::Stable2: return cuopt::remote::Stable2; + case pdlp_solver_mode_t::Methodical1: return cuopt::remote::Methodical1; + case pdlp_solver_mode_t::Fast1: return cuopt::remote::Fast1; + case pdlp_solver_mode_t::Stable3: return cuopt::remote::Stable3; + default: return cuopt::remote::Stable3; + } +} + +pdlp_solver_mode_t from_proto_pdlp_solver_mode(cuopt::remote::PDLPSolverMode v) +{ + switch (v) { + case cuopt::remote::Stable1: return pdlp_solver_mode_t::Stable1; + case cuopt::remote::Stable2: return pdlp_solver_mode_t::Stable2; + case cuopt::remote::Methodical1: return pdlp_solver_mode_t::Methodical1; + case cuopt::remote::Fast1: return pdlp_solver_mode_t::Fast1; + case cuopt::remote::Stable3: return pdlp_solver_mode_t::Stable3; + default: return pdlp_solver_mode_t::Stable3; + } +} + +cuopt::remote::LPMethod to_proto_lp_method(method_t v) +{ + switch (v) { + case method_t::Concurrent: return cuopt::remote::Concurrent; + case method_t::PDLP: return cuopt::remote::PDLP; + case method_t::DualSimplex: return cuopt::remote::DualSimplex; + case method_t::Barrier: return cuopt::remote::Barrier; + default: return cuopt::remote::Concurrent; + } +} + +method_t from_proto_lp_method(cuopt::remote::LPMethod v) +{ + switch (v) { + case cuopt::remote::Concurrent: return method_t::Concurrent; + case cuopt::remote::PDLP: return method_t::PDLP; + case cuopt::remote::DualSimplex: return method_t::DualSimplex; + case cuopt::remote::Barrier: return method_t::Barrier; + default: return method_t::Concurrent; + } +} diff --git a/cpp/codegen/generated/generated_enum_converters_solution.inc b/cpp/codegen/generated/generated_enum_converters_solution.inc new file mode 100644 index 000000000..046d49c92 --- /dev/null +++ b/cpp/codegen/generated/generated_enum_converters_solution.inc @@ -0,0 +1,63 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ +cuopt::remote::PDLPTerminationStatus to_proto_pdlp_termination_status(pdlp_termination_status_t v) +{ + switch (v) { + case pdlp_termination_status_t::NoTermination: return cuopt::remote::PDLP_NO_TERMINATION; + case pdlp_termination_status_t::NumericalError: return cuopt::remote::PDLP_NUMERICAL_ERROR; + case pdlp_termination_status_t::Optimal: return cuopt::remote::PDLP_OPTIMAL; + case pdlp_termination_status_t::PrimalInfeasible: return cuopt::remote::PDLP_PRIMAL_INFEASIBLE; + case pdlp_termination_status_t::DualInfeasible: return cuopt::remote::PDLP_DUAL_INFEASIBLE; + case pdlp_termination_status_t::IterationLimit: return cuopt::remote::PDLP_ITERATION_LIMIT; + case pdlp_termination_status_t::TimeLimit: return cuopt::remote::PDLP_TIME_LIMIT; + case pdlp_termination_status_t::ConcurrentLimit: return cuopt::remote::PDLP_CONCURRENT_LIMIT; + case pdlp_termination_status_t::PrimalFeasible: return cuopt::remote::PDLP_PRIMAL_FEASIBLE; + default: return cuopt::remote::PDLP_NO_TERMINATION; + } +} + +pdlp_termination_status_t from_proto_pdlp_termination_status(cuopt::remote::PDLPTerminationStatus v) +{ + switch (v) { + case cuopt::remote::PDLP_NO_TERMINATION: return pdlp_termination_status_t::NoTermination; + case cuopt::remote::PDLP_NUMERICAL_ERROR: return pdlp_termination_status_t::NumericalError; + case cuopt::remote::PDLP_OPTIMAL: return pdlp_termination_status_t::Optimal; + case cuopt::remote::PDLP_PRIMAL_INFEASIBLE: return pdlp_termination_status_t::PrimalInfeasible; + case cuopt::remote::PDLP_DUAL_INFEASIBLE: return pdlp_termination_status_t::DualInfeasible; + case cuopt::remote::PDLP_ITERATION_LIMIT: return pdlp_termination_status_t::IterationLimit; + case cuopt::remote::PDLP_TIME_LIMIT: return pdlp_termination_status_t::TimeLimit; + case cuopt::remote::PDLP_CONCURRENT_LIMIT: return pdlp_termination_status_t::ConcurrentLimit; + case cuopt::remote::PDLP_PRIMAL_FEASIBLE: return pdlp_termination_status_t::PrimalFeasible; + default: return pdlp_termination_status_t::NoTermination; + } +} + +cuopt::remote::MIPTerminationStatus to_proto_mip_termination_status(mip_termination_status_t v) +{ + switch (v) { + case mip_termination_status_t::NoTermination: return cuopt::remote::MIP_NO_TERMINATION; + case mip_termination_status_t::Optimal: return cuopt::remote::MIP_OPTIMAL; + case mip_termination_status_t::FeasibleFound: return cuopt::remote::MIP_FEASIBLE_FOUND; + case mip_termination_status_t::Infeasible: return cuopt::remote::MIP_INFEASIBLE; + case mip_termination_status_t::Unbounded: return cuopt::remote::MIP_UNBOUNDED; + case mip_termination_status_t::TimeLimit: return cuopt::remote::MIP_TIME_LIMIT; + case mip_termination_status_t::WorkLimit: return cuopt::remote::MIP_WORK_LIMIT; + default: return cuopt::remote::MIP_NO_TERMINATION; + } +} + +mip_termination_status_t from_proto_mip_termination_status(cuopt::remote::MIPTerminationStatus v) +{ + switch (v) { + case cuopt::remote::MIP_NO_TERMINATION: return mip_termination_status_t::NoTermination; + case cuopt::remote::MIP_OPTIMAL: return mip_termination_status_t::Optimal; + case cuopt::remote::MIP_FEASIBLE_FOUND: return mip_termination_status_t::FeasibleFound; + case cuopt::remote::MIP_INFEASIBLE: return mip_termination_status_t::Infeasible; + case cuopt::remote::MIP_UNBOUNDED: return mip_termination_status_t::Unbounded; + case cuopt::remote::MIP_TIME_LIMIT: return mip_termination_status_t::TimeLimit; + case cuopt::remote::MIP_WORK_LIMIT: return mip_termination_status_t::WorkLimit; + default: return mip_termination_status_t::NoTermination; + } +} diff --git a/cpp/codegen/generated/generated_estimate_lp_size.inc b/cpp/codegen/generated/generated_estimate_lp_size.inc new file mode 100644 index 000000000..b12880c10 --- /dev/null +++ b/cpp/codegen/generated/generated_estimate_lp_size.inc @@ -0,0 +1,22 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + size_t est = 0; + est += static_cast(solution.get_primal_solution_size()) * sizeof(double); + est += static_cast(solution.get_dual_solution_size()) * sizeof(double); + est += static_cast(solution.get_reduced_cost_size()) * sizeof(double); + if (solution.has_warm_start_data()) { + const auto& ws = solution.get_cpu_pdlp_warm_start_data(); + est += ws.current_primal_solution_.size() * sizeof(double); + est += ws.current_dual_solution_.size() * sizeof(double); + est += ws.initial_primal_average_.size() * sizeof(double); + est += ws.initial_dual_average_.size() * sizeof(double); + est += ws.current_ATY_.size() * sizeof(double); + est += ws.sum_primal_solutions_.size() * sizeof(double); + est += ws.sum_dual_solutions_.size() * sizeof(double); + est += ws.last_restart_duality_gap_primal_solution_.size() * sizeof(double); + est += ws.last_restart_duality_gap_dual_solution_.size() * sizeof(double); + } + est += 512; + return est; diff --git a/cpp/codegen/generated/generated_estimate_mip_size.inc b/cpp/codegen/generated/generated_estimate_mip_size.inc new file mode 100644 index 000000000..2e5c08792 --- /dev/null +++ b/cpp/codegen/generated/generated_estimate_mip_size.inc @@ -0,0 +1,8 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + size_t est = 0; + est += static_cast(solution.get_solution_size()) * sizeof(double); + est += 256; + return est; diff --git a/cpp/codegen/generated/generated_estimate_problem_size.inc b/cpp/codegen/generated/generated_estimate_problem_size.inc new file mode 100644 index 000000000..e88e5b839 --- /dev/null +++ b/cpp/codegen/generated/generated_estimate_problem_size.inc @@ -0,0 +1,23 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + size_t est = 0; + for (const auto& s : cpu_problem.get_variable_names()) est += s.size() + 2; + for (const auto& s : cpu_problem.get_row_names()) est += s.size() + 2; + est += cpu_problem.get_constraint_matrix_values_host().size() * sizeof(double); + est += cpu_problem.get_constraint_matrix_indices_host().size() * 5; + est += cpu_problem.get_constraint_matrix_offsets_host().size() * 5; + est += cpu_problem.get_objective_coefficients_host().size() * sizeof(double); + est += cpu_problem.get_constraint_bounds_host().size() * sizeof(double); + est += cpu_problem.get_variable_lower_bounds_host().size() * sizeof(double); + est += cpu_problem.get_variable_upper_bounds_host().size() * sizeof(double); + est += cpu_problem.get_constraint_lower_bounds_host().size() * sizeof(double); + est += cpu_problem.get_constraint_upper_bounds_host().size() * sizeof(double); + est += cpu_problem.get_row_types_host().size(); + est += cpu_problem.get_variable_types_host().size() * 4; + est += cpu_problem.get_quadratic_objective_values_host().size() * sizeof(double); + est += cpu_problem.get_quadratic_objective_indices_host().size() * 5; + est += cpu_problem.get_quadratic_objective_offsets_host().size() * 5; + est += 512; + return est; diff --git a/cpp/codegen/generated/generated_lp_chunked_header.inc b/cpp/codegen/generated/generated_lp_chunked_header.inc new file mode 100644 index 000000000..4c8b9d864 --- /dev/null +++ b/cpp/codegen/generated/generated_lp_chunked_header.inc @@ -0,0 +1,41 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + header->set_problem_category(cuopt::remote::LP); + header->set_lp_termination_status(to_proto_pdlp_termination_status(solution.get_termination_status())); + header->set_error_message(solution.get_error_status().what()); + header->set_l2_primal_residual(solution.get_l2_primal_residual()); + header->set_l2_dual_residual(solution.get_l2_dual_residual()); + header->set_primal_objective(solution.get_objective_value()); + header->set_dual_objective(solution.get_dual_objective_value()); + header->set_gap(solution.get_gap()); + header->set_nb_iterations(solution.get_num_iterations()); + header->set_solve_time(solution.get_solve_time()); + header->set_solved_by_pdlp(solution.is_solved_by_pdlp()); + + add_result_array_descriptor(header, cuopt::remote::RESULT_PRIMAL_SOLUTION, solution.get_primal_solution_host().size(), sizeof(double)); + add_result_array_descriptor(header, cuopt::remote::RESULT_DUAL_SOLUTION, solution.get_dual_solution_host().size(), sizeof(double)); + add_result_array_descriptor(header, cuopt::remote::RESULT_REDUCED_COST, solution.get_reduced_cost_host().size(), sizeof(double)); + + if (solution.has_warm_start_data()) { + const auto& ws = solution.get_cpu_pdlp_warm_start_data(); + header->set_initial_primal_weight_(static_cast(ws.initial_primal_weight_)); + header->set_initial_step_size_(static_cast(ws.initial_step_size_)); + header->set_total_pdlp_iterations_(static_cast(ws.total_pdlp_iterations_)); + header->set_total_pdhg_iterations_(static_cast(ws.total_pdhg_iterations_)); + header->set_last_candidate_kkt_score_(static_cast(ws.last_candidate_kkt_score_)); + header->set_last_restart_kkt_score_(static_cast(ws.last_restart_kkt_score_)); + header->set_sum_solution_weight_(static_cast(ws.sum_solution_weight_)); + header->set_iterations_since_last_restart_(static_cast(ws.iterations_since_last_restart_)); + + add_result_array_descriptor(header, cuopt::remote::RESULT_CURRENT_PRIMAL_SOLUTION_, ws.current_primal_solution_.size(), sizeof(double)); + add_result_array_descriptor(header, cuopt::remote::RESULT_CURRENT_DUAL_SOLUTION_, ws.current_dual_solution_.size(), sizeof(double)); + add_result_array_descriptor(header, cuopt::remote::RESULT_INITIAL_PRIMAL_AVERAGE_, ws.initial_primal_average_.size(), sizeof(double)); + add_result_array_descriptor(header, cuopt::remote::RESULT_INITIAL_DUAL_AVERAGE_, ws.initial_dual_average_.size(), sizeof(double)); + add_result_array_descriptor(header, cuopt::remote::RESULT_CURRENT_ATY_, ws.current_ATY_.size(), sizeof(double)); + add_result_array_descriptor(header, cuopt::remote::RESULT_SUM_PRIMAL_SOLUTIONS_, ws.sum_primal_solutions_.size(), sizeof(double)); + add_result_array_descriptor(header, cuopt::remote::RESULT_SUM_DUAL_SOLUTIONS_, ws.sum_dual_solutions_.size(), sizeof(double)); + add_result_array_descriptor(header, cuopt::remote::RESULT_LAST_RESTART_DUALITY_GAP_PRIMAL_SOLUTION_, ws.last_restart_duality_gap_primal_solution_.size(), sizeof(double)); + add_result_array_descriptor(header, cuopt::remote::RESULT_LAST_RESTART_DUALITY_GAP_DUAL_SOLUTION_, ws.last_restart_duality_gap_dual_solution_.size(), sizeof(double)); + } diff --git a/cpp/codegen/generated/generated_lp_solution_to_proto.inc b/cpp/codegen/generated/generated_lp_solution_to_proto.inc new file mode 100644 index 000000000..c1199da06 --- /dev/null +++ b/cpp/codegen/generated/generated_lp_solution_to_proto.inc @@ -0,0 +1,43 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + pb_solution->set_lp_termination_status(to_proto_pdlp_termination_status(solution.get_termination_status())); + pb_solution->set_error_message(solution.get_error_status().what()); + pb_solution->set_l2_primal_residual(solution.get_l2_primal_residual()); + pb_solution->set_l2_dual_residual(solution.get_l2_dual_residual()); + pb_solution->set_primal_objective(solution.get_objective_value()); + pb_solution->set_dual_objective(solution.get_dual_objective_value()); + pb_solution->set_gap(solution.get_gap()); + pb_solution->set_nb_iterations(solution.get_num_iterations()); + pb_solution->set_solve_time(solution.get_solve_time()); + pb_solution->set_solved_by_pdlp(solution.is_solved_by_pdlp()); + + const auto& _primal_solution = solution.get_primal_solution_host(); + for (const auto& v : _primal_solution) pb_solution->add_primal_solution(static_cast(v)); + const auto& _dual_solution = solution.get_dual_solution_host(); + for (const auto& v : _dual_solution) pb_solution->add_dual_solution(static_cast(v)); + const auto& _reduced_cost = solution.get_reduced_cost_host(); + for (const auto& v : _reduced_cost) pb_solution->add_reduced_cost(static_cast(v)); + + if (solution.has_warm_start_data()) { + auto* pb_ws = pb_solution->mutable_warm_start_data(); + const auto& ws = solution.get_cpu_pdlp_warm_start_data(); + for (const auto& v : ws.current_primal_solution_) pb_ws->add_current_primal_solution_(static_cast(v)); + for (const auto& v : ws.current_dual_solution_) pb_ws->add_current_dual_solution_(static_cast(v)); + for (const auto& v : ws.initial_primal_average_) pb_ws->add_initial_primal_average_(static_cast(v)); + for (const auto& v : ws.initial_dual_average_) pb_ws->add_initial_dual_average_(static_cast(v)); + for (const auto& v : ws.current_ATY_) pb_ws->add_current_aty_(static_cast(v)); + for (const auto& v : ws.sum_primal_solutions_) pb_ws->add_sum_primal_solutions_(static_cast(v)); + for (const auto& v : ws.sum_dual_solutions_) pb_ws->add_sum_dual_solutions_(static_cast(v)); + for (const auto& v : ws.last_restart_duality_gap_primal_solution_) pb_ws->add_last_restart_duality_gap_primal_solution_(static_cast(v)); + for (const auto& v : ws.last_restart_duality_gap_dual_solution_) pb_ws->add_last_restart_duality_gap_dual_solution_(static_cast(v)); + pb_ws->set_initial_primal_weight_(static_cast(ws.initial_primal_weight_)); + pb_ws->set_initial_step_size_(static_cast(ws.initial_step_size_)); + pb_ws->set_total_pdlp_iterations_(static_cast(ws.total_pdlp_iterations_)); + pb_ws->set_total_pdhg_iterations_(static_cast(ws.total_pdhg_iterations_)); + pb_ws->set_last_candidate_kkt_score_(static_cast(ws.last_candidate_kkt_score_)); + pb_ws->set_last_restart_kkt_score_(static_cast(ws.last_restart_kkt_score_)); + pb_ws->set_sum_solution_weight_(static_cast(ws.sum_solution_weight_)); + pb_ws->set_iterations_since_last_restart_(static_cast(ws.iterations_since_last_restart_)); + } diff --git a/cpp/codegen/generated/generated_mip_chunked_header.inc b/cpp/codegen/generated/generated_mip_chunked_header.inc new file mode 100644 index 000000000..c6d0ed492 --- /dev/null +++ b/cpp/codegen/generated/generated_mip_chunked_header.inc @@ -0,0 +1,19 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + header->set_problem_category(cuopt::remote::MIP); + header->set_mip_termination_status(to_proto_mip_termination_status(solution.get_termination_status())); + header->set_mip_error_message(solution.get_error_status().what()); + header->set_mip_objective(solution.get_objective_value()); + header->set_mip_gap(solution.get_mip_gap()); + header->set_solution_bound(solution.get_solution_bound()); + header->set_total_solve_time(solution.get_solve_time()); + header->set_presolve_time(solution.get_presolve_time()); + header->set_max_constraint_violation(solution.get_max_constraint_violation()); + header->set_max_int_violation(solution.get_max_int_violation()); + header->set_max_variable_bound_violation(solution.get_max_variable_bound_violation()); + header->set_nodes(solution.get_num_nodes()); + header->set_simplex_iterations(solution.get_num_simplex_iterations()); + + add_result_array_descriptor(header, cuopt::remote::RESULT_MIP_SOLUTION, solution.get_solution_host().size(), sizeof(double)); diff --git a/cpp/codegen/generated/generated_mip_settings_to_proto.inc b/cpp/codegen/generated/generated_mip_settings_to_proto.inc new file mode 100644 index 000000000..7d48e385f --- /dev/null +++ b/cpp/codegen/generated/generated_mip_settings_to_proto.inc @@ -0,0 +1,36 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + pb_settings->set_time_limit(settings.time_limit); + pb_settings->set_relative_mip_gap(settings.tolerances.relative_mip_gap); + pb_settings->set_absolute_mip_gap(settings.tolerances.absolute_mip_gap); + pb_settings->set_integrality_tolerance(settings.tolerances.integrality_tolerance); + pb_settings->set_absolute_tolerance(settings.tolerances.absolute_tolerance); + pb_settings->set_relative_tolerance(settings.tolerances.relative_tolerance); + pb_settings->set_presolve_absolute_tolerance(settings.tolerances.presolve_absolute_tolerance); + pb_settings->set_log_to_console(settings.log_to_console); + pb_settings->set_heuristics_only(settings.heuristics_only); + pb_settings->set_num_cpu_threads(settings.num_cpu_threads); + pb_settings->set_num_gpus(settings.num_gpus); + pb_settings->set_presolver(static_cast(settings.presolver)); + pb_settings->set_mip_scaling(settings.mip_scaling); + pb_settings->set_work_limit(settings.work_limit); + if (settings.node_limit == std::numeric_limits::max()) { + pb_settings->set_node_limit(-1); + } else { + pb_settings->set_node_limit(settings.node_limit); + } + pb_settings->set_reliability_branching(settings.reliability_branching); + pb_settings->set_mip_batch_pdlp_strong_branching(settings.mip_batch_pdlp_strong_branching); + pb_settings->set_max_cut_passes(settings.max_cut_passes); + pb_settings->set_mir_cuts(settings.mir_cuts); + pb_settings->set_mixed_integer_gomory_cuts(settings.mixed_integer_gomory_cuts); + pb_settings->set_knapsack_cuts(settings.knapsack_cuts); + pb_settings->set_clique_cuts(settings.clique_cuts); + pb_settings->set_strong_chvatal_gomory_cuts(settings.strong_chvatal_gomory_cuts); + pb_settings->set_reduced_cost_strengthening(settings.reduced_cost_strengthening); + pb_settings->set_cut_change_threshold(settings.cut_change_threshold); + pb_settings->set_cut_min_orthogonality(settings.cut_min_orthogonality); + pb_settings->set_determinism_mode(settings.determinism_mode); + pb_settings->set_seed(settings.seed); diff --git a/cpp/codegen/generated/generated_mip_solution_to_proto.inc b/cpp/codegen/generated/generated_mip_solution_to_proto.inc new file mode 100644 index 000000000..6fc58dd9d --- /dev/null +++ b/cpp/codegen/generated/generated_mip_solution_to_proto.inc @@ -0,0 +1,19 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + pb_solution->set_mip_termination_status(to_proto_mip_termination_status(solution.get_termination_status())); + pb_solution->set_mip_error_message(solution.get_error_status().what()); + pb_solution->set_mip_objective(solution.get_objective_value()); + pb_solution->set_mip_gap(solution.get_mip_gap()); + pb_solution->set_solution_bound(solution.get_solution_bound()); + pb_solution->set_total_solve_time(solution.get_solve_time()); + pb_solution->set_presolve_time(solution.get_presolve_time()); + pb_solution->set_max_constraint_violation(solution.get_max_constraint_violation()); + pb_solution->set_max_int_violation(solution.get_max_int_violation()); + pb_solution->set_max_variable_bound_violation(solution.get_max_variable_bound_violation()); + pb_solution->set_nodes(solution.get_num_nodes()); + pb_solution->set_simplex_iterations(solution.get_num_simplex_iterations()); + + const auto& _mip_solution = solution.get_solution_host(); + for (const auto& v : _mip_solution) pb_solution->add_mip_solution(static_cast(v)); diff --git a/cpp/codegen/generated/generated_pdlp_settings_to_proto.inc b/cpp/codegen/generated/generated_pdlp_settings_to_proto.inc new file mode 100644 index 000000000..4ef96095c --- /dev/null +++ b/cpp/codegen/generated/generated_pdlp_settings_to_proto.inc @@ -0,0 +1,38 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + pb_settings->set_absolute_gap_tolerance(settings.tolerances.absolute_gap_tolerance); + pb_settings->set_relative_gap_tolerance(settings.tolerances.relative_gap_tolerance); + pb_settings->set_primal_infeasible_tolerance(settings.tolerances.primal_infeasible_tolerance); + pb_settings->set_dual_infeasible_tolerance(settings.tolerances.dual_infeasible_tolerance); + pb_settings->set_absolute_dual_tolerance(settings.tolerances.absolute_dual_tolerance); + pb_settings->set_relative_dual_tolerance(settings.tolerances.relative_dual_tolerance); + pb_settings->set_absolute_primal_tolerance(settings.tolerances.absolute_primal_tolerance); + pb_settings->set_relative_primal_tolerance(settings.tolerances.relative_primal_tolerance); + pb_settings->set_time_limit(settings.time_limit); + if (settings.iteration_limit == std::numeric_limits::max()) { + pb_settings->set_iteration_limit(-1); + } else { + pb_settings->set_iteration_limit(static_cast(settings.iteration_limit)); + } + pb_settings->set_log_to_console(settings.log_to_console); + pb_settings->set_detect_infeasibility(settings.detect_infeasibility); + pb_settings->set_strict_infeasibility(settings.strict_infeasibility); + pb_settings->set_pdlp_solver_mode(to_proto_pdlp_solver_mode(settings.pdlp_solver_mode)); + pb_settings->set_method(to_proto_lp_method(settings.method)); + pb_settings->set_presolver(static_cast(settings.presolver)); + pb_settings->set_dual_postsolve(settings.dual_postsolve); + pb_settings->set_crossover(settings.crossover); + pb_settings->set_num_gpus(settings.num_gpus); + pb_settings->set_per_constraint_residual(settings.per_constraint_residual); + pb_settings->set_cudss_deterministic(settings.cudss_deterministic); + pb_settings->set_folding(settings.folding); + pb_settings->set_augmented(settings.augmented); + pb_settings->set_dualize(settings.dualize); + pb_settings->set_ordering(settings.ordering); + pb_settings->set_barrier_dual_initial_point(settings.barrier_dual_initial_point); + pb_settings->set_eliminate_dense_columns(settings.eliminate_dense_columns); + pb_settings->set_save_best_primal_so_far(settings.save_best_primal_so_far); + pb_settings->set_first_primal_feasible(settings.first_primal_feasible); + pb_settings->set_pdlp_precision(static_cast(settings.pdlp_precision)); diff --git a/cpp/codegen/generated/generated_populate_chunked_header_lp.inc b/cpp/codegen/generated/generated_populate_chunked_header_lp.inc new file mode 100644 index 000000000..a21b8a6e5 --- /dev/null +++ b/cpp/codegen/generated/generated_populate_chunked_header_lp.inc @@ -0,0 +1,16 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + auto* rh = header->mutable_header(); + rh->set_version(1); + rh->set_problem_category(cuopt::remote::LP); + + header->set_problem_name(cpu_problem.get_problem_name()); + header->set_objective_name(cpu_problem.get_objective_name()); + header->set_maximize(cpu_problem.get_sense()); + header->set_objective_scaling_factor(cpu_problem.get_objective_scaling_factor()); + header->set_objective_offset(cpu_problem.get_objective_offset()); + header->set_problem_category(to_proto_problem_category(cpu_problem.get_problem_category())); + + map_pdlp_settings_to_proto(settings, header->mutable_lp_settings()); diff --git a/cpp/codegen/generated/generated_populate_chunked_header_mip.inc b/cpp/codegen/generated/generated_populate_chunked_header_mip.inc new file mode 100644 index 000000000..c57a6f1b8 --- /dev/null +++ b/cpp/codegen/generated/generated_populate_chunked_header_mip.inc @@ -0,0 +1,17 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + auto* rh = header->mutable_header(); + rh->set_version(1); + rh->set_problem_category(cuopt::remote::MIP); + + header->set_problem_name(cpu_problem.get_problem_name()); + header->set_objective_name(cpu_problem.get_objective_name()); + header->set_maximize(cpu_problem.get_sense()); + header->set_objective_scaling_factor(cpu_problem.get_objective_scaling_factor()); + header->set_objective_offset(cpu_problem.get_objective_offset()); + header->set_problem_category(to_proto_problem_category(cpu_problem.get_problem_category())); + + map_mip_settings_to_proto(settings, header->mutable_mip_settings()); + header->set_enable_incumbents(enable_incumbents); diff --git a/cpp/codegen/generated/generated_problem_to_proto.inc b/cpp/codegen/generated/generated_problem_to_proto.inc new file mode 100644 index 000000000..70f2e880a --- /dev/null +++ b/cpp/codegen/generated/generated_problem_to_proto.inc @@ -0,0 +1,71 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + pb_problem->set_problem_name(cpu_problem.get_problem_name()); + pb_problem->set_objective_name(cpu_problem.get_objective_name()); + pb_problem->set_maximize(cpu_problem.get_sense()); + pb_problem->set_objective_scaling_factor(cpu_problem.get_objective_scaling_factor()); + pb_problem->set_objective_offset(cpu_problem.get_objective_offset()); + pb_problem->set_problem_category(to_proto_problem_category(cpu_problem.get_problem_category())); + + for (const auto& s : cpu_problem.get_variable_names()) pb_problem->add_variable_names(s); + for (const auto& s : cpu_problem.get_row_names()) pb_problem->add_row_names(s); + { + auto _c = cpu_problem.get_objective_coefficients_host(); + for (const auto& v : _c) pb_problem->add_c(static_cast(v)); + } + { + auto _b = cpu_problem.get_constraint_bounds_host(); + for (const auto& v : _b) pb_problem->add_b(static_cast(v)); + } + { + auto _variable_lower_bounds = cpu_problem.get_variable_lower_bounds_host(); + for (const auto& v : _variable_lower_bounds) pb_problem->add_variable_lower_bounds(static_cast(v)); + } + { + auto _variable_upper_bounds = cpu_problem.get_variable_upper_bounds_host(); + for (const auto& v : _variable_upper_bounds) pb_problem->add_variable_upper_bounds(static_cast(v)); + } + { + auto _constraint_lower_bounds = cpu_problem.get_constraint_lower_bounds_host(); + for (const auto& v : _constraint_lower_bounds) pb_problem->add_constraint_lower_bounds(static_cast(v)); + } + { + auto _constraint_upper_bounds = cpu_problem.get_constraint_upper_bounds_host(); + for (const auto& v : _constraint_upper_bounds) pb_problem->add_constraint_upper_bounds(static_cast(v)); + } + { + auto _row_types = cpu_problem.get_row_types_host(); + if (!_row_types.empty()) { + pb_problem->set_row_types(std::string(_row_types.begin(), _row_types.end())); + } + } + { + auto _variable_types = cpu_problem.get_variable_types_host(); + for (const auto& v : _variable_types) pb_problem->add_variable_types(to_proto_variable_type(v)); + } + { + auto _A_values = cpu_problem.get_constraint_matrix_values_host(); + for (const auto& v : _A_values) pb_problem->add_a_values(static_cast(v)); + } + { + auto _A_indices = cpu_problem.get_constraint_matrix_indices_host(); + for (const auto& v : _A_indices) pb_problem->add_a_indices(static_cast(v)); + } + { + auto _A_offsets = cpu_problem.get_constraint_matrix_offsets_host(); + for (const auto& v : _A_offsets) pb_problem->add_a_offsets(static_cast(v)); + } + { + auto _Q_values = cpu_problem.get_quadratic_objective_values_host(); + for (const auto& v : _Q_values) pb_problem->add_q_values(static_cast(v)); + } + { + auto _Q_indices = cpu_problem.get_quadratic_objective_indices_host(); + for (const auto& v : _Q_indices) pb_problem->add_q_indices(static_cast(v)); + } + { + auto _Q_offsets = cpu_problem.get_quadratic_objective_offsets_host(); + for (const auto& v : _Q_offsets) pb_problem->add_q_offsets(static_cast(v)); + } diff --git a/cpp/codegen/generated/generated_proto_to_lp_solution.inc b/cpp/codegen/generated/generated_proto_to_lp_solution.inc new file mode 100644 index 000000000..9af897400 --- /dev/null +++ b/cpp/codegen/generated/generated_proto_to_lp_solution.inc @@ -0,0 +1,42 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + std::vector primal_solution(pb_solution.primal_solution().begin(), pb_solution.primal_solution().end()); + std::vector dual_solution(pb_solution.dual_solution().begin(), pb_solution.dual_solution().end()); + std::vector reduced_cost(pb_solution.reduced_cost().begin(), pb_solution.reduced_cost().end()); + + auto _lp_termination_status = from_proto_pdlp_termination_status(pb_solution.lp_termination_status()); + auto _l2_primal_residual = static_cast(pb_solution.l2_primal_residual()); + auto _l2_dual_residual = static_cast(pb_solution.l2_dual_residual()); + auto _primal_objective = static_cast(pb_solution.primal_objective()); + auto _dual_objective = static_cast(pb_solution.dual_objective()); + auto _gap = static_cast(pb_solution.gap()); + auto _nb_iterations = static_cast(pb_solution.nb_iterations()); + auto _solve_time = static_cast(pb_solution.solve_time()); + auto _solved_by_pdlp = pb_solution.solved_by_pdlp(); + + if (pb_solution.has_warm_start_data()) { + const auto& pb_ws = pb_solution.warm_start_data(); + cpu_pdlp_warm_start_data_t ws; + ws.current_primal_solution_.assign(pb_ws.current_primal_solution_().begin(), pb_ws.current_primal_solution_().end()); + ws.current_dual_solution_.assign(pb_ws.current_dual_solution_().begin(), pb_ws.current_dual_solution_().end()); + ws.initial_primal_average_.assign(pb_ws.initial_primal_average_().begin(), pb_ws.initial_primal_average_().end()); + ws.initial_dual_average_.assign(pb_ws.initial_dual_average_().begin(), pb_ws.initial_dual_average_().end()); + ws.current_ATY_.assign(pb_ws.current_aty_().begin(), pb_ws.current_aty_().end()); + ws.sum_primal_solutions_.assign(pb_ws.sum_primal_solutions_().begin(), pb_ws.sum_primal_solutions_().end()); + ws.sum_dual_solutions_.assign(pb_ws.sum_dual_solutions_().begin(), pb_ws.sum_dual_solutions_().end()); + ws.last_restart_duality_gap_primal_solution_.assign(pb_ws.last_restart_duality_gap_primal_solution_().begin(), pb_ws.last_restart_duality_gap_primal_solution_().end()); + ws.last_restart_duality_gap_dual_solution_.assign(pb_ws.last_restart_duality_gap_dual_solution_().begin(), pb_ws.last_restart_duality_gap_dual_solution_().end()); + ws.initial_primal_weight_ = static_cast(pb_ws.initial_primal_weight_()); + ws.initial_step_size_ = static_cast(pb_ws.initial_step_size_()); + ws.total_pdlp_iterations_ = static_cast(pb_ws.total_pdlp_iterations_()); + ws.total_pdhg_iterations_ = static_cast(pb_ws.total_pdhg_iterations_()); + ws.last_candidate_kkt_score_ = static_cast(pb_ws.last_candidate_kkt_score_()); + ws.last_restart_kkt_score_ = static_cast(pb_ws.last_restart_kkt_score_()); + ws.sum_solution_weight_ = static_cast(pb_ws.sum_solution_weight_()); + ws.iterations_since_last_restart_ = static_cast(pb_ws.iterations_since_last_restart_()); + return cpu_lp_solution_t(std::move(primal_solution), std::move(dual_solution), std::move(reduced_cost), _lp_termination_status, _primal_objective, _dual_objective, _solve_time, _l2_primal_residual, _l2_dual_residual, _gap, _nb_iterations, _solved_by_pdlp, std::move(ws)); + } + + return cpu_lp_solution_t(std::move(primal_solution), std::move(dual_solution), std::move(reduced_cost), _lp_termination_status, _primal_objective, _dual_objective, _solve_time, _l2_primal_residual, _l2_dual_residual, _gap, _nb_iterations, _solved_by_pdlp); diff --git a/cpp/codegen/generated/generated_proto_to_mip_settings.inc b/cpp/codegen/generated/generated_proto_to_mip_settings.inc new file mode 100644 index 000000000..fbd448d9f --- /dev/null +++ b/cpp/codegen/generated/generated_proto_to_mip_settings.inc @@ -0,0 +1,34 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + settings.time_limit = pb_settings.time_limit(); + settings.tolerances.relative_mip_gap = pb_settings.relative_mip_gap(); + settings.tolerances.absolute_mip_gap = pb_settings.absolute_mip_gap(); + settings.tolerances.integrality_tolerance = pb_settings.integrality_tolerance(); + settings.tolerances.absolute_tolerance = pb_settings.absolute_tolerance(); + settings.tolerances.relative_tolerance = pb_settings.relative_tolerance(); + settings.tolerances.presolve_absolute_tolerance = pb_settings.presolve_absolute_tolerance(); + settings.log_to_console = pb_settings.log_to_console(); + settings.heuristics_only = pb_settings.heuristics_only(); + settings.num_cpu_threads = pb_settings.num_cpu_threads(); + settings.num_gpus = pb_settings.num_gpus(); + settings.presolver = static_cast(pb_settings.presolver()); + settings.mip_scaling = pb_settings.mip_scaling(); + settings.work_limit = pb_settings.work_limit(); + if (pb_settings.node_limit() >= 0) { + settings.node_limit = static_cast(pb_settings.node_limit()); + } + settings.reliability_branching = pb_settings.reliability_branching(); + settings.mip_batch_pdlp_strong_branching = pb_settings.mip_batch_pdlp_strong_branching(); + settings.max_cut_passes = pb_settings.max_cut_passes(); + settings.mir_cuts = pb_settings.mir_cuts(); + settings.mixed_integer_gomory_cuts = pb_settings.mixed_integer_gomory_cuts(); + settings.knapsack_cuts = pb_settings.knapsack_cuts(); + settings.clique_cuts = pb_settings.clique_cuts(); + settings.strong_chvatal_gomory_cuts = pb_settings.strong_chvatal_gomory_cuts(); + settings.reduced_cost_strengthening = pb_settings.reduced_cost_strengthening(); + settings.cut_change_threshold = pb_settings.cut_change_threshold(); + settings.cut_min_orthogonality = pb_settings.cut_min_orthogonality(); + settings.determinism_mode = pb_settings.determinism_mode(); + settings.seed = pb_settings.seed(); diff --git a/cpp/codegen/generated/generated_proto_to_mip_solution.inc b/cpp/codegen/generated/generated_proto_to_mip_solution.inc new file mode 100644 index 000000000..b2af4aa41 --- /dev/null +++ b/cpp/codegen/generated/generated_proto_to_mip_solution.inc @@ -0,0 +1,18 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + std::vector mip_solution(pb_solution.mip_solution().begin(), pb_solution.mip_solution().end()); + + auto _mip_termination_status = from_proto_mip_termination_status(pb_solution.mip_termination_status()); + auto _mip_objective = static_cast(pb_solution.mip_objective()); + auto _mip_gap = static_cast(pb_solution.mip_gap()); + auto _solution_bound = static_cast(pb_solution.solution_bound()); + auto _total_solve_time = static_cast(pb_solution.total_solve_time()); + auto _presolve_time = static_cast(pb_solution.presolve_time()); + auto _max_constraint_violation = static_cast(pb_solution.max_constraint_violation()); + auto _max_int_violation = static_cast(pb_solution.max_int_violation()); + auto _max_variable_bound_violation = static_cast(pb_solution.max_variable_bound_violation()); + auto _nodes = static_cast(pb_solution.nodes()); + auto _simplex_iterations = static_cast(pb_solution.simplex_iterations()); + return cpu_mip_solution_t(std::move(mip_solution), _mip_termination_status, _mip_objective, _mip_gap, _solution_bound, _total_solve_time, _presolve_time, _max_constraint_violation, _max_int_violation, _max_variable_bound_violation, _nodes, _simplex_iterations); diff --git a/cpp/codegen/generated/generated_proto_to_pdlp_settings.inc b/cpp/codegen/generated/generated_proto_to_pdlp_settings.inc new file mode 100644 index 000000000..1e92fdf1a --- /dev/null +++ b/cpp/codegen/generated/generated_proto_to_pdlp_settings.inc @@ -0,0 +1,36 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + settings.tolerances.absolute_gap_tolerance = pb_settings.absolute_gap_tolerance(); + settings.tolerances.relative_gap_tolerance = pb_settings.relative_gap_tolerance(); + settings.tolerances.primal_infeasible_tolerance = pb_settings.primal_infeasible_tolerance(); + settings.tolerances.dual_infeasible_tolerance = pb_settings.dual_infeasible_tolerance(); + settings.tolerances.absolute_dual_tolerance = pb_settings.absolute_dual_tolerance(); + settings.tolerances.relative_dual_tolerance = pb_settings.relative_dual_tolerance(); + settings.tolerances.absolute_primal_tolerance = pb_settings.absolute_primal_tolerance(); + settings.tolerances.relative_primal_tolerance = pb_settings.relative_primal_tolerance(); + settings.time_limit = pb_settings.time_limit(); + if (pb_settings.iteration_limit() >= 0) { + settings.iteration_limit = static_cast(pb_settings.iteration_limit()); + } + settings.log_to_console = pb_settings.log_to_console(); + settings.detect_infeasibility = pb_settings.detect_infeasibility(); + settings.strict_infeasibility = pb_settings.strict_infeasibility(); + settings.pdlp_solver_mode = from_proto_pdlp_solver_mode(pb_settings.pdlp_solver_mode()); + settings.method = from_proto_lp_method(pb_settings.method()); + settings.presolver = static_cast(pb_settings.presolver()); + settings.dual_postsolve = pb_settings.dual_postsolve(); + settings.crossover = pb_settings.crossover(); + settings.num_gpus = pb_settings.num_gpus(); + settings.per_constraint_residual = pb_settings.per_constraint_residual(); + settings.cudss_deterministic = pb_settings.cudss_deterministic(); + settings.folding = pb_settings.folding(); + settings.augmented = pb_settings.augmented(); + settings.dualize = pb_settings.dualize(); + settings.ordering = pb_settings.ordering(); + settings.barrier_dual_initial_point = pb_settings.barrier_dual_initial_point(); + settings.eliminate_dense_columns = pb_settings.eliminate_dense_columns(); + settings.save_best_primal_so_far = pb_settings.save_best_primal_so_far(); + settings.first_primal_feasible = pb_settings.first_primal_feasible(); + settings.pdlp_precision = static_cast(pb_settings.pdlp_precision()); diff --git a/cpp/codegen/generated/generated_proto_to_problem.inc b/cpp/codegen/generated/generated_proto_to_problem.inc new file mode 100644 index 000000000..a13dde453 --- /dev/null +++ b/cpp/codegen/generated/generated_proto_to_problem.inc @@ -0,0 +1,69 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ + cpu_problem.set_problem_name(pb_problem.problem_name()); + cpu_problem.set_objective_name(pb_problem.objective_name()); + cpu_problem.set_maximize(pb_problem.maximize()); + cpu_problem.set_objective_scaling_factor(pb_problem.objective_scaling_factor()); + cpu_problem.set_objective_offset(pb_problem.objective_offset()); + cpu_problem.set_problem_category(from_proto_problem_category(pb_problem.problem_category())); + + if (pb_problem.a_values_size() > 0) { + std::vector A_values(pb_problem.a_values().begin(), pb_problem.a_values().end()); + std::vector A_indices(pb_problem.a_indices().begin(), pb_problem.a_indices().end()); + std::vector A_offsets(pb_problem.a_offsets().begin(), pb_problem.a_offsets().end()); + cpu_problem.set_csr_constraint_matrix(A_values.data(), static_cast(A_values.size()), A_indices.data(), static_cast(A_indices.size()), A_offsets.data(), static_cast(A_offsets.size())); + } + + if (pb_problem.q_values_size() > 0) { + std::vector Q_values(pb_problem.q_values().begin(), pb_problem.q_values().end()); + std::vector Q_indices(pb_problem.q_indices().begin(), pb_problem.q_indices().end()); + std::vector Q_offsets(pb_problem.q_offsets().begin(), pb_problem.q_offsets().end()); + cpu_problem.set_quadratic_objective_matrix(Q_values.data(), static_cast(Q_values.size()), Q_indices.data(), static_cast(Q_indices.size()), Q_offsets.data(), static_cast(Q_offsets.size())); + } + + if (pb_problem.variable_names_size() > 0) { + std::vector variable_names(pb_problem.variable_names().begin(), pb_problem.variable_names().end()); + cpu_problem.set_variable_names(variable_names); + } + if (pb_problem.row_names_size() > 0) { + std::vector row_names(pb_problem.row_names().begin(), pb_problem.row_names().end()); + cpu_problem.set_row_names(row_names); + } + { + std::vector c(pb_problem.c().begin(), pb_problem.c().end()); + cpu_problem.set_objective_coefficients(c.data(), static_cast(c.size())); + } + if (pb_problem.b_size() > 0) { + std::vector b(pb_problem.b().begin(), pb_problem.b().end()); + cpu_problem.set_constraint_bounds(b.data(), static_cast(b.size())); + } + { + std::vector variable_lower_bounds(pb_problem.variable_lower_bounds().begin(), pb_problem.variable_lower_bounds().end()); + cpu_problem.set_variable_lower_bounds(variable_lower_bounds.data(), static_cast(variable_lower_bounds.size())); + } + { + std::vector variable_upper_bounds(pb_problem.variable_upper_bounds().begin(), pb_problem.variable_upper_bounds().end()); + cpu_problem.set_variable_upper_bounds(variable_upper_bounds.data(), static_cast(variable_upper_bounds.size())); + } + if (pb_problem.constraint_lower_bounds_size() > 0) { + std::vector constraint_lower_bounds(pb_problem.constraint_lower_bounds().begin(), pb_problem.constraint_lower_bounds().end()); + cpu_problem.set_constraint_lower_bounds(constraint_lower_bounds.data(), static_cast(constraint_lower_bounds.size())); + } + if (pb_problem.constraint_upper_bounds_size() > 0) { + std::vector constraint_upper_bounds(pb_problem.constraint_upper_bounds().begin(), pb_problem.constraint_upper_bounds().end()); + cpu_problem.set_constraint_upper_bounds(constraint_upper_bounds.data(), static_cast(constraint_upper_bounds.size())); + } + if (!pb_problem.row_types().empty()) { + const std::string& row_types_str = pb_problem.row_types(); + cpu_problem.set_row_types(row_types_str.data(), static_cast(row_types_str.size())); + } + if (pb_problem.variable_types_size() > 0) { + std::vector variable_types; + variable_types.reserve(pb_problem.variable_types_size()); + for (const auto& v : pb_problem.variable_types()) { + variable_types.push_back(from_proto_variable_type(static_cast(v))); + } + cpu_problem.set_variable_types(variable_types.data(), static_cast(variable_types.size())); + } diff --git a/cpp/codegen/generated/generated_result_enums.proto.inc b/cpp/codegen/generated/generated_result_enums.proto.inc new file mode 100644 index 000000000..a32d4a773 --- /dev/null +++ b/cpp/codegen/generated/generated_result_enums.proto.inc @@ -0,0 +1,18 @@ +// ============================================================================ +// AUTO-GENERATED by codegen/generate_conversions.py from field_registry.yaml +// DO NOT EDIT MANUALLY — regenerate with: python generate_conversions.py +// ============================================================================ +// ResultFieldId enum entries + RESULT_PRIMAL_SOLUTION = 0; + RESULT_DUAL_SOLUTION = 1; + RESULT_REDUCED_COST = 2; + RESULT_CURRENT_PRIMAL_SOLUTION_ = 3; + RESULT_CURRENT_DUAL_SOLUTION_ = 4; + RESULT_INITIAL_PRIMAL_AVERAGE_ = 5; + RESULT_INITIAL_DUAL_AVERAGE_ = 6; + RESULT_CURRENT_ATY_ = 7; + RESULT_SUM_PRIMAL_SOLUTIONS_ = 8; + RESULT_SUM_DUAL_SOLUTIONS_ = 9; + RESULT_LAST_RESTART_DUALITY_GAP_PRIMAL_SOLUTION_ = 10; + RESULT_LAST_RESTART_DUALITY_GAP_DUAL_SOLUTION_ = 11; + RESULT_MIP_SOLUTION = 12; diff --git a/cpp/include/cuopt/linear_programming/cpu_optimization_problem.hpp b/cpp/include/cuopt/linear_programming/cpu_optimization_problem.hpp index 009a8ce84..00f23748b 100644 --- a/cpp/include/cuopt/linear_programming/cpu_optimization_problem.hpp +++ b/cpp/include/cuopt/linear_programming/cpu_optimization_problem.hpp @@ -111,6 +111,18 @@ class cpu_optimization_problem_t : public optimization_problem_interface_t& get_quadratic_objective_offsets() const override; const std::vector& get_quadratic_objective_indices() const override; const std::vector& get_quadratic_objective_values() const override; + const std::vector& get_quadratic_objective_offsets_host() const + { + return get_quadratic_objective_offsets(); + } + const std::vector& get_quadratic_objective_indices_host() const + { + return get_quadratic_objective_indices(); + } + const std::vector& get_quadratic_objective_values_host() const + { + return get_quadratic_objective_values(); + } bool has_quadratic_objective() const override; // Host getters - these are the only supported getters for CPU implementation diff --git a/cpp/src/grpc/client/grpc_client.cpp b/cpp/src/grpc/client/grpc_client.cpp index 49839cd9b..59c6bfcb5 100644 --- a/cpp/src/grpc/client/grpc_client.cpp +++ b/cpp/src/grpc/client/grpc_client.cpp @@ -1034,7 +1034,7 @@ bool grpc_client_t::download_chunked_result(const std::string& job_id, GRPC_CLIENT_DEBUG_LOG(config_, "[grpc_client] ChunkedDownload started, download_id=" << download_id << " arrays=" << header->arrays_size() - << " is_mip=" << header->is_mip()); + << " problem_category=" << header->problem_category()); // --- 2. Fetch each array via GetResultChunk RPCs --- int64_t chunk_data_budget = config_.chunk_size_bytes; diff --git a/cpp/src/grpc/client/grpc_client.hpp b/cpp/src/grpc/client/grpc_client.hpp index f8579b327..58a40f5eb 100644 --- a/cpp/src/grpc/client/grpc_client.hpp +++ b/cpp/src/grpc/client/grpc_client.hpp @@ -52,7 +52,7 @@ void grpc_test_mark_as_connected(class grpc_client_t& client); * - Result retrieval uses chunked download for results exceeding max_message_bytes. */ struct grpc_client_config_t { - std::string server_address = "localhost:8765"; + std::string server_address = "localhost:5001"; int poll_interval_ms = 1000; // How often to poll for job status int timeout_seconds = 0; // Max time to wait for job completion (0 = no limit) bool stream_logs = false; // Whether to stream logs from server @@ -204,7 +204,7 @@ struct remote_mip_result_t { * * Usage: * @code - * grpc_client_t client("localhost:8765"); + * grpc_client_t client("localhost:5001"); * if (!client.connect()) { ... handle error ... } * * auto result = client.solve_lp(problem, settings); diff --git a/cpp/src/grpc/cuopt_remote.proto b/cpp/src/grpc/cuopt_remote.proto index d2617e0ef..c14298248 100644 --- a/cpp/src/grpc/cuopt_remote.proto +++ b/cpp/src/grpc/cuopt_remote.proto @@ -1,170 +1,21 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 syntax = "proto3"; package cuopt.remote; +// Data type definitions (enums, messages for problem/solution/settings) are +// auto-generated from field_registry.yaml. Using "import public" so that any +// file importing cuopt_remote.proto also sees the data types. +import public "cuopt_remote_data.proto"; + // Protocol version and metadata message RequestHeader { uint32 version = 1; // Protocol version (currently 1) ProblemCategory problem_category = 2; // LP or MIP } -enum ProblemCategory { - LP = 0; - MIP = 1; -} - -// Optimization problem representation (field names match cpu_optimization_problem_t) -message OptimizationProblem { - // Problem metadata - string problem_name = 1; - string objective_name = 2; - bool maximize = 3; - double objective_scaling_factor = 4; - double objective_offset = 5; - - // Variable and row names (optional) - repeated string variable_names = 6; - repeated string row_names = 7; - - // Constraint matrix A in CSR format - repeated double A = 8; - repeated int32 A_indices = 9; - repeated int32 A_offsets = 10; - - // Problem vectors - repeated double c = 11; // objective coefficients - repeated double b = 12; // constraint bounds (RHS) - repeated double variable_lower_bounds = 13; - repeated double variable_upper_bounds = 14; - - // Constraint bounds (alternative to b + row_types) - repeated double constraint_lower_bounds = 15; - repeated double constraint_upper_bounds = 16; - bytes row_types = 17; // char array: 'E' (=), 'L' (<=), 'G' (>=), 'N' (objective) - - // Variable types - bytes variable_types = 18; // char array: 'C' (continuous), 'I' (integer), 'B' (binary) - - // Initial solutions - repeated double initial_primal_solution = 19; - repeated double initial_dual_solution = 20; - - // Quadratic objective matrix Q in CSR format - repeated double Q_values = 21; - repeated int32 Q_indices = 22; - repeated int32 Q_offsets = 23; -} - -// PDLP solver mode enum (matches cuOpt pdlp_solver_mode_t) -// Matches cuOpt pdlp_solver_mode_t enum values -enum PDLPSolverMode { - Stable1 = 0; - Stable2 = 1; - Methodical1 = 2; - Fast1 = 3; - Stable3 = 4; -} - -// Matches cuOpt method_t enum values -enum LPMethod { - Concurrent = 0; - PDLP = 1; - DualSimplex = 2; - Barrier = 3; -} - -// PDLP solver settings (field names match cuOpt Python/C++ API) -message PDLPSolverSettings { - // Termination tolerances - double absolute_gap_tolerance = 1; - double relative_gap_tolerance = 2; - double primal_infeasible_tolerance = 3; - double dual_infeasible_tolerance = 4; - double absolute_dual_tolerance = 5; - double relative_dual_tolerance = 6; - double absolute_primal_tolerance = 7; - double relative_primal_tolerance = 8; - - // Limits - double time_limit = 10; - // Iteration limit. Sentinel: set to -1 to mean "unset/use server defaults". - // Note: proto3 numeric fields default to 0 when omitted, so clients should - // explicitly use -1 (or a positive value) to avoid accidentally requesting 0 iterations. - int64 iteration_limit = 11; - - // Solver configuration - bool log_to_console = 20; - bool detect_infeasibility = 21; - bool strict_infeasibility = 22; - PDLPSolverMode pdlp_solver_mode = 23; - LPMethod method = 24; - int32 presolver = 25; - bool dual_postsolve = 26; - bool crossover = 27; - int32 num_gpus = 28; - - bool per_constraint_residual = 30; - bool cudss_deterministic = 31; - int32 folding = 32; - int32 augmented = 33; - int32 dualize = 34; - int32 ordering = 35; - int32 barrier_dual_initial_point = 36; - bool eliminate_dense_columns = 37; - bool save_best_primal_so_far = 38; - bool first_primal_feasible = 39; - int32 pdlp_precision = 40; - - // Warm start data (if provided) - PDLPWarmStartData warm_start_data = 50; -} - -message PDLPWarmStartData { - repeated double current_primal_solution = 1; - repeated double current_dual_solution = 2; - repeated double initial_primal_average = 3; - repeated double initial_dual_average = 4; - repeated double current_ATY = 5; - repeated double sum_primal_solutions = 6; - repeated double sum_dual_solutions = 7; - repeated double last_restart_duality_gap_primal_solution = 8; - repeated double last_restart_duality_gap_dual_solution = 9; - - double initial_primal_weight = 10; - double initial_step_size = 11; - int32 total_pdlp_iterations = 12; - int32 total_pdhg_iterations = 13; - double last_candidate_kkt_score = 14; - double last_restart_kkt_score = 15; - double sum_solution_weight = 16; - int32 iterations_since_last_restart = 17; -} - -// MIP solver settings (field names match cuOpt Python/C++ API) -message MIPSolverSettings { - // Limits - double time_limit = 1; - - // Tolerances - double relative_mip_gap = 2; - double absolute_mip_gap = 3; - double integrality_tolerance = 4; - double absolute_tolerance = 5; - double relative_tolerance = 6; - double presolve_absolute_tolerance = 7; - - // Solver configuration - bool log_to_console = 10; - bool heuristics_only = 11; - int32 num_cpu_threads = 12; - int32 num_gpus = 13; - int32 presolver = 14; - bool mip_scaling = 15; -} - // LP solve request message SolveLPRequest { RequestHeader header = 1; @@ -180,115 +31,6 @@ message SolveMIPRequest { optional bool enable_incumbents = 4; } -// LP solution -message LPSolution { - // Solution vectors - repeated double primal_solution = 1; - repeated double dual_solution = 2; - repeated double reduced_cost = 3; - - // Warm start data for next solve - PDLPWarmStartData warm_start_data = 4; - - // Termination information - PDLPTerminationStatus termination_status = 10; - string error_message = 11; - - // Solution statistics - double l2_primal_residual = 20; - double l2_dual_residual = 21; - double primal_objective = 22; - double dual_objective = 23; - double gap = 24; - int32 nb_iterations = 25; - double solve_time = 26; - bool solved_by_pdlp = 27; -} - -enum PDLPTerminationStatus { - PDLP_NO_TERMINATION = 0; - PDLP_NUMERICAL_ERROR = 1; - PDLP_OPTIMAL = 2; - PDLP_PRIMAL_INFEASIBLE = 3; - PDLP_DUAL_INFEASIBLE = 4; - PDLP_ITERATION_LIMIT = 5; - PDLP_TIME_LIMIT = 6; - PDLP_CONCURRENT_LIMIT = 7; - PDLP_PRIMAL_FEASIBLE = 8; -} - -// MIP solution -message MIPSolution { - repeated double solution = 1; - - MIPTerminationStatus termination_status = 10; - string error_message = 11; - - double objective = 20; - double mip_gap = 21; - double solution_bound = 22; - double total_solve_time = 23; - double presolve_time = 24; - double max_constraint_violation = 25; - double max_int_violation = 26; - double max_variable_bound_violation = 27; - int32 nodes = 28; - int32 simplex_iterations = 29; -} - -enum MIPTerminationStatus { - MIP_NO_TERMINATION = 0; - MIP_OPTIMAL = 1; - MIP_FEASIBLE_FOUND = 2; - MIP_INFEASIBLE = 3; - MIP_UNBOUNDED = 4; - MIP_TIME_LIMIT = 5; - MIP_WORK_LIMIT = 6; -} - -// Array field identifiers for chunked array transfers -// Used to identify which problem array a chunk belongs to -enum ArrayFieldId { - FIELD_A_VALUES = 0; - FIELD_A_INDICES = 1; - FIELD_A_OFFSETS = 2; - FIELD_C = 3; - FIELD_B = 4; - FIELD_VARIABLE_LOWER_BOUNDS = 5; - FIELD_VARIABLE_UPPER_BOUNDS = 6; - FIELD_CONSTRAINT_LOWER_BOUNDS = 7; - FIELD_CONSTRAINT_UPPER_BOUNDS = 8; - FIELD_ROW_TYPES = 9; - FIELD_VARIABLE_TYPES = 10; - FIELD_Q_VALUES = 11; - FIELD_Q_INDICES = 12; - FIELD_Q_OFFSETS = 13; - FIELD_INITIAL_PRIMAL = 14; - FIELD_INITIAL_DUAL = 15; - // String arrays (null-separated bytes, sent as chunks alongside numeric data) - FIELD_VARIABLE_NAMES = 20; - FIELD_ROW_NAMES = 21; -} - -// Result array field identifiers for chunked result downloads -// Used to identify which result array a chunk belongs to -enum ResultFieldId { - RESULT_PRIMAL_SOLUTION = 0; - RESULT_DUAL_SOLUTION = 1; - RESULT_REDUCED_COST = 2; - RESULT_MIP_SOLUTION = 3; - // Warm start arrays (LP only) - RESULT_WS_CURRENT_PRIMAL = 10; - RESULT_WS_CURRENT_DUAL = 11; - RESULT_WS_INITIAL_PRIMAL_AVG = 12; - RESULT_WS_INITIAL_DUAL_AVG = 13; - RESULT_WS_CURRENT_ATY = 14; - RESULT_WS_SUM_PRIMAL = 15; - RESULT_WS_SUM_DUAL = 16; - RESULT_WS_LAST_RESTART_GAP_PRIMAL = 17; - RESULT_WS_LAST_RESTART_GAP_DUAL = 18; -} - // Job status for async operations enum JobStatus { QUEUED = 0; // Job submitted, waiting in queue diff --git a/cpp/src/grpc/cuopt_remote_service.proto b/cpp/src/grpc/cuopt_remote_service.proto index 86777baba..0212ff385 100644 --- a/cpp/src/grpc/cuopt_remote_service.proto +++ b/cpp/src/grpc/cuopt_remote_service.proto @@ -120,6 +120,7 @@ message ChunkedProblemHeader { double objective_offset = 4; string problem_name = 5; string objective_name = 6; + ProblemCategory problem_category = 9; // String arrays (included here since they are rarely the size bottleneck) repeated string variable_names = 7; @@ -183,57 +184,8 @@ message GetResultRequest { string job_id = 1; } -// Metadata about a single result array available for chunked download -message ResultArrayDescriptor { - ResultFieldId field_id = 1; - int64 total_elements = 2; - int64 element_size_bytes = 3; // 8 for double, 4 for int32, etc. -} - -// Header for chunked result download - carries all scalar/enum/string fields -// from LPSolution or MIPSolution. Array data is sent via GetResultChunk. -message ChunkedResultHeader { - bool is_mip = 1; - - // LP result scalars - PDLPTerminationStatus lp_termination_status = 10; - string error_message = 11; - double l2_primal_residual = 12; - double l2_dual_residual = 13; - double primal_objective = 14; - double dual_objective = 15; - double gap = 16; - int32 nb_iterations = 17; - double solve_time = 18; - bool solved_by_pdlp = 19; - - // MIP result scalars - MIPTerminationStatus mip_termination_status = 30; - string mip_error_message = 31; - double mip_objective = 32; - double mip_gap = 33; - double solution_bound = 34; - double total_solve_time = 35; - double presolve_time = 36; - double max_constraint_violation = 37; - double max_int_violation = 38; - double max_variable_bound_violation = 39; - int32 nodes = 40; - int32 simplex_iterations = 41; - - // LP warm start scalars (included in header since they are small) - double ws_initial_primal_weight = 60; - double ws_initial_step_size = 61; - int32 ws_total_pdlp_iterations = 62; - int32 ws_total_pdhg_iterations = 63; - double ws_last_candidate_kkt_score = 64; - double ws_last_restart_kkt_score = 65; - double ws_sum_solution_weight = 66; - int32 ws_iterations_since_last_restart = 67; - - // Array metadata so client knows what to fetch - repeated ResultArrayDescriptor arrays = 50; -} +// ResultArrayDescriptor and ChunkedResultHeader are defined in +// cuopt_remote_data.proto (auto-generated from field_registry.yaml). message StartChunkedDownloadRequest { string job_id = 1; diff --git a/cpp/src/grpc/grpc_problem_mapper.cpp b/cpp/src/grpc/grpc_problem_mapper.cpp index 7a7bdde64..2cdf7890d 100644 --- a/cpp/src/grpc/grpc_problem_mapper.cpp +++ b/cpp/src/grpc/grpc_problem_mapper.cpp @@ -10,7 +10,6 @@ #include #include #include -#include #include #include "grpc_settings_mapper.hpp" @@ -22,242 +21,81 @@ namespace cuopt::linear_programming { -template -void map_problem_to_proto(const cpu_optimization_problem_t& cpu_problem, - cuopt::remote::OptimizationProblem* pb_problem) -{ - // Basic problem metadata - pb_problem->set_problem_name(cpu_problem.get_problem_name()); - pb_problem->set_objective_name(cpu_problem.get_objective_name()); - pb_problem->set_maximize(cpu_problem.get_sense()); - pb_problem->set_objective_scaling_factor(cpu_problem.get_objective_scaling_factor()); - pb_problem->set_objective_offset(cpu_problem.get_objective_offset()); - - // Get constraint matrix data from host memory - auto values = cpu_problem.get_constraint_matrix_values_host(); - auto indices = cpu_problem.get_constraint_matrix_indices_host(); - auto offsets = cpu_problem.get_constraint_matrix_offsets_host(); - - // Constraint matrix A in CSR format - for (const auto& val : values) { - pb_problem->add_a(static_cast(val)); - } - for (const auto& idx : indices) { - pb_problem->add_a_indices(static_cast(idx)); - } - for (const auto& off : offsets) { - pb_problem->add_a_offsets(static_cast(off)); - } - - // Objective coefficients - auto obj_coeffs = cpu_problem.get_objective_coefficients_host(); - for (const auto& c : obj_coeffs) { - pb_problem->add_c(static_cast(c)); - } +namespace { +#include "generated_enum_converters_problem.inc" - // Variable bounds - auto var_lb = cpu_problem.get_variable_lower_bounds_host(); - auto var_ub = cpu_problem.get_variable_upper_bounds_host(); - for (const auto& lb : var_lb) { - pb_problem->add_variable_lower_bounds(static_cast(lb)); - } - for (const auto& ub : var_ub) { - pb_problem->add_variable_upper_bounds(static_cast(ub)); - } +template +void chunk_typed_array(std::vector& out, + cuopt::remote::ArrayFieldId field_id, + const std::vector& data, + const std::string& upload_id, + int64_t chunk_data_budget) +{ + if (data.empty()) return; - // Constraint bounds - auto con_lb = cpu_problem.get_constraint_lower_bounds_host(); - auto con_ub = cpu_problem.get_constraint_upper_bounds_host(); + const int64_t elem_size = static_cast(sizeof(T)); + const int64_t total_elements = static_cast(data.size()); - if (!con_lb.empty() && !con_ub.empty()) { - for (const auto& lb : con_lb) { - pb_problem->add_constraint_lower_bounds(static_cast(lb)); - } - for (const auto& ub : con_ub) { - pb_problem->add_constraint_upper_bounds(static_cast(ub)); - } - } + int64_t elems_per_chunk = chunk_data_budget / elem_size; + if (elems_per_chunk <= 0) elems_per_chunk = 1; - // Row types (if available) - auto row_types = cpu_problem.get_row_types_host(); - if (!row_types.empty()) { - pb_problem->set_row_types(std::string(row_types.begin(), row_types.end())); - } + const auto* raw = reinterpret_cast(data.data()); - // Constraint bounds (RHS) - if available - auto b = cpu_problem.get_constraint_bounds_host(); - if (!b.empty()) { - for (const auto& rhs : b) { - pb_problem->add_b(static_cast(rhs)); - } - } + for (int64_t offset = 0; offset < total_elements; offset += elems_per_chunk) { + int64_t count = std::min(elems_per_chunk, total_elements - offset); + int64_t byte_offset = offset * elem_size; + int64_t byte_count = count * elem_size; - // Variable names - const auto& var_names = cpu_problem.get_variable_names(); - for (const auto& name : var_names) { - pb_problem->add_variable_names(name); + cuopt::remote::SendArrayChunkRequest req; + req.set_upload_id(upload_id); + auto* ac = req.mutable_chunk(); + ac->set_field_id(field_id); + ac->set_element_offset(offset); + ac->set_total_elements(total_elements); + ac->set_data(raw + byte_offset, byte_count); + out.push_back(std::move(req)); } +} - // Row names - const auto& row_names = cpu_problem.get_row_names(); - for (const auto& name : row_names) { - pb_problem->add_row_names(name); - } +void chunk_byte_blob(std::vector& out, + cuopt::remote::ArrayFieldId field_id, + const std::vector& data, + const std::string& upload_id, + int64_t chunk_data_budget) +{ + chunk_typed_array(out, field_id, data, upload_id, chunk_data_budget); +} - // Variable types (for MIP problems) - auto var_types = cpu_problem.get_variable_types_host(); - if (!var_types.empty()) { - // Convert var_t enum to char representation - std::string var_types_str; - var_types_str.reserve(var_types.size()); - for (const auto& vt : var_types) { - switch (vt) { - case var_t::CONTINUOUS: var_types_str.push_back('C'); break; - case var_t::INTEGER: var_types_str.push_back('I'); break; - default: - throw std::runtime_error("map_problem_to_proto: unknown var_t value " + - std::to_string(static_cast(vt))); - } - } - pb_problem->set_variable_types(var_types_str); - } +std::vector names_to_blob(const std::vector& names) +{ + if (names.empty()) return {}; + size_t total = 0; + for (const auto& n : names) + total += n.size() + 1; + std::vector blob(total); + size_t pos = 0; + for (const auto& n : names) { + std::memcpy(blob.data() + pos, n.data(), n.size()); + pos += n.size(); + blob[pos++] = '\0'; + } + return blob; +} - // Quadratic objective matrix Q (for QPS problems) - if (cpu_problem.has_quadratic_objective()) { - const auto& q_values = cpu_problem.get_quadratic_objective_values(); - const auto& q_indices = cpu_problem.get_quadratic_objective_indices(); - const auto& q_offsets = cpu_problem.get_quadratic_objective_offsets(); +} // namespace - for (const auto& val : q_values) { - pb_problem->add_q_values(static_cast(val)); - } - for (const auto& idx : q_indices) { - pb_problem->add_q_indices(static_cast(idx)); - } - for (const auto& off : q_offsets) { - pb_problem->add_q_offsets(static_cast(off)); - } - } +template +void map_problem_to_proto(const cpu_optimization_problem_t& cpu_problem, + cuopt::remote::OptimizationProblem* pb_problem) +{ +#include "generated_problem_to_proto.inc" } template void map_proto_to_problem(const cuopt::remote::OptimizationProblem& pb_problem, cpu_optimization_problem_t& cpu_problem) { - // Basic problem metadata - cpu_problem.set_problem_name(pb_problem.problem_name()); - cpu_problem.set_objective_name(pb_problem.objective_name()); - cpu_problem.set_maximize(pb_problem.maximize()); - cpu_problem.set_objective_scaling_factor(pb_problem.objective_scaling_factor()); - cpu_problem.set_objective_offset(pb_problem.objective_offset()); - - // Constraint matrix A in CSR format - std::vector values(pb_problem.a().begin(), pb_problem.a().end()); - std::vector indices(pb_problem.a_indices().begin(), pb_problem.a_indices().end()); - std::vector offsets(pb_problem.a_offsets().begin(), pb_problem.a_offsets().end()); - - cpu_problem.set_csr_constraint_matrix(values.data(), - static_cast(values.size()), - indices.data(), - static_cast(indices.size()), - offsets.data(), - static_cast(offsets.size())); - - // Objective coefficients - std::vector obj(pb_problem.c().begin(), pb_problem.c().end()); - cpu_problem.set_objective_coefficients(obj.data(), static_cast(obj.size())); - - // Variable bounds - std::vector var_lb(pb_problem.variable_lower_bounds().begin(), - pb_problem.variable_lower_bounds().end()); - std::vector var_ub(pb_problem.variable_upper_bounds().begin(), - pb_problem.variable_upper_bounds().end()); - cpu_problem.set_variable_lower_bounds(var_lb.data(), static_cast(var_lb.size())); - cpu_problem.set_variable_upper_bounds(var_ub.data(), static_cast(var_ub.size())); - - // Constraint bounds (prefer lower/upper bounds if available) - if (pb_problem.constraint_lower_bounds_size() > 0 && - pb_problem.constraint_upper_bounds_size() > 0 && - pb_problem.constraint_lower_bounds_size() == pb_problem.constraint_upper_bounds_size()) { - std::vector con_lb(pb_problem.constraint_lower_bounds().begin(), - pb_problem.constraint_lower_bounds().end()); - std::vector con_ub(pb_problem.constraint_upper_bounds().begin(), - pb_problem.constraint_upper_bounds().end()); - cpu_problem.set_constraint_lower_bounds(con_lb.data(), static_cast(con_lb.size())); - cpu_problem.set_constraint_upper_bounds(con_ub.data(), static_cast(con_ub.size())); - } else if (pb_problem.b_size() > 0) { - // Use b (RHS) + row_types format - std::vector b(pb_problem.b().begin(), pb_problem.b().end()); - cpu_problem.set_constraint_bounds(b.data(), static_cast(b.size())); - - if (!pb_problem.row_types().empty()) { - const std::string& row_types_str = pb_problem.row_types(); - cpu_problem.set_row_types(row_types_str.data(), static_cast(row_types_str.size())); - } - } - - // Variable names - if (pb_problem.variable_names_size() > 0) { - std::vector var_names(pb_problem.variable_names().begin(), - pb_problem.variable_names().end()); - cpu_problem.set_variable_names(var_names); - } - - // Row names - if (pb_problem.row_names_size() > 0) { - std::vector row_names(pb_problem.row_names().begin(), - pb_problem.row_names().end()); - cpu_problem.set_row_names(row_names); - } - - // Variable types - if (!pb_problem.variable_types().empty()) { - const std::string& var_types_str = pb_problem.variable_types(); - // Convert char representation to var_t enum - std::vector var_types; - var_types.reserve(var_types_str.size()); - for (char c : var_types_str) { - switch (c) { - case 'C': var_types.push_back(var_t::CONTINUOUS); break; - case 'I': - case 'B': var_types.push_back(var_t::INTEGER); break; - default: - throw std::runtime_error(std::string("Unknown variable type character '") + c + - "' in variable_types string (expected 'C', 'I', or 'B')"); - } - } - cpu_problem.set_variable_types(var_types.data(), static_cast(var_types.size())); - } - - // Quadratic objective matrix Q (for QPS problems) - if (pb_problem.q_values_size() > 0) { - std::vector q_values(pb_problem.q_values().begin(), pb_problem.q_values().end()); - std::vector q_indices(pb_problem.q_indices().begin(), pb_problem.q_indices().end()); - std::vector q_offsets(pb_problem.q_offsets().begin(), pb_problem.q_offsets().end()); - - cpu_problem.set_quadratic_objective_matrix(q_values.data(), - static_cast(q_values.size()), - q_indices.data(), - static_cast(q_indices.size()), - q_offsets.data(), - static_cast(q_offsets.size())); - } - - // Infer problem category from variable types - if (!pb_problem.variable_types().empty()) { - const std::string& var_types_str = pb_problem.variable_types(); - bool has_integers = false; - for (char c : var_types_str) { - if (c == 'I' || c == 'B') { - has_integers = true; - break; - } - } - cpu_problem.set_problem_category(has_integers ? problem_category_t::MIP - : problem_category_t::LP); - } else { - cpu_problem.set_problem_category(problem_category_t::LP); - } +#include "generated_proto_to_problem.inc" } // ============================================================================ @@ -267,51 +105,7 @@ void map_proto_to_problem(const cuopt::remote::OptimizationProblem& pb_problem, template size_t estimate_problem_proto_size(const cpu_optimization_problem_t& cpu_problem) { - size_t est = 0; - - // Constraint matrix CSR arrays - auto values = cpu_problem.get_constraint_matrix_values_host(); - auto indices = cpu_problem.get_constraint_matrix_indices_host(); - auto offsets = cpu_problem.get_constraint_matrix_offsets_host(); - est += values.size() * sizeof(double); // packed repeated double - est += indices.size() * 5; // varint int32 (worst case 5 bytes each) - est += offsets.size() * 5; - - // Objective coefficients - est += cpu_problem.get_objective_coefficients_host().size() * sizeof(double); - - // Variable bounds - est += cpu_problem.get_variable_lower_bounds_host().size() * sizeof(double); - est += cpu_problem.get_variable_upper_bounds_host().size() * sizeof(double); - - // Constraint bounds - est += cpu_problem.get_constraint_lower_bounds_host().size() * sizeof(double); - est += cpu_problem.get_constraint_upper_bounds_host().size() * sizeof(double); - est += cpu_problem.get_constraint_bounds_host().size() * sizeof(double); - - // Row types and variable types - est += cpu_problem.get_row_types_host().size(); - est += cpu_problem.get_variable_types_host().size(); - - // Quadratic objective - if (cpu_problem.has_quadratic_objective()) { - est += cpu_problem.get_quadratic_objective_values().size() * sizeof(double); - est += cpu_problem.get_quadratic_objective_indices().size() * 5; - est += cpu_problem.get_quadratic_objective_offsets().size() * 5; - } - - // String arrays (rough estimate) - for (const auto& name : cpu_problem.get_variable_names()) { - est += name.size() + 2; // string + tag + length varint - } - for (const auto& name : cpu_problem.get_row_names()) { - est += name.size() + 2; - } - - // Protobuf overhead for tags, submessage lengths, etc. - est += 512; - - return est; +#include "generated_estimate_problem_size.inc" } // ============================================================================ @@ -323,22 +117,7 @@ void populate_chunked_header_lp(const cpu_optimization_problem_t& cpu_ const pdlp_solver_settings_t& settings, cuopt::remote::ChunkedProblemHeader* header) { - // Request header - auto* rh = header->mutable_header(); - rh->set_version(1); - rh->set_problem_category(cuopt::remote::LP); - - header->set_maximize(cpu_problem.get_sense()); - header->set_objective_scaling_factor(cpu_problem.get_objective_scaling_factor()); - header->set_objective_offset(cpu_problem.get_objective_offset()); - header->set_problem_name(cpu_problem.get_problem_name()); - header->set_objective_name(cpu_problem.get_objective_name()); - - // Variable/row names are sent as chunked arrays, not in the header, - // to avoid the header exceeding gRPC max message size for large problems. - - // LP settings - map_pdlp_settings_to_proto(settings, header->mutable_lp_settings()); +#include "generated_populate_chunked_header_lp.inc" } template @@ -347,22 +126,7 @@ void populate_chunked_header_mip(const cpu_optimization_problem_t& cpu bool enable_incumbents, cuopt::remote::ChunkedProblemHeader* header) { - // Request header - auto* rh = header->mutable_header(); - rh->set_version(1); - rh->set_problem_category(cuopt::remote::MIP); - - header->set_maximize(cpu_problem.get_sense()); - header->set_objective_scaling_factor(cpu_problem.get_objective_scaling_factor()); - header->set_objective_offset(cpu_problem.get_objective_offset()); - header->set_problem_name(cpu_problem.get_problem_name()); - header->set_objective_name(cpu_problem.get_objective_name()); - - // Variable/row names are sent as chunked arrays, not in the header. - - // MIP settings - map_mip_settings_to_proto(settings, header->mutable_mip_settings()); - header->set_enable_incumbents(enable_incumbents); +#include "generated_populate_chunked_header_mip.inc" } // ============================================================================ @@ -373,24 +137,7 @@ template void map_chunked_header_to_problem(const cuopt::remote::ChunkedProblemHeader& header, cpu_optimization_problem_t& cpu_problem) { - cpu_problem.set_problem_name(header.problem_name()); - cpu_problem.set_objective_name(header.objective_name()); - cpu_problem.set_maximize(header.maximize()); - cpu_problem.set_objective_scaling_factor(header.objective_scaling_factor()); - cpu_problem.set_objective_offset(header.objective_offset()); - - // String arrays - if (header.variable_names_size() > 0) { - std::vector var_names(header.variable_names().begin(), - header.variable_names().end()); - cpu_problem.set_variable_names(var_names); - } - if (header.row_names_size() > 0) { - std::vector row_names(header.row_names().begin(), header.row_names().end()); - cpu_problem.set_row_names(row_names); - } - - // Problem category inferred later when variable_types array is set +#include "generated_chunked_header_to_problem.inc" } // ============================================================================ @@ -402,298 +149,20 @@ void map_chunked_arrays_to_problem(const cuopt::remote::ChunkedProblemHeader& he const std::map>& arrays, cpu_optimization_problem_t& cpu_problem) { - map_chunked_header_to_problem(header, cpu_problem); - - auto get_doubles = [&](int32_t field_id) -> std::vector { - auto it = arrays.find(field_id); - if (it == arrays.end() || it->second.empty()) return {}; - if (it->second.size() % sizeof(double) != 0) return {}; - size_t n = it->second.size() / sizeof(double); - if constexpr (std::is_same_v) { - std::vector v(n); - std::memcpy(v.data(), it->second.data(), n * sizeof(double)); - return v; - } else { - std::vector tmp(n); - std::memcpy(tmp.data(), it->second.data(), n * sizeof(double)); - return std::vector(tmp.begin(), tmp.end()); - } - }; - - auto get_ints = [&](int32_t field_id) -> std::vector { - auto it = arrays.find(field_id); - if (it == arrays.end() || it->second.empty()) return {}; - if (it->second.size() % sizeof(int32_t) != 0) return {}; - size_t n = it->second.size() / sizeof(int32_t); - if constexpr (std::is_same_v) { - std::vector v(n); - std::memcpy(v.data(), it->second.data(), n * sizeof(int32_t)); - return v; - } else { - std::vector tmp(n); - std::memcpy(tmp.data(), it->second.data(), n * sizeof(int32_t)); - return std::vector(tmp.begin(), tmp.end()); - } - }; - - auto get_bytes = [&](int32_t field_id) -> std::string { - auto it = arrays.find(field_id); - if (it == arrays.end() || it->second.empty()) return {}; - return std::string(reinterpret_cast(it->second.data()), it->second.size()); - }; - - auto get_string_list = [&](int32_t field_id) -> std::vector { - auto it = arrays.find(field_id); - if (it == arrays.end() || it->second.empty()) return {}; - std::vector names; - const char* s = reinterpret_cast(it->second.data()); - const char* s_end = s + it->second.size(); - while (s < s_end) { - const char* nul = static_cast(std::memchr(s, '\0', s_end - s)); - if (!nul) nul = s_end; - names.emplace_back(s, nul); - if (nul == s_end) break; - s = nul + 1; - } - return names; - }; - - // CSR constraint matrix - auto a_values = get_doubles(cuopt::remote::FIELD_A_VALUES); - auto a_indices = get_ints(cuopt::remote::FIELD_A_INDICES); - auto a_offsets = get_ints(cuopt::remote::FIELD_A_OFFSETS); - if (!a_values.empty() && !a_indices.empty() && !a_offsets.empty()) { - cpu_problem.set_csr_constraint_matrix(a_values.data(), - static_cast(a_values.size()), - a_indices.data(), - static_cast(a_indices.size()), - a_offsets.data(), - static_cast(a_offsets.size())); - } - - // Objective coefficients - auto c_vec = get_doubles(cuopt::remote::FIELD_C); - if (!c_vec.empty()) { - cpu_problem.set_objective_coefficients(c_vec.data(), static_cast(c_vec.size())); - } - - // Variable bounds - auto var_lb = get_doubles(cuopt::remote::FIELD_VARIABLE_LOWER_BOUNDS); - auto var_ub = get_doubles(cuopt::remote::FIELD_VARIABLE_UPPER_BOUNDS); - if (!var_lb.empty()) { - cpu_problem.set_variable_lower_bounds(var_lb.data(), static_cast(var_lb.size())); - } - if (!var_ub.empty()) { - cpu_problem.set_variable_upper_bounds(var_ub.data(), static_cast(var_ub.size())); - } - - // Constraint bounds - auto con_lb = get_doubles(cuopt::remote::FIELD_CONSTRAINT_LOWER_BOUNDS); - auto con_ub = get_doubles(cuopt::remote::FIELD_CONSTRAINT_UPPER_BOUNDS); - if (!con_lb.empty()) { - cpu_problem.set_constraint_lower_bounds(con_lb.data(), static_cast(con_lb.size())); - } - if (!con_ub.empty()) { - cpu_problem.set_constraint_upper_bounds(con_ub.data(), static_cast(con_ub.size())); - } - - auto b_vec = get_doubles(cuopt::remote::FIELD_B); - if (!b_vec.empty()) { - cpu_problem.set_constraint_bounds(b_vec.data(), static_cast(b_vec.size())); - } - - // Row types - auto row_types_str = get_bytes(cuopt::remote::FIELD_ROW_TYPES); - if (!row_types_str.empty()) { - cpu_problem.set_row_types(row_types_str.data(), static_cast(row_types_str.size())); - } - - // Variable types + problem category - auto var_types_str = get_bytes(cuopt::remote::FIELD_VARIABLE_TYPES); - if (!var_types_str.empty()) { - std::vector vtypes; - vtypes.reserve(var_types_str.size()); - bool has_ints = false; - for (char c : var_types_str) { - switch (c) { - case 'C': vtypes.push_back(var_t::CONTINUOUS); break; - case 'I': - case 'B': - vtypes.push_back(var_t::INTEGER); - has_ints = true; - break; - default: - throw std::runtime_error(std::string("Unknown variable type character '") + c + - "' in chunked variable_types (expected 'C', 'I', or 'B')"); - } - } - cpu_problem.set_variable_types(vtypes.data(), static_cast(vtypes.size())); - cpu_problem.set_problem_category(has_ints ? problem_category_t::MIP : problem_category_t::LP); - } else { - cpu_problem.set_problem_category(problem_category_t::LP); - } - - // Quadratic objective - auto q_values = get_doubles(cuopt::remote::FIELD_Q_VALUES); - auto q_indices = get_ints(cuopt::remote::FIELD_Q_INDICES); - auto q_offsets = get_ints(cuopt::remote::FIELD_Q_OFFSETS); - if (!q_values.empty() && !q_indices.empty() && !q_offsets.empty()) { - cpu_problem.set_quadratic_objective_matrix(q_values.data(), - static_cast(q_values.size()), - q_indices.data(), - static_cast(q_indices.size()), - q_offsets.data(), - static_cast(q_offsets.size())); - } - - // String arrays (may also be in header; these override if present as chunked arrays) - auto var_names = get_string_list(cuopt::remote::FIELD_VARIABLE_NAMES); - if (!var_names.empty()) { cpu_problem.set_variable_names(var_names); } - auto row_names = get_string_list(cuopt::remote::FIELD_ROW_NAMES); - if (!row_names.empty()) { cpu_problem.set_row_names(row_names); } +#include "generated_chunked_arrays_to_problem.inc" } // ============================================================================= // Chunked array request building (client-side) // ============================================================================= -namespace { - -template -void chunk_typed_array(std::vector& out, - cuopt::remote::ArrayFieldId field_id, - const std::vector& data, - const std::string& upload_id, - int64_t chunk_data_budget) -{ - if (data.empty()) return; - - const int64_t elem_size = static_cast(sizeof(T)); - const int64_t total_elements = static_cast(data.size()); - - int64_t elems_per_chunk = chunk_data_budget / elem_size; - if (elems_per_chunk <= 0) elems_per_chunk = 1; - - const auto* raw = reinterpret_cast(data.data()); - - for (int64_t offset = 0; offset < total_elements; offset += elems_per_chunk) { - int64_t count = std::min(elems_per_chunk, total_elements - offset); - int64_t byte_offset = offset * elem_size; - int64_t byte_count = count * elem_size; - - cuopt::remote::SendArrayChunkRequest req; - req.set_upload_id(upload_id); - auto* ac = req.mutable_chunk(); - ac->set_field_id(field_id); - ac->set_element_offset(offset); - ac->set_total_elements(total_elements); - ac->set_data(raw + byte_offset, byte_count); - out.push_back(std::move(req)); - } -} - -void chunk_byte_blob(std::vector& out, - cuopt::remote::ArrayFieldId field_id, - const std::vector& data, - const std::string& upload_id, - int64_t chunk_data_budget) -{ - chunk_typed_array(out, field_id, data, upload_id, chunk_data_budget); -} - -} // namespace - template std::vector build_array_chunk_requests( const cpu_optimization_problem_t& problem, const std::string& upload_id, int64_t chunk_size_bytes) { - std::vector requests; - - auto values = problem.get_constraint_matrix_values_host(); - auto indices = problem.get_constraint_matrix_indices_host(); - auto offsets = problem.get_constraint_matrix_offsets_host(); - auto obj = problem.get_objective_coefficients_host(); - auto var_lb = problem.get_variable_lower_bounds_host(); - auto var_ub = problem.get_variable_upper_bounds_host(); - auto con_lb = problem.get_constraint_lower_bounds_host(); - auto con_ub = problem.get_constraint_upper_bounds_host(); - auto b = problem.get_constraint_bounds_host(); - - chunk_typed_array(requests, cuopt::remote::FIELD_A_VALUES, values, upload_id, chunk_size_bytes); - chunk_typed_array(requests, cuopt::remote::FIELD_A_INDICES, indices, upload_id, chunk_size_bytes); - chunk_typed_array(requests, cuopt::remote::FIELD_A_OFFSETS, offsets, upload_id, chunk_size_bytes); - chunk_typed_array(requests, cuopt::remote::FIELD_C, obj, upload_id, chunk_size_bytes); - chunk_typed_array( - requests, cuopt::remote::FIELD_VARIABLE_LOWER_BOUNDS, var_lb, upload_id, chunk_size_bytes); - chunk_typed_array( - requests, cuopt::remote::FIELD_VARIABLE_UPPER_BOUNDS, var_ub, upload_id, chunk_size_bytes); - chunk_typed_array( - requests, cuopt::remote::FIELD_CONSTRAINT_LOWER_BOUNDS, con_lb, upload_id, chunk_size_bytes); - chunk_typed_array( - requests, cuopt::remote::FIELD_CONSTRAINT_UPPER_BOUNDS, con_ub, upload_id, chunk_size_bytes); - chunk_typed_array(requests, cuopt::remote::FIELD_B, b, upload_id, chunk_size_bytes); - - auto row_types = problem.get_row_types_host(); - if (!row_types.empty()) { - std::vector rt_bytes(row_types.begin(), row_types.end()); - chunk_byte_blob( - requests, cuopt::remote::FIELD_ROW_TYPES, rt_bytes, upload_id, chunk_size_bytes); - } - - auto var_types = problem.get_variable_types_host(); - if (!var_types.empty()) { - std::vector vt_bytes; - vt_bytes.reserve(var_types.size()); - for (const auto& vt : var_types) { - switch (vt) { - case var_t::CONTINUOUS: vt_bytes.push_back('C'); break; - case var_t::INTEGER: vt_bytes.push_back('I'); break; - default: - throw std::runtime_error("chunk_problem_to_proto: unknown var_t value " + - std::to_string(static_cast(vt))); - } - } - chunk_byte_blob( - requests, cuopt::remote::FIELD_VARIABLE_TYPES, vt_bytes, upload_id, chunk_size_bytes); - } - - if (problem.has_quadratic_objective()) { - const auto& q_values = problem.get_quadratic_objective_values(); - const auto& q_indices = problem.get_quadratic_objective_indices(); - const auto& q_offsets = problem.get_quadratic_objective_offsets(); - chunk_typed_array( - requests, cuopt::remote::FIELD_Q_VALUES, q_values, upload_id, chunk_size_bytes); - chunk_typed_array( - requests, cuopt::remote::FIELD_Q_INDICES, q_indices, upload_id, chunk_size_bytes); - chunk_typed_array( - requests, cuopt::remote::FIELD_Q_OFFSETS, q_offsets, upload_id, chunk_size_bytes); - } - - auto names_to_blob = [](const std::vector& names) -> std::vector { - if (names.empty()) return {}; - size_t total = 0; - for (const auto& n : names) - total += n.size() + 1; - std::vector blob(total); - size_t pos = 0; - for (const auto& n : names) { - std::memcpy(blob.data() + pos, n.data(), n.size()); - pos += n.size(); - blob[pos++] = '\0'; - } - return blob; - }; - - auto var_names_blob = names_to_blob(problem.get_variable_names()); - auto row_names_blob = names_to_blob(problem.get_row_names()); - chunk_byte_blob( - requests, cuopt::remote::FIELD_VARIABLE_NAMES, var_names_blob, upload_id, chunk_size_bytes); - chunk_byte_blob( - requests, cuopt::remote::FIELD_ROW_NAMES, row_names_blob, upload_id, chunk_size_bytes); - - return requests; +#include "generated_build_array_chunks.inc" } // Explicit template instantiations diff --git a/cpp/src/grpc/grpc_settings_mapper.cpp b/cpp/src/grpc/grpc_settings_mapper.cpp index 8885b2e35..1d341b6fc 100644 --- a/cpp/src/grpc/grpc_settings_mapper.cpp +++ b/cpp/src/grpc/grpc_settings_mapper.cpp @@ -18,219 +18,35 @@ namespace cuopt::linear_programming { namespace { - -// Convert cuOpt pdlp_solver_mode_t to protobuf enum -cuopt::remote::PDLPSolverMode to_proto_pdlp_mode(pdlp_solver_mode_t mode) -{ - switch (mode) { - case pdlp_solver_mode_t::Stable1: return cuopt::remote::Stable1; - case pdlp_solver_mode_t::Stable2: return cuopt::remote::Stable2; - case pdlp_solver_mode_t::Methodical1: return cuopt::remote::Methodical1; - case pdlp_solver_mode_t::Fast1: return cuopt::remote::Fast1; - case pdlp_solver_mode_t::Stable3: return cuopt::remote::Stable3; - } - throw std::invalid_argument("Unknown pdlp_solver_mode_t: " + - std::to_string(static_cast(mode))); -} - -// Convert protobuf enum to cuOpt pdlp_solver_mode_t -pdlp_solver_mode_t from_proto_pdlp_mode(cuopt::remote::PDLPSolverMode mode) -{ - switch (mode) { - case cuopt::remote::Stable1: return pdlp_solver_mode_t::Stable1; - case cuopt::remote::Stable2: return pdlp_solver_mode_t::Stable2; - case cuopt::remote::Methodical1: return pdlp_solver_mode_t::Methodical1; - case cuopt::remote::Fast1: return pdlp_solver_mode_t::Fast1; - case cuopt::remote::Stable3: return pdlp_solver_mode_t::Stable3; - } - throw std::invalid_argument("Unknown PDLPSolverMode: " + std::to_string(static_cast(mode))); -} - -// Convert cuOpt method_t to protobuf enum -cuopt::remote::LPMethod to_proto_method(method_t method) -{ - switch (method) { - case method_t::Concurrent: return cuopt::remote::Concurrent; - case method_t::PDLP: return cuopt::remote::PDLP; - case method_t::DualSimplex: return cuopt::remote::DualSimplex; - case method_t::Barrier: return cuopt::remote::Barrier; - } - throw std::invalid_argument("Unknown method_t: " + std::to_string(static_cast(method))); -} - -// Convert protobuf enum to cuOpt method_t -method_t from_proto_method(cuopt::remote::LPMethod method) -{ - switch (method) { - case cuopt::remote::Concurrent: return method_t::Concurrent; - case cuopt::remote::PDLP: return method_t::PDLP; - case cuopt::remote::DualSimplex: return method_t::DualSimplex; - case cuopt::remote::Barrier: return method_t::Barrier; - } - throw std::invalid_argument("Unknown LPMethod: " + std::to_string(static_cast(method))); -} - -} // anonymous namespace +#include "generated_enum_converters_settings.inc" +} // namespace template void map_pdlp_settings_to_proto(const pdlp_solver_settings_t& settings, cuopt::remote::PDLPSolverSettings* pb_settings) { - // Termination tolerances (all names match cuOpt API) - pb_settings->set_absolute_gap_tolerance(settings.tolerances.absolute_gap_tolerance); - pb_settings->set_relative_gap_tolerance(settings.tolerances.relative_gap_tolerance); - pb_settings->set_primal_infeasible_tolerance(settings.tolerances.primal_infeasible_tolerance); - pb_settings->set_dual_infeasible_tolerance(settings.tolerances.dual_infeasible_tolerance); - pb_settings->set_absolute_dual_tolerance(settings.tolerances.absolute_dual_tolerance); - pb_settings->set_relative_dual_tolerance(settings.tolerances.relative_dual_tolerance); - pb_settings->set_absolute_primal_tolerance(settings.tolerances.absolute_primal_tolerance); - pb_settings->set_relative_primal_tolerance(settings.tolerances.relative_primal_tolerance); - - // Limits - pb_settings->set_time_limit(settings.time_limit); - // Avoid emitting a huge number when the iteration limit is the library default. - // Use -1 sentinel for "unset/use server defaults". - if (settings.iteration_limit == std::numeric_limits::max()) { - pb_settings->set_iteration_limit(-1); - } else { - pb_settings->set_iteration_limit(static_cast(settings.iteration_limit)); - } - - // Solver configuration - pb_settings->set_log_to_console(settings.log_to_console); - pb_settings->set_detect_infeasibility(settings.detect_infeasibility); - pb_settings->set_strict_infeasibility(settings.strict_infeasibility); - pb_settings->set_pdlp_solver_mode(to_proto_pdlp_mode(settings.pdlp_solver_mode)); - pb_settings->set_method(to_proto_method(settings.method)); - pb_settings->set_presolver(static_cast(settings.presolver)); - pb_settings->set_dual_postsolve(settings.dual_postsolve); - pb_settings->set_crossover(settings.crossover); - pb_settings->set_num_gpus(settings.num_gpus); - - pb_settings->set_per_constraint_residual(settings.per_constraint_residual); - pb_settings->set_cudss_deterministic(settings.cudss_deterministic); - pb_settings->set_folding(settings.folding); - pb_settings->set_augmented(settings.augmented); - pb_settings->set_dualize(settings.dualize); - pb_settings->set_ordering(settings.ordering); - pb_settings->set_barrier_dual_initial_point(settings.barrier_dual_initial_point); - pb_settings->set_eliminate_dense_columns(settings.eliminate_dense_columns); - pb_settings->set_pdlp_precision(static_cast(settings.pdlp_precision)); - pb_settings->set_save_best_primal_so_far(settings.save_best_primal_so_far); - pb_settings->set_first_primal_feasible(settings.first_primal_feasible); +#include "generated_pdlp_settings_to_proto.inc" } template void map_proto_to_pdlp_settings(const cuopt::remote::PDLPSolverSettings& pb_settings, pdlp_solver_settings_t& settings) { - // Termination tolerances (all names match cuOpt API) - settings.tolerances.absolute_gap_tolerance = pb_settings.absolute_gap_tolerance(); - settings.tolerances.relative_gap_tolerance = pb_settings.relative_gap_tolerance(); - settings.tolerances.primal_infeasible_tolerance = pb_settings.primal_infeasible_tolerance(); - settings.tolerances.dual_infeasible_tolerance = pb_settings.dual_infeasible_tolerance(); - settings.tolerances.absolute_dual_tolerance = pb_settings.absolute_dual_tolerance(); - settings.tolerances.relative_dual_tolerance = pb_settings.relative_dual_tolerance(); - settings.tolerances.absolute_primal_tolerance = pb_settings.absolute_primal_tolerance(); - settings.tolerances.relative_primal_tolerance = pb_settings.relative_primal_tolerance(); - - // Limits - settings.time_limit = pb_settings.time_limit(); - // proto3 defaults numeric fields to 0; treat negative iteration_limit as "unset" - // so the server keeps the library default (typically max()). - if (pb_settings.iteration_limit() >= 0) { - const auto limit = pb_settings.iteration_limit(); - settings.iteration_limit = (limit > static_cast(std::numeric_limits::max())) - ? std::numeric_limits::max() - : static_cast(limit); - } - - // Solver configuration - settings.log_to_console = pb_settings.log_to_console(); - settings.detect_infeasibility = pb_settings.detect_infeasibility(); - settings.strict_infeasibility = pb_settings.strict_infeasibility(); - settings.pdlp_solver_mode = from_proto_pdlp_mode(pb_settings.pdlp_solver_mode()); - settings.method = from_proto_method(pb_settings.method()); - { - auto pv = pb_settings.presolver(); - settings.presolver = (pv >= CUOPT_PRESOLVE_DEFAULT && pv <= CUOPT_PRESOLVE_PSLP) - ? static_cast(pv) - : presolver_t::Default; - } - settings.dual_postsolve = pb_settings.dual_postsolve(); - settings.crossover = pb_settings.crossover(); - settings.num_gpus = pb_settings.num_gpus(); - - settings.per_constraint_residual = pb_settings.per_constraint_residual(); - settings.cudss_deterministic = pb_settings.cudss_deterministic(); - settings.folding = pb_settings.folding(); - settings.augmented = pb_settings.augmented(); - settings.dualize = pb_settings.dualize(); - settings.ordering = pb_settings.ordering(); - settings.barrier_dual_initial_point = pb_settings.barrier_dual_initial_point(); - settings.eliminate_dense_columns = pb_settings.eliminate_dense_columns(); - { - auto pv = pb_settings.pdlp_precision(); - settings.pdlp_precision = - (pv >= CUOPT_PDLP_DEFAULT_PRECISION && pv <= CUOPT_PDLP_MIXED_PRECISION) - ? static_cast(pv) - : pdlp_precision_t::DefaultPrecision; - } - settings.save_best_primal_so_far = pb_settings.save_best_primal_so_far(); - settings.first_primal_feasible = pb_settings.first_primal_feasible(); +#include "generated_proto_to_pdlp_settings.inc" } template void map_mip_settings_to_proto(const mip_solver_settings_t& settings, cuopt::remote::MIPSolverSettings* pb_settings) { - // Limits - pb_settings->set_time_limit(settings.time_limit); - - // Tolerances (all names match cuOpt API) - pb_settings->set_relative_mip_gap(settings.tolerances.relative_mip_gap); - pb_settings->set_absolute_mip_gap(settings.tolerances.absolute_mip_gap); - pb_settings->set_integrality_tolerance(settings.tolerances.integrality_tolerance); - pb_settings->set_absolute_tolerance(settings.tolerances.absolute_tolerance); - pb_settings->set_relative_tolerance(settings.tolerances.relative_tolerance); - pb_settings->set_presolve_absolute_tolerance(settings.tolerances.presolve_absolute_tolerance); - - // Solver configuration - pb_settings->set_log_to_console(settings.log_to_console); - pb_settings->set_heuristics_only(settings.heuristics_only); - pb_settings->set_num_cpu_threads(settings.num_cpu_threads); - pb_settings->set_num_gpus(settings.num_gpus); - pb_settings->set_presolver(static_cast(settings.presolver)); - pb_settings->set_mip_scaling(settings.mip_scaling); +#include "generated_mip_settings_to_proto.inc" } template void map_proto_to_mip_settings(const cuopt::remote::MIPSolverSettings& pb_settings, mip_solver_settings_t& settings) { - // Limits - settings.time_limit = pb_settings.time_limit(); - - // Tolerances (all names match cuOpt API) - settings.tolerances.relative_mip_gap = pb_settings.relative_mip_gap(); - settings.tolerances.absolute_mip_gap = pb_settings.absolute_mip_gap(); - settings.tolerances.integrality_tolerance = pb_settings.integrality_tolerance(); - settings.tolerances.absolute_tolerance = pb_settings.absolute_tolerance(); - settings.tolerances.relative_tolerance = pb_settings.relative_tolerance(); - settings.tolerances.presolve_absolute_tolerance = pb_settings.presolve_absolute_tolerance(); - - // Solver configuration - settings.log_to_console = pb_settings.log_to_console(); - settings.heuristics_only = pb_settings.heuristics_only(); - settings.num_cpu_threads = pb_settings.num_cpu_threads(); - settings.num_gpus = pb_settings.num_gpus(); - { - auto pv = pb_settings.presolver(); - settings.presolver = (pv >= CUOPT_PRESOLVE_DEFAULT && pv <= CUOPT_PRESOLVE_PSLP) - ? static_cast(pv) - : presolver_t::Default; - } - settings.mip_scaling = pb_settings.mip_scaling(); +#include "generated_proto_to_mip_settings.inc" } // Explicit template instantiations diff --git a/cpp/src/grpc/grpc_solution_mapper.cpp b/cpp/src/grpc/grpc_solution_mapper.cpp index 700fd12c9..5f2acfa59 100644 --- a/cpp/src/grpc/grpc_solution_mapper.cpp +++ b/cpp/src/grpc/grpc_solution_mapper.cpp @@ -17,264 +17,86 @@ namespace cuopt::linear_programming { -// Convert cuOpt termination status to protobuf enum -cuopt::remote::PDLPTerminationStatus to_proto_pdlp_status(pdlp_termination_status_t status) -{ - switch (status) { - case pdlp_termination_status_t::NoTermination: return cuopt::remote::PDLP_NO_TERMINATION; - case pdlp_termination_status_t::NumericalError: return cuopt::remote::PDLP_NUMERICAL_ERROR; - case pdlp_termination_status_t::Optimal: return cuopt::remote::PDLP_OPTIMAL; - case pdlp_termination_status_t::PrimalInfeasible: return cuopt::remote::PDLP_PRIMAL_INFEASIBLE; - case pdlp_termination_status_t::DualInfeasible: return cuopt::remote::PDLP_DUAL_INFEASIBLE; - case pdlp_termination_status_t::IterationLimit: return cuopt::remote::PDLP_ITERATION_LIMIT; - case pdlp_termination_status_t::TimeLimit: return cuopt::remote::PDLP_TIME_LIMIT; - case pdlp_termination_status_t::ConcurrentLimit: return cuopt::remote::PDLP_CONCURRENT_LIMIT; - case pdlp_termination_status_t::PrimalFeasible: return cuopt::remote::PDLP_PRIMAL_FEASIBLE; - default: return cuopt::remote::PDLP_NO_TERMINATION; - } -} +namespace { +#include "generated_enum_converters_solution.inc" -// Convert protobuf enum to cuOpt termination status -pdlp_termination_status_t from_proto_pdlp_status(cuopt::remote::PDLPTerminationStatus status) +void add_result_array_descriptor(cuopt::remote::ChunkedResultHeader* header, + cuopt::remote::ResultFieldId fid, + int64_t count, + int64_t elem_size) { - switch (status) { - case cuopt::remote::PDLP_NO_TERMINATION: return pdlp_termination_status_t::NoTermination; - case cuopt::remote::PDLP_NUMERICAL_ERROR: return pdlp_termination_status_t::NumericalError; - case cuopt::remote::PDLP_OPTIMAL: return pdlp_termination_status_t::Optimal; - case cuopt::remote::PDLP_PRIMAL_INFEASIBLE: return pdlp_termination_status_t::PrimalInfeasible; - case cuopt::remote::PDLP_DUAL_INFEASIBLE: return pdlp_termination_status_t::DualInfeasible; - case cuopt::remote::PDLP_ITERATION_LIMIT: return pdlp_termination_status_t::IterationLimit; - case cuopt::remote::PDLP_TIME_LIMIT: return pdlp_termination_status_t::TimeLimit; - case cuopt::remote::PDLP_CONCURRENT_LIMIT: return pdlp_termination_status_t::ConcurrentLimit; - case cuopt::remote::PDLP_PRIMAL_FEASIBLE: return pdlp_termination_status_t::PrimalFeasible; - default: return pdlp_termination_status_t::NoTermination; - } + if (count <= 0) return; + auto* desc = header->add_arrays(); + desc->set_field_id(fid); + desc->set_total_elements(count); + desc->set_element_size_bytes(elem_size); } -// Convert MIP termination status -cuopt::remote::MIPTerminationStatus to_proto_mip_status(mip_termination_status_t status) +template +std::vector doubles_to_bytes(const std::vector& vec) { - switch (status) { - case mip_termination_status_t::NoTermination: return cuopt::remote::MIP_NO_TERMINATION; - case mip_termination_status_t::Optimal: return cuopt::remote::MIP_OPTIMAL; - case mip_termination_status_t::FeasibleFound: return cuopt::remote::MIP_FEASIBLE_FOUND; - case mip_termination_status_t::Infeasible: return cuopt::remote::MIP_INFEASIBLE; - case mip_termination_status_t::Unbounded: return cuopt::remote::MIP_UNBOUNDED; - case mip_termination_status_t::TimeLimit: return cuopt::remote::MIP_TIME_LIMIT; - case mip_termination_status_t::WorkLimit: return cuopt::remote::MIP_WORK_LIMIT; - default: return cuopt::remote::MIP_NO_TERMINATION; - } + std::vector tmp(vec.begin(), vec.end()); + std::vector bytes(tmp.size() * sizeof(double)); + std::memcpy(bytes.data(), tmp.data(), bytes.size()); + return bytes; } -mip_termination_status_t from_proto_mip_status(cuopt::remote::MIPTerminationStatus status) +template +std::vector bytes_to_typed(const std::map>& arrays, + int32_t field_id) { - switch (status) { - case cuopt::remote::MIP_NO_TERMINATION: return mip_termination_status_t::NoTermination; - case cuopt::remote::MIP_OPTIMAL: return mip_termination_status_t::Optimal; - case cuopt::remote::MIP_FEASIBLE_FOUND: return mip_termination_status_t::FeasibleFound; - case cuopt::remote::MIP_INFEASIBLE: return mip_termination_status_t::Infeasible; - case cuopt::remote::MIP_UNBOUNDED: return mip_termination_status_t::Unbounded; - case cuopt::remote::MIP_TIME_LIMIT: return mip_termination_status_t::TimeLimit; - case cuopt::remote::MIP_WORK_LIMIT: return mip_termination_status_t::WorkLimit; - default: return mip_termination_status_t::NoTermination; + auto it = arrays.find(field_id); + if (it == arrays.end() || it->second.empty()) return {}; + + const auto& raw = it->second; + if constexpr (std::is_same_v) { + if (raw.size() % sizeof(double) != 0) return {}; + size_t n = raw.size() / sizeof(double); + std::vector tmp(n); + std::memcpy(tmp.data(), raw.data(), n * sizeof(double)); + return std::vector(tmp.begin(), tmp.end()); + } else if constexpr (std::is_same_v) { + if (raw.size() % sizeof(double) != 0) return {}; + size_t n = raw.size() / sizeof(double); + std::vector v(n); + std::memcpy(v.data(), raw.data(), n * sizeof(double)); + return v; + } else { + if (raw.size() % sizeof(T) != 0) return {}; + size_t n = raw.size() / sizeof(T); + std::vector v(n); + std::memcpy(v.data(), raw.data(), n * sizeof(T)); + return v; } } +} // namespace + template void map_lp_solution_to_proto(const cpu_lp_solution_t& solution, cuopt::remote::LPSolution* pb_solution) { - pb_solution->set_termination_status(to_proto_pdlp_status(solution.get_termination_status())); - pb_solution->set_error_message(solution.get_error_status().what()); - - // Solution vectors - CPU solution already has data in host memory - const auto& primal = solution.get_primal_solution_host(); - const auto& dual = solution.get_dual_solution_host(); - const auto& reduced_cost = solution.get_reduced_cost_host(); - - for (const auto& v : primal) { - pb_solution->add_primal_solution(static_cast(v)); - } - for (const auto& v : dual) { - pb_solution->add_dual_solution(static_cast(v)); - } - for (const auto& v : reduced_cost) { - pb_solution->add_reduced_cost(static_cast(v)); - } - - // Statistics - pb_solution->set_l2_primal_residual(solution.get_l2_primal_residual()); - pb_solution->set_l2_dual_residual(solution.get_l2_dual_residual()); - pb_solution->set_primal_objective(solution.get_objective_value()); - pb_solution->set_dual_objective(solution.get_dual_objective_value()); - pb_solution->set_gap(solution.get_gap()); - pb_solution->set_nb_iterations(solution.get_num_iterations()); - pb_solution->set_solve_time(solution.get_solve_time()); - pb_solution->set_solved_by_pdlp(solution.is_solved_by_pdlp()); - - if (solution.has_warm_start_data()) { - auto* pb_ws = pb_solution->mutable_warm_start_data(); - const auto& ws = solution.get_cpu_pdlp_warm_start_data(); - - for (const auto& v : ws.current_primal_solution_) - pb_ws->add_current_primal_solution(static_cast(v)); - for (const auto& v : ws.current_dual_solution_) - pb_ws->add_current_dual_solution(static_cast(v)); - for (const auto& v : ws.initial_primal_average_) - pb_ws->add_initial_primal_average(static_cast(v)); - for (const auto& v : ws.initial_dual_average_) - pb_ws->add_initial_dual_average(static_cast(v)); - for (const auto& v : ws.current_ATY_) - pb_ws->add_current_aty(static_cast(v)); - for (const auto& v : ws.sum_primal_solutions_) - pb_ws->add_sum_primal_solutions(static_cast(v)); - for (const auto& v : ws.sum_dual_solutions_) - pb_ws->add_sum_dual_solutions(static_cast(v)); - for (const auto& v : ws.last_restart_duality_gap_primal_solution_) - pb_ws->add_last_restart_duality_gap_primal_solution(static_cast(v)); - for (const auto& v : ws.last_restart_duality_gap_dual_solution_) - pb_ws->add_last_restart_duality_gap_dual_solution(static_cast(v)); - - pb_ws->set_initial_primal_weight(static_cast(ws.initial_primal_weight_)); - pb_ws->set_initial_step_size(static_cast(ws.initial_step_size_)); - pb_ws->set_total_pdlp_iterations(static_cast(ws.total_pdlp_iterations_)); - pb_ws->set_total_pdhg_iterations(static_cast(ws.total_pdhg_iterations_)); - pb_ws->set_last_candidate_kkt_score(static_cast(ws.last_candidate_kkt_score_)); - pb_ws->set_last_restart_kkt_score(static_cast(ws.last_restart_kkt_score_)); - pb_ws->set_sum_solution_weight(static_cast(ws.sum_solution_weight_)); - pb_ws->set_iterations_since_last_restart( - static_cast(ws.iterations_since_last_restart_)); - } +#include "generated_lp_solution_to_proto.inc" } template cpu_lp_solution_t map_proto_to_lp_solution(const cuopt::remote::LPSolution& pb_solution) { - // Convert solution vectors - std::vector primal(pb_solution.primal_solution().begin(), - pb_solution.primal_solution().end()); - std::vector dual(pb_solution.dual_solution().begin(), pb_solution.dual_solution().end()); - std::vector reduced_cost(pb_solution.reduced_cost().begin(), - pb_solution.reduced_cost().end()); - - auto status = from_proto_pdlp_status(pb_solution.termination_status()); - auto obj = static_cast(pb_solution.primal_objective()); - auto dual_obj = static_cast(pb_solution.dual_objective()); - auto solve_t = pb_solution.solve_time(); - auto l2_pr = static_cast(pb_solution.l2_primal_residual()); - auto l2_dr = static_cast(pb_solution.l2_dual_residual()); - auto g = static_cast(pb_solution.gap()); - auto iters = static_cast(pb_solution.nb_iterations()); - auto by_pdlp = pb_solution.solved_by_pdlp(); - - if (pb_solution.has_warm_start_data()) { - const auto& pb_ws = pb_solution.warm_start_data(); - cpu_pdlp_warm_start_data_t ws; - - ws.current_primal_solution_.assign(pb_ws.current_primal_solution().begin(), - pb_ws.current_primal_solution().end()); - ws.current_dual_solution_.assign(pb_ws.current_dual_solution().begin(), - pb_ws.current_dual_solution().end()); - ws.initial_primal_average_.assign(pb_ws.initial_primal_average().begin(), - pb_ws.initial_primal_average().end()); - ws.initial_dual_average_.assign(pb_ws.initial_dual_average().begin(), - pb_ws.initial_dual_average().end()); - ws.current_ATY_.assign(pb_ws.current_aty().begin(), pb_ws.current_aty().end()); - ws.sum_primal_solutions_.assign(pb_ws.sum_primal_solutions().begin(), - pb_ws.sum_primal_solutions().end()); - ws.sum_dual_solutions_.assign(pb_ws.sum_dual_solutions().begin(), - pb_ws.sum_dual_solutions().end()); - ws.last_restart_duality_gap_primal_solution_.assign( - pb_ws.last_restart_duality_gap_primal_solution().begin(), - pb_ws.last_restart_duality_gap_primal_solution().end()); - ws.last_restart_duality_gap_dual_solution_.assign( - pb_ws.last_restart_duality_gap_dual_solution().begin(), - pb_ws.last_restart_duality_gap_dual_solution().end()); - - ws.initial_primal_weight_ = static_cast(pb_ws.initial_primal_weight()); - ws.initial_step_size_ = static_cast(pb_ws.initial_step_size()); - ws.total_pdlp_iterations_ = static_cast(pb_ws.total_pdlp_iterations()); - ws.total_pdhg_iterations_ = static_cast(pb_ws.total_pdhg_iterations()); - ws.last_candidate_kkt_score_ = static_cast(pb_ws.last_candidate_kkt_score()); - ws.last_restart_kkt_score_ = static_cast(pb_ws.last_restart_kkt_score()); - ws.sum_solution_weight_ = static_cast(pb_ws.sum_solution_weight()); - ws.iterations_since_last_restart_ = static_cast(pb_ws.iterations_since_last_restart()); - - return cpu_lp_solution_t(std::move(primal), - std::move(dual), - std::move(reduced_cost), - status, - obj, - dual_obj, - solve_t, - l2_pr, - l2_dr, - g, - iters, - by_pdlp, - std::move(ws)); - } - - return cpu_lp_solution_t(std::move(primal), - std::move(dual), - std::move(reduced_cost), - status, - obj, - dual_obj, - solve_t, - l2_pr, - l2_dr, - g, - iters, - by_pdlp); +#include "generated_proto_to_lp_solution.inc" } template void map_mip_solution_to_proto(const cpu_mip_solution_t& solution, cuopt::remote::MIPSolution* pb_solution) { - pb_solution->set_termination_status(to_proto_mip_status(solution.get_termination_status())); - pb_solution->set_error_message(solution.get_error_status().what()); - - // Solution vector - CPU solution already has data in host memory - const auto& sol_vec = solution.get_solution_host(); - for (const auto& v : sol_vec) { - pb_solution->add_solution(static_cast(v)); - } - - // Solution statistics - pb_solution->set_objective(solution.get_objective_value()); - pb_solution->set_mip_gap(solution.get_mip_gap()); - pb_solution->set_solution_bound(solution.get_solution_bound()); - pb_solution->set_total_solve_time(solution.get_solve_time()); - pb_solution->set_presolve_time(solution.get_presolve_time()); - pb_solution->set_max_constraint_violation(solution.get_max_constraint_violation()); - pb_solution->set_max_int_violation(solution.get_max_int_violation()); - pb_solution->set_max_variable_bound_violation(solution.get_max_variable_bound_violation()); - pb_solution->set_nodes(solution.get_num_nodes()); - pb_solution->set_simplex_iterations(solution.get_num_simplex_iterations()); +#include "generated_mip_solution_to_proto.inc" } template cpu_mip_solution_t map_proto_to_mip_solution( const cuopt::remote::MIPSolution& pb_solution) { - // Convert solution vector - std::vector solution_vec(pb_solution.solution().begin(), pb_solution.solution().end()); - - // Create CPU MIP solution with data - return cpu_mip_solution_t(std::move(solution_vec), - from_proto_mip_status(pb_solution.termination_status()), - static_cast(pb_solution.objective()), - static_cast(pb_solution.mip_gap()), - static_cast(pb_solution.solution_bound()), - pb_solution.total_solve_time(), - pb_solution.presolve_time(), - static_cast(pb_solution.max_constraint_violation()), - static_cast(pb_solution.max_int_violation()), - static_cast(pb_solution.max_variable_bound_violation()), - static_cast(pb_solution.nodes()), - static_cast(pb_solution.simplex_iterations())); +#include "generated_proto_to_mip_solution.inc" } // ============================================================================ @@ -284,156 +106,31 @@ cpu_mip_solution_t map_proto_to_mip_solution( template size_t estimate_lp_solution_proto_size(const cpu_lp_solution_t& solution) { - size_t est = 0; - est += static_cast(solution.get_primal_solution_size()) * sizeof(double); - est += static_cast(solution.get_dual_solution_size()) * sizeof(double); - est += static_cast(solution.get_reduced_cost_size()) * sizeof(double); - if (solution.has_warm_start_data()) { - const auto& ws = solution.get_cpu_pdlp_warm_start_data(); - est += ws.current_primal_solution_.size() * sizeof(double); - est += ws.current_dual_solution_.size() * sizeof(double); - est += ws.initial_primal_average_.size() * sizeof(double); - est += ws.initial_dual_average_.size() * sizeof(double); - est += ws.current_ATY_.size() * sizeof(double); - est += ws.sum_primal_solutions_.size() * sizeof(double); - est += ws.sum_dual_solutions_.size() * sizeof(double); - est += ws.last_restart_duality_gap_primal_solution_.size() * sizeof(double); - est += ws.last_restart_duality_gap_dual_solution_.size() * sizeof(double); - } - est += 512; // scalars + tags overhead - return est; +#include "generated_estimate_lp_size.inc" } template size_t estimate_mip_solution_proto_size(const cpu_mip_solution_t& solution) { - size_t est = 0; - est += static_cast(solution.get_solution_size()) * sizeof(double); - est += 256; // scalars + tags overhead - return est; +#include "generated_estimate_mip_size.inc" } // ============================================================================ // Chunked result header population // ============================================================================ -namespace { -void add_result_array_descriptor(cuopt::remote::ChunkedResultHeader* header, - cuopt::remote::ResultFieldId fid, - int64_t count, - int64_t elem_size) -{ - if (count <= 0) return; - auto* desc = header->add_arrays(); - desc->set_field_id(fid); - desc->set_total_elements(count); - desc->set_element_size_bytes(elem_size); -} - -template -std::vector doubles_to_bytes(const std::vector& vec) -{ - std::vector tmp(vec.begin(), vec.end()); - std::vector bytes(tmp.size() * sizeof(double)); - std::memcpy(bytes.data(), tmp.data(), bytes.size()); - return bytes; -} -} // namespace - template void populate_chunked_result_header_lp(const cpu_lp_solution_t& solution, cuopt::remote::ChunkedResultHeader* header) { - header->set_is_mip(false); - header->set_lp_termination_status(to_proto_pdlp_status(solution.get_termination_status())); - header->set_error_message(solution.get_error_status().what()); - header->set_l2_primal_residual(solution.get_l2_primal_residual()); - header->set_l2_dual_residual(solution.get_l2_dual_residual()); - header->set_primal_objective(solution.get_objective_value()); - header->set_dual_objective(solution.get_dual_objective_value()); - header->set_gap(solution.get_gap()); - header->set_nb_iterations(solution.get_num_iterations()); - header->set_solve_time(solution.get_solve_time()); - header->set_solved_by_pdlp(solution.is_solved_by_pdlp()); - - const auto& primal = solution.get_primal_solution_host(); - const auto& dual = solution.get_dual_solution_host(); - const auto& reduced_cost = solution.get_reduced_cost_host(); - - add_result_array_descriptor( - header, cuopt::remote::RESULT_PRIMAL_SOLUTION, primal.size(), sizeof(double)); - add_result_array_descriptor( - header, cuopt::remote::RESULT_DUAL_SOLUTION, dual.size(), sizeof(double)); - add_result_array_descriptor( - header, cuopt::remote::RESULT_REDUCED_COST, reduced_cost.size(), sizeof(double)); - - if (solution.has_warm_start_data()) { - const auto& ws = solution.get_cpu_pdlp_warm_start_data(); - header->set_ws_initial_primal_weight(static_cast(ws.initial_primal_weight_)); - header->set_ws_initial_step_size(static_cast(ws.initial_step_size_)); - header->set_ws_total_pdlp_iterations(static_cast(ws.total_pdlp_iterations_)); - header->set_ws_total_pdhg_iterations(static_cast(ws.total_pdhg_iterations_)); - header->set_ws_last_candidate_kkt_score(static_cast(ws.last_candidate_kkt_score_)); - header->set_ws_last_restart_kkt_score(static_cast(ws.last_restart_kkt_score_)); - header->set_ws_sum_solution_weight(static_cast(ws.sum_solution_weight_)); - header->set_ws_iterations_since_last_restart( - static_cast(ws.iterations_since_last_restart_)); - - add_result_array_descriptor(header, - cuopt::remote::RESULT_WS_CURRENT_PRIMAL, - ws.current_primal_solution_.size(), - sizeof(double)); - add_result_array_descriptor(header, - cuopt::remote::RESULT_WS_CURRENT_DUAL, - ws.current_dual_solution_.size(), - sizeof(double)); - add_result_array_descriptor(header, - cuopt::remote::RESULT_WS_INITIAL_PRIMAL_AVG, - ws.initial_primal_average_.size(), - sizeof(double)); - add_result_array_descriptor(header, - cuopt::remote::RESULT_WS_INITIAL_DUAL_AVG, - ws.initial_dual_average_.size(), - sizeof(double)); - add_result_array_descriptor( - header, cuopt::remote::RESULT_WS_CURRENT_ATY, ws.current_ATY_.size(), sizeof(double)); - add_result_array_descriptor( - header, cuopt::remote::RESULT_WS_SUM_PRIMAL, ws.sum_primal_solutions_.size(), sizeof(double)); - add_result_array_descriptor( - header, cuopt::remote::RESULT_WS_SUM_DUAL, ws.sum_dual_solutions_.size(), sizeof(double)); - add_result_array_descriptor(header, - cuopt::remote::RESULT_WS_LAST_RESTART_GAP_PRIMAL, - ws.last_restart_duality_gap_primal_solution_.size(), - sizeof(double)); - add_result_array_descriptor(header, - cuopt::remote::RESULT_WS_LAST_RESTART_GAP_DUAL, - ws.last_restart_duality_gap_dual_solution_.size(), - sizeof(double)); - } +#include "generated_lp_chunked_header.inc" } template void populate_chunked_result_header_mip(const cpu_mip_solution_t& solution, cuopt::remote::ChunkedResultHeader* header) { - header->set_is_mip(true); - header->set_mip_termination_status(to_proto_mip_status(solution.get_termination_status())); - header->set_mip_error_message(solution.get_error_status().what()); - header->set_mip_objective(solution.get_objective_value()); - header->set_mip_gap(solution.get_mip_gap()); - header->set_solution_bound(solution.get_solution_bound()); - header->set_total_solve_time(solution.get_solve_time()); - header->set_presolve_time(solution.get_presolve_time()); - header->set_max_constraint_violation(solution.get_max_constraint_violation()); - header->set_max_int_violation(solution.get_max_int_violation()); - header->set_max_variable_bound_violation(solution.get_max_variable_bound_violation()); - header->set_nodes(solution.get_num_nodes()); - header->set_simplex_iterations(solution.get_num_simplex_iterations()); - - add_result_array_descriptor(header, - cuopt::remote::RESULT_MIP_SOLUTION, - solution.get_solution_host().size(), - sizeof(double)); +#include "generated_mip_chunked_header.inc" } // ============================================================================ @@ -444,176 +141,26 @@ template std::map> collect_lp_solution_arrays( const cpu_lp_solution_t& solution) { - std::map> arrays; - - const auto& primal = solution.get_primal_solution_host(); - const auto& dual = solution.get_dual_solution_host(); - const auto& reduced_cost = solution.get_reduced_cost_host(); - - if (!primal.empty()) { arrays[cuopt::remote::RESULT_PRIMAL_SOLUTION] = doubles_to_bytes(primal); } - if (!dual.empty()) { arrays[cuopt::remote::RESULT_DUAL_SOLUTION] = doubles_to_bytes(dual); } - if (!reduced_cost.empty()) { - arrays[cuopt::remote::RESULT_REDUCED_COST] = doubles_to_bytes(reduced_cost); - } - - if (solution.has_warm_start_data()) { - const auto& ws = solution.get_cpu_pdlp_warm_start_data(); - if (!ws.current_primal_solution_.empty()) { - arrays[cuopt::remote::RESULT_WS_CURRENT_PRIMAL] = - doubles_to_bytes(ws.current_primal_solution_); - } - if (!ws.current_dual_solution_.empty()) { - arrays[cuopt::remote::RESULT_WS_CURRENT_DUAL] = doubles_to_bytes(ws.current_dual_solution_); - } - if (!ws.initial_primal_average_.empty()) { - arrays[cuopt::remote::RESULT_WS_INITIAL_PRIMAL_AVG] = - doubles_to_bytes(ws.initial_primal_average_); - } - if (!ws.initial_dual_average_.empty()) { - arrays[cuopt::remote::RESULT_WS_INITIAL_DUAL_AVG] = - doubles_to_bytes(ws.initial_dual_average_); - } - if (!ws.current_ATY_.empty()) { - arrays[cuopt::remote::RESULT_WS_CURRENT_ATY] = doubles_to_bytes(ws.current_ATY_); - } - if (!ws.sum_primal_solutions_.empty()) { - arrays[cuopt::remote::RESULT_WS_SUM_PRIMAL] = doubles_to_bytes(ws.sum_primal_solutions_); - } - if (!ws.sum_dual_solutions_.empty()) { - arrays[cuopt::remote::RESULT_WS_SUM_DUAL] = doubles_to_bytes(ws.sum_dual_solutions_); - } - if (!ws.last_restart_duality_gap_primal_solution_.empty()) { - arrays[cuopt::remote::RESULT_WS_LAST_RESTART_GAP_PRIMAL] = - doubles_to_bytes(ws.last_restart_duality_gap_primal_solution_); - } - if (!ws.last_restart_duality_gap_dual_solution_.empty()) { - arrays[cuopt::remote::RESULT_WS_LAST_RESTART_GAP_DUAL] = - doubles_to_bytes(ws.last_restart_duality_gap_dual_solution_); - } - } - - return arrays; +#include "generated_collect_lp_arrays.inc" } template std::map> collect_mip_solution_arrays( const cpu_mip_solution_t& solution) { - std::map> arrays; - const auto& sol_vec = solution.get_solution_host(); - if (!sol_vec.empty()) { arrays[cuopt::remote::RESULT_MIP_SOLUTION] = doubles_to_bytes(sol_vec); } - return arrays; +#include "generated_collect_mip_arrays.inc" } // ============================================================================ // Chunked result -> solution (client-side) // ============================================================================ -namespace { - -template -std::vector bytes_to_typed(const std::map>& arrays, - int32_t field_id) -{ - auto it = arrays.find(field_id); - if (it == arrays.end() || it->second.empty()) return {}; - - const auto& raw = it->second; - if constexpr (std::is_same_v) { - if (raw.size() % sizeof(double) != 0) return {}; - size_t n = raw.size() / sizeof(double); - std::vector tmp(n); - std::memcpy(tmp.data(), raw.data(), n * sizeof(double)); - return std::vector(tmp.begin(), tmp.end()); - } else if constexpr (std::is_same_v) { - if (raw.size() % sizeof(double) != 0) return {}; - size_t n = raw.size() / sizeof(double); - std::vector v(n); - std::memcpy(v.data(), raw.data(), n * sizeof(double)); - return v; - } else { - if (raw.size() % sizeof(T) != 0) return {}; - size_t n = raw.size() / sizeof(T); - std::vector v(n); - std::memcpy(v.data(), raw.data(), n * sizeof(T)); - return v; - } -} - -} // namespace - template cpu_lp_solution_t chunked_result_to_lp_solution( const cuopt::remote::ChunkedResultHeader& h, const std::map>& arrays) { - auto primal = bytes_to_typed(arrays, cuopt::remote::RESULT_PRIMAL_SOLUTION); - auto dual = bytes_to_typed(arrays, cuopt::remote::RESULT_DUAL_SOLUTION); - auto reduced_cost = bytes_to_typed(arrays, cuopt::remote::RESULT_REDUCED_COST); - - auto status = from_proto_pdlp_status(h.lp_termination_status()); - auto obj = static_cast(h.primal_objective()); - auto dual_obj = static_cast(h.dual_objective()); - auto solve_t = h.solve_time(); - auto l2_pr = static_cast(h.l2_primal_residual()); - auto l2_dr = static_cast(h.l2_dual_residual()); - auto g = static_cast(h.gap()); - auto iters = static_cast(h.nb_iterations()); - auto by_pdlp = h.solved_by_pdlp(); - - auto ws_primal = bytes_to_typed(arrays, cuopt::remote::RESULT_WS_CURRENT_PRIMAL); - if (!ws_primal.empty()) { - cpu_pdlp_warm_start_data_t ws; - ws.current_primal_solution_ = std::move(ws_primal); - ws.current_dual_solution_ = bytes_to_typed(arrays, cuopt::remote::RESULT_WS_CURRENT_DUAL); - ws.initial_primal_average_ = - bytes_to_typed(arrays, cuopt::remote::RESULT_WS_INITIAL_PRIMAL_AVG); - ws.initial_dual_average_ = - bytes_to_typed(arrays, cuopt::remote::RESULT_WS_INITIAL_DUAL_AVG); - ws.current_ATY_ = bytes_to_typed(arrays, cuopt::remote::RESULT_WS_CURRENT_ATY); - ws.sum_primal_solutions_ = bytes_to_typed(arrays, cuopt::remote::RESULT_WS_SUM_PRIMAL); - ws.sum_dual_solutions_ = bytes_to_typed(arrays, cuopt::remote::RESULT_WS_SUM_DUAL); - ws.last_restart_duality_gap_primal_solution_ = - bytes_to_typed(arrays, cuopt::remote::RESULT_WS_LAST_RESTART_GAP_PRIMAL); - ws.last_restart_duality_gap_dual_solution_ = - bytes_to_typed(arrays, cuopt::remote::RESULT_WS_LAST_RESTART_GAP_DUAL); - - ws.initial_primal_weight_ = static_cast(h.ws_initial_primal_weight()); - ws.initial_step_size_ = static_cast(h.ws_initial_step_size()); - ws.total_pdlp_iterations_ = static_cast(h.ws_total_pdlp_iterations()); - ws.total_pdhg_iterations_ = static_cast(h.ws_total_pdhg_iterations()); - ws.last_candidate_kkt_score_ = static_cast(h.ws_last_candidate_kkt_score()); - ws.last_restart_kkt_score_ = static_cast(h.ws_last_restart_kkt_score()); - ws.sum_solution_weight_ = static_cast(h.ws_sum_solution_weight()); - ws.iterations_since_last_restart_ = static_cast(h.ws_iterations_since_last_restart()); - - return cpu_lp_solution_t(std::move(primal), - std::move(dual), - std::move(reduced_cost), - status, - obj, - dual_obj, - solve_t, - l2_pr, - l2_dr, - g, - iters, - by_pdlp, - std::move(ws)); - } - - return cpu_lp_solution_t(std::move(primal), - std::move(dual), - std::move(reduced_cost), - status, - obj, - dual_obj, - solve_t, - l2_pr, - l2_dr, - g, - iters, - by_pdlp); +#include "generated_chunked_to_lp_solution.inc" } template @@ -621,20 +168,7 @@ cpu_mip_solution_t chunked_result_to_mip_solution( const cuopt::remote::ChunkedResultHeader& h, const std::map>& arrays) { - auto sol_vec = bytes_to_typed(arrays, cuopt::remote::RESULT_MIP_SOLUTION); - - return cpu_mip_solution_t(std::move(sol_vec), - from_proto_mip_status(h.mip_termination_status()), - static_cast(h.mip_objective()), - static_cast(h.mip_gap()), - static_cast(h.solution_bound()), - h.total_solve_time(), - h.presolve_time(), - static_cast(h.max_constraint_violation()), - static_cast(h.max_int_violation()), - static_cast(h.max_variable_bound_violation()), - static_cast(h.nodes()), - static_cast(h.simplex_iterations())); +#include "generated_chunked_to_mip_solution.inc" } // ============================================================================ diff --git a/cpp/src/grpc/grpc_solution_mapper.hpp b/cpp/src/grpc/grpc_solution_mapper.hpp index 127bdb2c9..58d37f730 100644 --- a/cpp/src/grpc/grpc_solution_mapper.hpp +++ b/cpp/src/grpc/grpc_solution_mapper.hpp @@ -58,34 +58,6 @@ template cpu_mip_solution_t map_proto_to_mip_solution( const cuopt::remote::MIPSolution& pb_solution); -/** - * @brief Convert cuOpt termination status to protobuf enum. - * @param status cuOpt PDLP termination status - * @return Protobuf PDLPTerminationStatus enum - */ -cuopt::remote::PDLPTerminationStatus to_proto_pdlp_status(pdlp_termination_status_t status); - -/** - * @brief Convert protobuf enum to cuOpt termination status. - * @param status Protobuf PDLPTerminationStatus enum - * @return cuOpt PDLP termination status - */ -pdlp_termination_status_t from_proto_pdlp_status(cuopt::remote::PDLPTerminationStatus status); - -/** - * @brief Convert cuOpt MIP termination status to protobuf enum. - * @param status cuOpt MIP termination status - * @return Protobuf MIPTerminationStatus enum - */ -cuopt::remote::MIPTerminationStatus to_proto_mip_status(mip_termination_status_t status); - -/** - * @brief Convert protobuf enum to cuOpt MIP termination status. - * @param status Protobuf MIPTerminationStatus enum - * @return cuOpt MIP termination status - */ -mip_termination_status_t from_proto_mip_status(cuopt::remote::MIPTerminationStatus status); - // ============================================================================ // Chunked result support (for results exceeding gRPC max message size) // ============================================================================ diff --git a/cpp/src/grpc/server/grpc_field_element_size.hpp b/cpp/src/grpc/server/grpc_field_element_size.hpp index f53e99dbf..529785379 100644 --- a/cpp/src/grpc/server/grpc_field_element_size.hpp +++ b/cpp/src/grpc/server/grpc_field_element_size.hpp @@ -3,10 +3,6 @@ * reserved. SPDX-License-Identifier: Apache-2.0 */ -// Codegen target: this file maps ArrayFieldId enum values to their element byte sizes. -// A future version of cpp/codegen/generate_conversions.py can produce this from -// a problem_arrays section in field_registry.yaml. - #pragma once #ifdef CUOPT_ENABLE_GRPC @@ -16,27 +12,7 @@ inline int64_t array_field_element_size(cuopt::remote::ArrayFieldId field_id) { - switch (field_id) { - case cuopt::remote::FIELD_A_VALUES: - case cuopt::remote::FIELD_C: - case cuopt::remote::FIELD_B: - case cuopt::remote::FIELD_VARIABLE_LOWER_BOUNDS: - case cuopt::remote::FIELD_VARIABLE_UPPER_BOUNDS: - case cuopt::remote::FIELD_CONSTRAINT_LOWER_BOUNDS: - case cuopt::remote::FIELD_CONSTRAINT_UPPER_BOUNDS: - case cuopt::remote::FIELD_Q_VALUES: - case cuopt::remote::FIELD_INITIAL_PRIMAL: - case cuopt::remote::FIELD_INITIAL_DUAL: return 8; - case cuopt::remote::FIELD_A_INDICES: - case cuopt::remote::FIELD_A_OFFSETS: - case cuopt::remote::FIELD_Q_INDICES: - case cuopt::remote::FIELD_Q_OFFSETS: return 4; - case cuopt::remote::FIELD_ROW_TYPES: - case cuopt::remote::FIELD_VARIABLE_TYPES: - case cuopt::remote::FIELD_VARIABLE_NAMES: - case cuopt::remote::FIELD_ROW_NAMES: return 1; - } - return -1; +#include "generated_array_field_element_size.inc" } #endif // CUOPT_ENABLE_GRPC diff --git a/cpp/src/grpc/server/grpc_server_main.cpp b/cpp/src/grpc/server/grpc_server_main.cpp index 5cc947a81..cb73469cc 100644 --- a/cpp/src/grpc/server/grpc_server_main.cpp +++ b/cpp/src/grpc/server/grpc_server_main.cpp @@ -65,7 +65,7 @@ int main(int argc, char** argv) argparse::ArgumentParser program("cuopt_grpc_server", version_string); - program.add_argument("-p", "--port").help("Listen port").default_value(8765).scan<'i', int>(); + program.add_argument("-p", "--port").help("Listen port").default_value(5001).scan<'i', int>(); program.add_argument("-w", "--workers") .help("Number of worker processes") diff --git a/cpp/src/grpc/server/grpc_server_types.hpp b/cpp/src/grpc/server/grpc_server_types.hpp index 7afc668fb..04fbce4a9 100644 --- a/cpp/src/grpc/server/grpc_server_types.hpp +++ b/cpp/src/grpc/server/grpc_server_types.hpp @@ -156,7 +156,7 @@ struct JobWaiter { // ============================================================================= struct ServerConfig { - int port = 8765; + int port = 5001; int num_workers = 1; bool verbose = true; bool log_to_console = false; diff --git a/cpp/tests/linear_programming/grpc/CMakeLists.txt b/cpp/tests/linear_programming/grpc/CMakeLists.txt index 8b9715857..94486b0cd 100644 --- a/cpp/tests/linear_programming/grpc/CMakeLists.txt +++ b/cpp/tests/linear_programming/grpc/CMakeLists.txt @@ -16,6 +16,7 @@ target_include_directories(GRPC_CLIENT_TEST "${CUOPT_SOURCE_DIR}/include" "${CUOPT_SOURCE_DIR}/src/grpc" "${CUOPT_SOURCE_DIR}/src/grpc/client" + "${CUOPT_SOURCE_DIR}/codegen/generated" "${CUOPT_TEST_DIR}" "${CMAKE_CURRENT_SOURCE_DIR}" # For grpc_client_test_helper.hpp "${CMAKE_BINARY_DIR}" # For generated protobuf headers @@ -59,6 +60,7 @@ target_include_directories(GRPC_PIPE_SERIALIZATION_TEST "${CUOPT_SOURCE_DIR}/include" "${CUOPT_SOURCE_DIR}/src/grpc" "${CUOPT_SOURCE_DIR}/src/grpc/server" + "${CUOPT_SOURCE_DIR}/codegen/generated" "${CMAKE_BINARY_DIR}" # For generated protobuf headers ) diff --git a/cpp/tests/linear_programming/grpc/grpc_client_test.cpp b/cpp/tests/linear_programming/grpc/grpc_client_test.cpp index 46a18dc02..b3bd6654a 100644 --- a/cpp/tests/linear_programming/grpc/grpc_client_test.cpp +++ b/cpp/tests/linear_programming/grpc/grpc_client_test.cpp @@ -23,6 +23,7 @@ #include #include "grpc_client.hpp" #include "grpc_service_mapper.hpp" +#include "grpc_settings_mapper.hpp" #include #include @@ -834,7 +835,7 @@ TEST_F(GrpcClientTest, ChunkedDownload_FallbackOnResourceExhausted) cuopt::remote::StartChunkedDownloadResponse* resp) { resp->set_download_id("dl-001"); auto* h = resp->mutable_header(); - h->set_is_mip(false); + h->set_problem_category(cuopt::remote::LP); h->set_lp_termination_status(cuopt::remote::PDLP_OPTIMAL); h->set_primal_objective(-464.753); auto* arr = h->add_arrays(); @@ -1124,7 +1125,7 @@ TEST_F(GrpcClientTest, SolveLP_SuccessWithPolling) cuopt::remote::LPSolution solution; solution.add_primal_solution(1.0); solution.set_primal_objective(1.0); - solution.set_termination_status(cuopt::remote::PDLP_OPTIMAL); + solution.set_lp_termination_status(cuopt::remote::PDLP_OPTIMAL); resp->mutable_lp_solution()->CopyFrom(solution); resp->set_status(cuopt::remote::SUCCESS); return grpc::Status::OK; @@ -1172,7 +1173,7 @@ TEST_F(GrpcClientTest, SolveLP_SuccessWithWait) cuopt::remote::LPSolution solution; solution.add_primal_solution(1.0); solution.set_primal_objective(1.0); - solution.set_termination_status(cuopt::remote::PDLP_OPTIMAL); + solution.set_lp_termination_status(cuopt::remote::PDLP_OPTIMAL); resp->mutable_lp_solution()->CopyFrom(solution); resp->set_status(cuopt::remote::SUCCESS); return grpc::Status::OK; @@ -1305,9 +1306,9 @@ TEST_F(GrpcClientTest, SolveMIP_Success) const cuopt::remote::GetResultRequest&, cuopt::remote::ResultResponse* resp) { cuopt::remote::MIPSolution solution; - solution.add_solution(1.0); - solution.set_objective(1.0); - solution.set_termination_status(cuopt::remote::MIP_OPTIMAL); + solution.add_mip_solution(1.0); + solution.set_mip_objective(1.0); + solution.set_mip_termination_status(cuopt::remote::MIP_OPTIMAL); resp->mutable_mip_solution()->CopyFrom(solution); resp->set_status(cuopt::remote::SUCCESS); return grpc::Status::OK; @@ -1632,3 +1633,201 @@ TEST_F(GrpcClientTest, SubmitLP_UnaryForSmallPayload) EXPECT_TRUE(result.success) << "Error: " << result.error_message; EXPECT_EQ(result.job_id, "unary-lp-001"); } + +// ============================================================================= +// Settings mapper round-trip tests (no mock stub / server required) +// ============================================================================= + +TEST(SettingsMapperTest, PDLPSettingsRoundTrip_AllFields) +{ + pdlp_solver_settings_t original; + + // Tolerances + original.tolerances.absolute_gap_tolerance = 1e-5; + original.tolerances.relative_gap_tolerance = 1e-6; + original.tolerances.primal_infeasible_tolerance = 1e-11; + original.tolerances.dual_infeasible_tolerance = 1e-12; + original.tolerances.absolute_dual_tolerance = 2e-4; + original.tolerances.relative_dual_tolerance = 3e-4; + original.tolerances.absolute_primal_tolerance = 4e-4; + original.tolerances.relative_primal_tolerance = 5e-4; + + // Limits + original.time_limit = 99.5; + original.iteration_limit = 50000; + + // Solver configuration + original.log_to_console = false; + original.detect_infeasibility = true; + original.strict_infeasibility = true; + original.pdlp_solver_mode = pdlp_solver_mode_t::Fast1; + original.method = method_t::Barrier; + original.presolver = presolver_t::Papilo; + original.dual_postsolve = false; + original.crossover = true; + original.num_gpus = 4; + + original.per_constraint_residual = true; + original.cudss_deterministic = true; + original.folding = 2; + original.augmented = 1; + original.dualize = 0; + original.ordering = 3; + original.barrier_dual_initial_point = 1; + original.eliminate_dense_columns = false; + original.save_best_primal_so_far = true; + original.first_primal_feasible = true; + original.pdlp_precision = pdlp_precision_t::MixedPrecision; + + cuopt::remote::PDLPSolverSettings proto; + map_pdlp_settings_to_proto(original, &proto); + + pdlp_solver_settings_t restored; + map_proto_to_pdlp_settings(proto, restored); + + // Tolerances + EXPECT_DOUBLE_EQ(restored.tolerances.absolute_gap_tolerance, + original.tolerances.absolute_gap_tolerance); + EXPECT_DOUBLE_EQ(restored.tolerances.relative_gap_tolerance, + original.tolerances.relative_gap_tolerance); + EXPECT_DOUBLE_EQ(restored.tolerances.primal_infeasible_tolerance, + original.tolerances.primal_infeasible_tolerance); + EXPECT_DOUBLE_EQ(restored.tolerances.dual_infeasible_tolerance, + original.tolerances.dual_infeasible_tolerance); + EXPECT_DOUBLE_EQ(restored.tolerances.absolute_dual_tolerance, + original.tolerances.absolute_dual_tolerance); + EXPECT_DOUBLE_EQ(restored.tolerances.relative_dual_tolerance, + original.tolerances.relative_dual_tolerance); + EXPECT_DOUBLE_EQ(restored.tolerances.absolute_primal_tolerance, + original.tolerances.absolute_primal_tolerance); + EXPECT_DOUBLE_EQ(restored.tolerances.relative_primal_tolerance, + original.tolerances.relative_primal_tolerance); + + // Limits + EXPECT_DOUBLE_EQ(restored.time_limit, original.time_limit); + EXPECT_EQ(restored.iteration_limit, original.iteration_limit); + + // Solver configuration + EXPECT_EQ(restored.log_to_console, original.log_to_console); + EXPECT_EQ(restored.detect_infeasibility, original.detect_infeasibility); + EXPECT_EQ(restored.strict_infeasibility, original.strict_infeasibility); + EXPECT_EQ(restored.pdlp_solver_mode, original.pdlp_solver_mode); + EXPECT_EQ(restored.method, original.method); + EXPECT_EQ(static_cast(restored.presolver), static_cast(original.presolver)); + EXPECT_EQ(restored.dual_postsolve, original.dual_postsolve); + EXPECT_EQ(restored.crossover, original.crossover); + EXPECT_EQ(restored.num_gpus, original.num_gpus); + + EXPECT_EQ(restored.per_constraint_residual, original.per_constraint_residual); + EXPECT_EQ(restored.cudss_deterministic, original.cudss_deterministic); + EXPECT_EQ(restored.folding, original.folding); + EXPECT_EQ(restored.augmented, original.augmented); + EXPECT_EQ(restored.dualize, original.dualize); + EXPECT_EQ(restored.ordering, original.ordering); + EXPECT_EQ(restored.barrier_dual_initial_point, original.barrier_dual_initial_point); + EXPECT_EQ(restored.eliminate_dense_columns, original.eliminate_dense_columns); + EXPECT_EQ(restored.save_best_primal_so_far, original.save_best_primal_so_far); + EXPECT_EQ(restored.first_primal_feasible, original.first_primal_feasible); + EXPECT_EQ(static_cast(restored.pdlp_precision), static_cast(original.pdlp_precision)); +} + +TEST(SettingsMapperTest, PDLPSettings_IterationLimitSentinel) +{ + pdlp_solver_settings_t original; + original.iteration_limit = std::numeric_limits::max(); + + cuopt::remote::PDLPSolverSettings proto; + map_pdlp_settings_to_proto(original, &proto); + EXPECT_EQ(proto.iteration_limit(), -1) << "max() should serialize as -1 sentinel"; + + pdlp_solver_settings_t restored; + restored.iteration_limit = 999; + map_proto_to_pdlp_settings(proto, restored); + EXPECT_EQ(restored.iteration_limit, 999) + << "Sentinel -1 should leave iteration_limit unchanged (default)"; +} + +TEST(SettingsMapperTest, MIPSettingsRoundTrip_AllFields) +{ + mip_solver_settings_t original; + original.time_limit = 42.5; + original.tolerances.relative_mip_gap = 0.01; + original.tolerances.absolute_mip_gap = 1e-8; + original.tolerances.integrality_tolerance = 1e-6; + original.tolerances.absolute_tolerance = 1e-7; + original.tolerances.relative_tolerance = 1e-13; + original.tolerances.presolve_absolute_tolerance = 1e-5; + original.log_to_console = false; + original.heuristics_only = true; + original.num_cpu_threads = 8; + original.num_gpus = 2; + original.mip_scaling = true; + + original.work_limit = 100.0; + original.node_limit = 5000; + original.reliability_branching = 3; + original.mip_batch_pdlp_strong_branching = 1; + original.max_cut_passes = 20; + original.mir_cuts = 2; + original.mixed_integer_gomory_cuts = 1; + original.knapsack_cuts = 0; + original.clique_cuts = 3; + original.strong_chvatal_gomory_cuts = -1; + original.reduced_cost_strengthening = 1; + original.cut_change_threshold = 0.05; + original.cut_min_orthogonality = 0.3; + original.determinism_mode = 1; + original.seed = 12345; + + cuopt::remote::MIPSolverSettings proto; + map_mip_settings_to_proto(original, &proto); + + mip_solver_settings_t restored; + map_proto_to_mip_settings(proto, restored); + + EXPECT_DOUBLE_EQ(restored.time_limit, original.time_limit); + EXPECT_DOUBLE_EQ(restored.tolerances.relative_mip_gap, original.tolerances.relative_mip_gap); + EXPECT_DOUBLE_EQ(restored.tolerances.absolute_mip_gap, original.tolerances.absolute_mip_gap); + EXPECT_DOUBLE_EQ(restored.tolerances.integrality_tolerance, + original.tolerances.integrality_tolerance); + EXPECT_DOUBLE_EQ(restored.tolerances.absolute_tolerance, original.tolerances.absolute_tolerance); + EXPECT_DOUBLE_EQ(restored.tolerances.relative_tolerance, original.tolerances.relative_tolerance); + EXPECT_DOUBLE_EQ(restored.tolerances.presolve_absolute_tolerance, + original.tolerances.presolve_absolute_tolerance); + EXPECT_EQ(restored.log_to_console, original.log_to_console); + EXPECT_EQ(restored.heuristics_only, original.heuristics_only); + EXPECT_EQ(restored.num_cpu_threads, original.num_cpu_threads); + EXPECT_EQ(restored.num_gpus, original.num_gpus); + EXPECT_EQ(restored.mip_scaling, original.mip_scaling); + + EXPECT_DOUBLE_EQ(restored.work_limit, original.work_limit); + EXPECT_EQ(restored.node_limit, original.node_limit); + EXPECT_EQ(restored.reliability_branching, original.reliability_branching); + EXPECT_EQ(restored.mip_batch_pdlp_strong_branching, original.mip_batch_pdlp_strong_branching); + EXPECT_EQ(restored.max_cut_passes, original.max_cut_passes); + EXPECT_EQ(restored.mir_cuts, original.mir_cuts); + EXPECT_EQ(restored.mixed_integer_gomory_cuts, original.mixed_integer_gomory_cuts); + EXPECT_EQ(restored.knapsack_cuts, original.knapsack_cuts); + EXPECT_EQ(restored.clique_cuts, original.clique_cuts); + EXPECT_EQ(restored.strong_chvatal_gomory_cuts, original.strong_chvatal_gomory_cuts); + EXPECT_EQ(restored.reduced_cost_strengthening, original.reduced_cost_strengthening); + EXPECT_DOUBLE_EQ(restored.cut_change_threshold, original.cut_change_threshold); + EXPECT_DOUBLE_EQ(restored.cut_min_orthogonality, original.cut_min_orthogonality); + EXPECT_EQ(restored.determinism_mode, original.determinism_mode); + EXPECT_EQ(restored.seed, original.seed); +} + +TEST(SettingsMapperTest, MIPSettings_NodeLimitSentinel) +{ + mip_solver_settings_t original; + original.node_limit = std::numeric_limits::max(); + + cuopt::remote::MIPSolverSettings proto; + map_mip_settings_to_proto(original, &proto); + EXPECT_EQ(proto.node_limit(), -1) << "max() should serialize as -1 sentinel"; + + mip_solver_settings_t restored; + restored.node_limit = 999; + map_proto_to_mip_settings(proto, restored); + EXPECT_EQ(restored.node_limit, 999) << "Sentinel -1 should leave node_limit unchanged (default)"; +} diff --git a/cpp/tests/linear_programming/grpc/grpc_pipe_serialization_test.cpp b/cpp/tests/linear_programming/grpc/grpc_pipe_serialization_test.cpp index cf237d611..d84574680 100644 --- a/cpp/tests/linear_programming/grpc/grpc_pipe_serialization_test.cpp +++ b/cpp/tests/linear_programming/grpc/grpc_pipe_serialization_test.cpp @@ -294,7 +294,7 @@ TEST(PipeSerialization, Result_RoundTrip) PipePair pp; ChunkedResultHeader header; - header.set_is_mip(false); + header.set_problem_category(LP); header.set_lp_termination_status(PDLP_OPTIMAL); header.set_primal_objective(42.5); header.set_solve_time(1.23); @@ -319,7 +319,7 @@ TEST(PipeSerialization, Result_RoundTrip) ASSERT_TRUE(write_ok); ASSERT_TRUE(read_ok); - EXPECT_FALSE(header_out.is_mip()); + EXPECT_EQ(header_out.problem_category(), LP); EXPECT_EQ(header_out.lp_termination_status(), PDLP_OPTIMAL); EXPECT_DOUBLE_EQ(header_out.primal_objective(), 42.5); EXPECT_DOUBLE_EQ(header_out.solve_time(), 1.23); @@ -334,7 +334,7 @@ TEST(PipeSerialization, Result_MIPFields) PipePair pp; ChunkedResultHeader header; - header.set_is_mip(true); + header.set_problem_category(MIP); header.set_mip_termination_status(MIP_OPTIMAL); header.set_mip_objective(99.0); header.set_mip_gap(0.001); @@ -356,7 +356,7 @@ TEST(PipeSerialization, Result_MIPFields) ASSERT_TRUE(write_ok); ASSERT_TRUE(read_ok); - EXPECT_TRUE(header_out.is_mip()); + EXPECT_EQ(header_out.problem_category(), MIP); EXPECT_EQ(header_out.mip_termination_status(), MIP_OPTIMAL); EXPECT_DOUBLE_EQ(header_out.mip_objective(), 99.0); @@ -369,7 +369,7 @@ TEST(PipeSerialization, Result_EmptyArrays) PipePair pp; ChunkedResultHeader header; - header.set_is_mip(false); + header.set_problem_category(LP); header.set_error_message("solver failed"); std::map> arrays; // no arrays (error case) @@ -398,7 +398,7 @@ TEST(PipeSerialization, ProtobufRoundTrip) PipePair pp; ChunkedResultHeader msg; - msg.set_is_mip(true); + msg.set_problem_category(MIP); msg.set_primal_objective(3.14); msg.set_error_message("hello"); @@ -412,7 +412,7 @@ TEST(PipeSerialization, ProtobufRoundTrip) ASSERT_TRUE(write_ok); ASSERT_TRUE(read_ok); - EXPECT_TRUE(msg_out.is_mip()); + EXPECT_EQ(msg_out.problem_category(), MIP); EXPECT_DOUBLE_EQ(msg_out.primal_objective(), 3.14); EXPECT_EQ(msg_out.error_message(), "hello"); } @@ -426,7 +426,7 @@ TEST(PipeSerialization, Result_LargeArray) PipePair pp; ChunkedResultHeader header; - header.set_is_mip(false); + header.set_problem_category(LP); header.set_primal_objective(0.0); // ~4 MiB array — large enough to require many kernel-level pipe iterations. diff --git a/dependencies.yaml b/dependencies.yaml index 84dc5eed0..b5d2db57c 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -299,6 +299,7 @@ dependencies: packages: - libboost-devel - cpp-argparse + - pyyaml - tbb-devel - zlib - bzip2 @@ -313,6 +314,7 @@ dependencies: - output_types: [conda] packages: - cuda-sanitizer-api + - pyyaml test_cpp_cuopt: common: - output_types: [conda] @@ -766,6 +768,7 @@ dependencies: - output_types: [conda, requirements] packages: - pre-commit + - ruamel.yaml>=0.18 - output_types: conda packages: - clang==20.1.4 diff --git a/docs/cuopt/grpc/GRPC_CODE_GENERATION.md b/docs/cuopt/grpc/GRPC_CODE_GENERATION.md new file mode 100644 index 000000000..0bb81086a --- /dev/null +++ b/docs/cuopt/grpc/GRPC_CODE_GENERATION.md @@ -0,0 +1,712 @@ +# Code Generation for gRPC Proto Definitions and C++ Conversion Code + +The code generator reads `field_registry.yaml` and produces `cuopt_remote_data.proto` +and C++ `.inc` files that are `#include`d directly into mapper source files. This +eliminates the need to hand-write repetitive conversion code or `.proto` definitions +— adding or removing a field is a one-line YAML change. + +## Quick Start + +```bash +# Regenerate after editing field_registry.yaml +python cpp/codegen/generate_conversions.py + +# Or with explicit paths: +python cpp/codegen/generate_conversions.py \ + --registry cpp/codegen/field_registry.yaml \ + --output-dir cpp/codegen/generated +``` + +The generator runs in ~100ms with no external dependencies beyond PyYAML (ships +with conda). The `--auto-number` and `--strip` options additionally require +`ruamel.yaml` (listed in the project's development dependencies). It runs +automatically before every `libcuopt` or `cuopt_grpc_server` build via +`build.sh`. + +## File Layout + +``` +cpp/codegen/ +├── field_registry.yaml # Source of truth for all fields +├── generate_conversions.py # Generator script +└── generated/ # Output (committed, regenerated on build) + ├── cuopt_remote_data.proto + ├── generated_result_enums.proto.inc + │ + │ # Enum converters (one per domain) + ├── generated_enum_converters_problem.inc + ├── generated_enum_converters_settings.inc + ├── generated_enum_converters_solution.inc + │ + │ # Settings conversions + ├── generated_pdlp_settings_to_proto.inc + ├── generated_proto_to_pdlp_settings.inc + ├── generated_mip_settings_to_proto.inc + ├── generated_proto_to_mip_settings.inc + │ + │ # LP solution conversions + ├── generated_lp_solution_to_proto.inc + ├── generated_proto_to_lp_solution.inc + ├── generated_lp_chunked_header.inc + ├── generated_collect_lp_arrays.inc + ├── generated_chunked_to_lp_solution.inc + ├── generated_estimate_lp_size.inc + │ + │ # MIP solution conversions + ├── generated_mip_solution_to_proto.inc + ├── generated_proto_to_mip_solution.inc + ├── generated_mip_chunked_header.inc + ├── generated_collect_mip_arrays.inc + ├── generated_chunked_to_mip_solution.inc + ├── generated_estimate_mip_size.inc + │ + │ # Problem conversions + ├── generated_problem_to_proto.inc + ├── generated_proto_to_problem.inc + ├── generated_estimate_problem_size.inc + ├── generated_populate_chunked_header_lp.inc + ├── generated_populate_chunked_header_mip.inc + ├── generated_chunked_header_to_problem.inc + ├── generated_chunked_arrays_to_problem.inc + ├── generated_build_array_chunks.inc + └── generated_array_field_element_size.inc +``` + +The generated `.inc` files are committed to the repo so that builds work without +running the generator. `build.sh` re-generates them before every `libcuopt` or +`cuopt_grpc_server` build to keep them in sync. CMake adds +`cpp/codegen/generated` to the include path for both targets, so the `.inc` +files are found at compile time with no copy step. + +--- + +## Registry Structure Overview + +`field_registry.yaml` has these top-level sections: + +| Section | Purpose | +|---|---| +| `enums` | Shared enum definitions (C++ ↔ proto converters) | +| `lp_solution` | LP solution scalar/array fields and constructor args | +| `mip_solution` | MIP solution scalar/array fields and constructor args | +| `pdlp_settings` | PDLP solver settings field mappings | +| `mip_settings` | MIP solver settings field mappings | +| `optimization_problem` | Problem input scalar/array fields and setter groups | + +--- + +## Convention-Over-Configuration Defaults + +The registry uses bare field names wherever possible. Defaults: + +| Context | Default type | Default getter | Default setter | +|---|---|---|---| +| Solution scalar | `double` | `get_()` | *(via constructor)* | +| Solution array | `double` | `get__host()` | *(via constructor)* | +| Problem scalar | `double` | `get_()` | `set_()` | +| Problem array | `repeated double` | `get__host()` | `set_()` | +| Settings field | `double` | struct member access | struct member assignment | + +Proto field names always match the registry field name. + +**Enum conventions** (derived from the YAML key unless overridden): + +| Property | Convention | +|---|---| +| `cpp_type` | `_t` (e.g. `pdlp_termination_status` → `pdlp_termination_status_t`) | +| `proto_type` | PascalCase from key, with known acronyms uppercased: PDLP, MIP, LP, QP, VRP, PDP, TSP (e.g. `pdlp_termination_status` → `PDLPTerminationStatus`) | +| `default` | First value in the `values` list (the proto3 zero-value) | +| `values` numbering | 0, 1, 2, ... (bare names auto-number; `{Name: N}` resets counter to N) | +| converter fns | `to_proto_()` / `from_proto_()` | + +--- + +## Enums + +Each entry under `enums:` defines a C++ ↔ proto enum mapping. The generator +produces `to_proto_()` and `from_proto_()` switch functions, split +into per-domain `.inc` files (`generated_enum_converters_problem.inc`, etc.). + +Most properties are derived by convention (see above). Only `domain` and +`values` are required for the common case. Values auto-number from 0 when +written as bare names: + +```yaml +enums: + # Minimal — bare names auto-number 0, 1, 2, ... + # proto_type, cpp_type, and default are all derived. + pdlp_termination_status: + domain: solution # groups into generated_enum_converters_solution.inc + proto_prefix: PDLP # proto value = PDLP_UPPER_SNAKE(CppName) + values: + - NoTermination # = 0 + - NumericalError # = 1 + - Optimal # = 2 + - PrimalInfeasible # = 3 + + # Override default when it's not the first value + pdlp_solver_mode: + domain: settings + default: Stable3 + values: + - Stable1 + - Stable2 + - Methodical1 + - Fast1 + - Stable3 + + # Override cpp_type when it doesn't follow _t + lp_method: + domain: settings + cpp_type: method_t + values: + - Concurrent + - PDLP + - DualSimplex + - Barrier + + # Explicit values reset the counter (C-style enum semantics): + # example_with_gaps: + # domain: solution + # values: + # - OK # = 0 + # - Warning: 10 # explicit → resets counter + # - Error # = 11 (continues from 10+1) + # - Fatal: 20 # explicit → resets counter + # - Panic # = 21 +``` + +### Enum properties + +| Property | Required | Default | Description | +|---|---|---|---| +| `domain` | yes | — | One of `problem`, `settings`, `solution`. Controls which `.inc` file the converters go into | +| `proto_type` | no | PascalCase from key | Protobuf enum type name. Derived via acronym-aware PascalCase (e.g. `pdlp_termination_status` → `PDLPTerminationStatus`). Override when the derived name doesn't match | +| `proto_prefix` | no | *(none)* | Prefix for proto value names. With prefix `PDLP`, C++ `Optimal` becomes proto `PDLP_OPTIMAL` | +| `cpp_type` | no | `_t` | C++ enum type (e.g. `pdlp_termination_status_t`). Override with `cpp_type: method_t` | +| `default` | no | first value | C++ enum value to return for unrecognized proto values. Defaults to the first entry in `values` (the proto3 zero-value). Override when the default differs (e.g. `pdlp_solver_mode` defaults to `Stable3`) | +| `values` | yes | — | List of enum entries. Bare names auto-number from 0. Use `{Name: N}` to override; subsequent bare names continue from N+1 (C-style enum semantics) | + +--- + +## LP / MIP Solution Sections + +Each solution section (`lp_solution`, `mip_solution`) generates six `.inc` files +covering unary proto conversion, chunked streaming, size estimation, and array +collection. + +```yaml +lp_solution: + cpp_type: "cpu_lp_solution_t" + + scalars: [...] + arrays: [...] + constructor_args: { scalars: [...] } + warm_start: { ... } # LP only +``` + +### Top-level properties + +| Property | Description | +|---|---| +| `cpp_type` | Fully-qualified C++ template type for the solution constructor | + +The generator derives `ChunkedResultHeader.problem_category` (LP or MIP) +automatically from the section name (`lp_solution` vs `mip_solution`). + +### Scalars + +Each entry describes one scalar field on both `ChunkedResultHeader` (for +chunked streaming) and the unary solution proto. + +```yaml +scalars: + # Minimal form — double type, getter = get_gap() + - gap: + field_num: 1006 + + # Enum type + - lp_termination_status: + field_num: 1000 + type: pdlp_termination_status # references an enum key + getter: get_termination_status() + + # Proto-only (set on header but NOT a constructor arg) + - error_message: + field_num: 1001 + type: string + getter: "get_error_status().what()" + proto_only: true +``` + +| Property | Default | Description | +|---|---|---| +| `field_num` | *(required)* | Proto field number in `ChunkedResultHeader`. LP: 1000–1999, MIP: 2000–2999, warm start: 3000–3999 | +| `type` | `double` | One of: `double`, `int32`, `bool`, `string`, or an enum key name | +| `getter` | `get_()` | C++ expression to read from the solution object | +| `proto_only` | `false` | If true, set on the proto header but not passed to the C++ constructor | + +#### Type-specific behavior + +| `type` | To-proto cast | From-proto cast | +|---|---|---| +| `double` | `static_cast(...)` | `static_cast(...)` | +| `int32` | `static_cast(...)` | `static_cast(...)` | +| `bool` | *(none)* | *(none)* | +| `string` | *(none)* | *(none)* | +| enum key | `to_proto_(...)` | `from_proto_(...)` | + +### Arrays + +Each entry describes a solution array, identified by a `ResultFieldId` enum value. + +```yaml +arrays: + - primal_solution: + field_num: 1 + array_id: 0 + + - mip_solution: + array_id: 12 + field_num: 1 + getter: get_solution_host() # override the default +``` + +| Property | Default | Description | +|---|---|---| +| `field_num` | *(required)* | Proto field number in the per-solution unary message | +| `array_id` | *(required)* | Numeric value for the `ResultFieldId` enum (global across LP + MIP) | +| `getter` | `get__host()` | C++ getter expression on the solution object | + +### Constructor Args + +Controls the positional argument order when reconstructing a C++ solution +object from proto data: + +```yaml +constructor_args: + scalars: + - lp_termination_status + - primal_objective + - dual_objective + # ... order must match the C++ constructor +``` + +Arrays are always passed first (in YAML declaration order) via `std::move`. +Then the scalars listed here. If a `warm_start` section exists and warm start +data is present, `std::move(ws)` is appended as the final argument. + +### Warm Start (LP only) + +Describes a conditional sub-object for PDLP warm start data: + +```yaml +warm_start: + presence_check: has_warm_start_data() # predicate on the solution object + getter: get_cpu_pdlp_warm_start_data() # accessor for the WS struct + + scalars: + - initial_primal_weight_: + field_num: 3000 + + arrays: + - current_primal_solution_: + field_num: 1 + array_id: 3 +``` + +| Property | Description | +|---|---| +| `presence_check` | C++ predicate expression to test if warm start data is present on the solution object | +| `getter` | C++ expression to access the warm start struct | + +Warm start field names match the C++ struct member names directly (e.g. +`initial_primal_weight_` maps to `ws.initial_primal_weight_`). The `member` +attribute is only needed if the proto field name cannot match the C++ name +due to ambiguity. + +Warm start detection during chunked deserialization is auto-derived: if the +first array in the warm start section is present (non-empty), warm start data +is considered present. + +--- + +## Settings Sections + +Each settings section (`pdlp_settings`, `mip_settings`) generates two `.inc` +files: `generated_{label}_settings_to_proto.inc` and +`generated_proto_to_{label}_settings.inc`. + +```yaml +pdlp_settings: + cpp_type: "pdlp_solver_settings_t" + proto_type: "cuopt::remote::PDLPSolverSettings" + + fields: + # Nested sub-struct — generates settings.tolerances. + - tolerances: + - absolute_gap_tolerance: + field_num: 1 + - relative_gap_tolerance: + field_num: 2 + + # Top-level fields + - time_limit: + field_num: 9 + - iteration_limit: + field_num: 10 + type: int64 + sentinel: + to_proto: "std::numeric_limits::max()" + proto_value: -1 + from_proto_guard: ">= 0" + from_proto_cast: "i_t" +``` + +Settings fields support **nesting**: a list-valued entry (like `tolerances` +above) represents a sub-struct. The generator automatically prefixes C++ member +access with the sub-struct path (e.g. `settings.tolerances.absolute_gap_tolerance`). + +### Settings field properties + +| Property | Default | Description | +|---|---|---| +| `field_num` | *(required)* | Proto field number in the settings message | +| `type` | `double` | One of: `double`, `int32`, `int64`, `bool`, `string`, or an enum key | +| `cpp_member` | `` (auto-prefixed by nesting path) | Explicit path to the C++ struct member | +| `to_proto_cast` | *(none)* | Explicit cast for C++ → proto (e.g. `int32_t`) | +| `from_proto_cast` | *(none)* | Explicit cast for proto → C++ (e.g. `presolver_t`) | +| `sentinel` | *(none)* | Special handling for sentinel values (see below) | + +### Sentinel values + +Some fields map a C++ default (like `max()`) to a proto sentinel (like `-1`): + +```yaml +- iteration_limit: + type: int64 + sentinel: + to_proto: "std::numeric_limits::max()" # if C++ value == this... + proto_value: -1 # ...emit this in proto + from_proto_guard: ">= 0" # only assign if proto value matches guard + from_proto_cast: "i_t" # cast applied when assigning +``` + +--- + +## Optimization Problem Section + +The `optimization_problem` section generates the most files — problem +serialization/deserialization for both unary and chunked gRPC paths. + +```yaml +optimization_problem: + cpp_type: "cpu_optimization_problem_t" + proto_message: OptimizationProblem + + scalars: [...] + arrays: [...] + setter_groups: { ... } +``` + +### Scalars + +```yaml +scalars: +- problem_name: + field_num: 1 + type: string +- maximize: + field_num: 3 + type: bool + getter: get_sense() +- problem_category: + field_num: 6 + type: problem_category # enum key reference +``` + +| Property | Default | Description | +|---|---|---| +| `field_num` | *(required)* | Proto field number in `OptimizationProblem` | +| `type` | `double` | One of: `double`, `int32`, `bool`, `string`, or an enum key | +| `getter` | `get_()` | C++ getter on `cpu_optimization_problem_t` | +| `setter` | `set_()` | C++ setter (can be overridden via `setter_getter_root`) | +| `setter_getter_root` | `` | Base name for default getter/setter derivation | + +### Arrays + +```yaml +arrays: +- variable_names: + array_id: 0 + field_num: 7 + type: repeated string + +- A_values: + array_id: 2 + field_num: 9 + setter_getter_root: constraint_matrix_values + setter_group: csr_constraint_matrix + +- variable_types: + array_id: 12 + field_num: 19 + type: repeated variable_type # repeated enum + +- row_types: + array_id: 11 + field_num: 18 + type: bytes + conditional: true +``` + +| Property | Default | Description | +|---|---|---| +| `field_num` | *(required)* | Proto field number in `OptimizationProblem` | +| `array_id` | *(required)* | Numeric value for the `ArrayFieldId` enum | +| `type` | `repeated double` | One of: `repeated double`, `repeated int32`, `repeated string`, `bytes`, or `repeated ` | +| `getter` | `get__host()` | C++ getter (strings use `get_()` without `_host`) | +| `setter` | `set_()` | C++ setter | +| `setter_getter_root` | `` | Base name for getter/setter derivation when different from field name | +| `setter_group` | *(none)* | Name of a multi-argument setter group (see below) | +| `conditional` | `false` | If true, serialization is guarded by an emptiness check | +| `skip_conversion` | `false` | If true, the field appears in the proto but is excluded from conversion code | + +### Setter Groups + +Some C++ setters take multiple arrays at once (e.g. CSR matrix = values + +indices + offsets). Setter groups handle this: + +```yaml +setter_groups: + csr_constraint_matrix: + setter: set_csr_constraint_matrix + fields: [A_values, A_indices, A_offsets] + + quadratic_objective: + setter: set_quadratic_objective_matrix + fields: [Q_values, Q_indices, Q_offsets] +``` + +| Property | Description | +|---|---| +| `setter` | C++ setter function name (called with all field arrays as arguments) | +| `fields` | Ordered list of array field names that are passed to the setter | + +Arrays that belong to a setter group are excluded from normal per-field +deserialization and handled as a batch instead. + +During deserialization, the generator automatically guards setter group calls +by checking if the first field has data (e.g. `if (pb_problem.a_values_size() > 0)`). +This is derived from the group structure — no explicit condition attribute is needed. + +--- + +## Field Number Allocation + +Field numbers are required for proto compatibility. They can be assigned +manually or auto-assigned. + +### Manual assignment + +Specify `field_num` and `array_id` on each field entry. This is the default +workflow. + +### Auto-assignment + +Run with `--auto-number` to fill in any missing `field_num` or `array_id` +values. This requires `ruamel.yaml` (preserves YAML comments and formatting): + +```bash +python cpp/codegen/generate_conversions.py --auto-number +``` + +### Stripping field numbers + +Run with `--strip` to remove all `field_num` and `array_id` values from +`field_registry.yaml`. This is useful for reviewing pure field definitions +without numbering clutter, or for forcing a full re-assignment via +`--auto-number`: + +```bash +python cpp/codegen/generate_conversions.py --strip +python cpp/codegen/generate_conversions.py --auto-number +``` + +If all field numbers have been stripped, running the generator without +`--auto-number` will produce an error. + +The numbering ranges: + +| Scope | Range | +|---|---| +| `optimization_problem` field_num | 1+ (contiguous, shared across scalars and arrays) | +| `optimization_problem` array_id | 0+ (separate namespace) | +| LP solution scalars (ChunkedResultHeader) | 1000–1999 | +| MIP solution scalars (ChunkedResultHeader) | 2000–2999 | +| Warm start scalars (ChunkedResultHeader) | 3000–3999 | +| Solution array array_id | 0+ (global pool shared across LP, MIP, and warm start) | +| Solution array field_num | 1+ per message (no cap) | +| Settings field_num | 1+ per message (no cap) | + +--- + +## What Gets Generated + +### `cuopt_remote_data.proto` + +A complete proto file with all data messages and enums derived from the +registry: `OptimizationProblem`, `PDLPSolverSettings`, `MIPSolverSettings`, +`PDLPWarmStartData`, `LPSolution`, `MIPSolution`, `ChunkedResultHeader`, +`ResultArrayDescriptor`, and the enums they reference (`PDLPTerminationStatus`, +`MIPTerminationStatus`, `PDLPSolverMode`, `LPMethod`, `VariableType`, +`ProblemCategory`, `ResultFieldId`, `ArrayFieldId`). + +The hand-maintained `cuopt_remote.proto` and `cuopt_remote_service.proto` can +import this generated file to avoid duplicating definitions. + +### Enum converter `.inc` files + +Per-domain C++ switch functions, split by the `domain` tag on each enum: + +- `generated_enum_converters_problem.inc` — enums with `domain: problem` +- `generated_enum_converters_settings.inc` — enums with `domain: settings` +- `generated_enum_converters_solution.inc` — enums with `domain: solution` + +### Settings `.inc` files + +- `generated_pdlp_settings_to_proto.inc` / `generated_proto_to_pdlp_settings.inc` +- `generated_mip_settings_to_proto.inc` / `generated_proto_to_mip_settings.inc` + +### Solution `.inc` files (6 per solution type) + +For each of LP and MIP: + +| File | Function body it provides | +|---|---| +| `generated_{lp,mip}_solution_to_proto.inc` | Unary C++ solution → proto | +| `generated_proto_to_{lp,mip}_solution.inc` | Unary proto → C++ solution | +| `generated_{lp,mip}_chunked_header.inc` | Populate `ChunkedResultHeader` | +| `generated_collect_{lp,mip}_arrays.inc` | Collect solution arrays as byte maps | +| `generated_chunked_to_{lp,mip}_solution.inc` | Reassemble C++ solution from chunked data | +| `generated_estimate_{lp,mip}_size.inc` | Estimate serialized proto size | + +### Problem `.inc` files + +| File | Function body it provides | +|---|---| +| `generated_problem_to_proto.inc` | C++ problem → unary proto | +| `generated_proto_to_problem.inc` | Unary proto → C++ problem | +| `generated_estimate_problem_size.inc` | Estimate serialized problem size | +| `generated_populate_chunked_header_{lp,mip}.inc` | Populate chunked problem header | +| `generated_chunked_header_to_problem.inc` | Set problem scalars from chunked header | +| `generated_chunked_arrays_to_problem.inc` | Set problem arrays from chunked byte maps | +| `generated_build_array_chunks.inc` | Build `SendArrayChunkRequest` list for upload | +| `generated_array_field_element_size.inc` | Switch body for per-field element byte size | + +--- + +## How `.inc` Files Are Consumed + +The `.inc` files are `#include`d directly inside C++ function bodies in: + +- `cpp/src/grpc/grpc_settings_mapper.cpp` +- `cpp/src/grpc/grpc_solution_mapper.cpp` +- `cpp/src/grpc/grpc_problem_mapper.cpp` +- `cpp/src/grpc/server/grpc_field_element_size.hpp` + +CMake adds `cpp/codegen/generated` to the include path for the `cuopt` and +`cuopt_grpc_server` targets, so the bare `#include "generated_*.inc"` directives +resolve without any copy step. + +--- + +## Adding a New Field — Walkthroughs + +### Add `dual_bound` (double) to MIP solution + +1. Add to `mip_solution.scalars`: + ```yaml + - dual_bound: + field_num: 2012 + ``` + +2. Add to `mip_solution.constructor_args.scalars` in the correct position: + ```yaml + constructor_args: + scalars: + - mip_termination_status + - mip_objective + # ... + - dual_bound # ← new, position must match C++ constructor + ``` + +3. Add the constructor parameter to `cpu_mip_solution_t`. + +4. Regenerate: + ```bash + python cpp/codegen/generate_conversions.py + ``` + +5. Build and test. + +The proto field in `ChunkedResultHeader` and the solution message are generated +automatically — no manual `.proto` edits needed. + +### Add `detect_infeasibility_v2` (bool) to PDLP settings + +1. Add to `pdlp_settings.fields`: + ```yaml + - detect_infeasibility_v2: + field_num: 31 + type: bool + ``` + +2. Add the C++ struct member to `pdlp_solver_settings_t`: + ```cpp + bool detect_infeasibility_v2{false}; + ``` + +3. Regenerate and build. You never touch `grpc_settings_mapper.cpp` or any + `.proto` file — the proto field in `PDLPSolverSettings` is generated + automatically. + +### Add a new array to the optimization problem + +1. Add to `optimization_problem.arrays`: + ```yaml + - my_new_array: + array_id: 18 + field_num: 25 + type: repeated double + ``` + +2. Add getter/setter to `cpu_optimization_problem_t`. + +3. Regenerate and build. + +The `ArrayFieldId` enum entry and the `OptimizationProblem` proto field are +generated automatically. + +### Add a tolerance to MIP settings (nested sub-struct) + +If the C++ member is nested under `tolerances.`, just add it inside the +`tolerances` list: + +```yaml +- tolerances: + - relative_mip_gap: + field_num: 2 + - my_new_tolerance: # ← new + field_num: 14 +``` + +The generator will access it as `settings.tolerances.my_new_tolerance`. + +--- + +## Related Documentation + +- `GRPC_INTERFACE.md` — Chunked transfer protocol, message size limits, error handling. +- `GRPC_SERVER_ARCHITECTURE.md` — Server process model, IPC, threads, job lifecycle. +- `GRPC_QUICK_START.md` — Starting the server and solving remotely from Python, CLI, or C. diff --git a/docs/cuopt/grpc/GRPC_INTERFACE.md b/docs/cuopt/grpc/GRPC_INTERFACE.md new file mode 100644 index 000000000..ae28bc69c --- /dev/null +++ b/docs/cuopt/grpc/GRPC_INTERFACE.md @@ -0,0 +1,133 @@ +# gRPC Chunked Transfer Protocol + +## Overview + +The cuOpt remote execution system uses gRPC for client-server communication. The interface +supports arbitrarily large optimization problems (multi-GB) through a chunked array transfer +protocol that uses only unary (request-response) RPCs — no bidirectional streaming. + +## Chunked Array Transfer Protocol + +### Why Chunking? + +gRPC has per-message size limits (configurable, default set to 256 MiB in cuOpt), and +protobuf has a hard 2 GB serialization limit. Optimization problems and their solutions +can exceed several gigabytes, so a chunked transfer mechanism is needed. + +The protocol uses only **unary RPCs** (no bidirectional streaming), which simplifies +error handling, load balancing, and proxy compatibility. + +### Upload Protocol (Large Problems) + +When the estimated serialized problem size exceeds 75% of `max_message_bytes`, the client +splits large arrays into chunks and sends them via multiple unary RPCs: + +``` +Client Server + | | + |-- StartChunkedUpload(header, settings) -----> | + |<-- upload_id, max_message_bytes -------------- | + | | + |-- SendArrayChunk(upload_id, field, data) ----> | + |<-- ok ---------------------------------------- | + | | + |-- SendArrayChunk(upload_id, field, data) ----> | + |<-- ok ---------------------------------------- | + | ... | + | | + |-- FinishChunkedUpload(upload_id) ------------> | + |<-- job_id ------------------------------------ | +``` + +**Key features:** +- `StartChunkedUpload` sends a `ChunkedProblemHeader` with all scalar fields and + array metadata (`ArrayDescriptor` for each large array: field ID, total elements, + element size) +- Each `SendArrayChunk` carries one chunk of one array, identified by `ArrayFieldId` + and `element_offset` +- The server reports `max_message_bytes` so the client can adapt chunk sizing +- `FinishChunkedUpload` triggers server-side reassembly and job submission + +### Download Protocol (Large Results) + +When the result exceeds the gRPC max message size, the client fetches it via +chunked unary RPCs (mirrors the upload pattern): + +``` +Client Server + | | + |-- StartChunkedDownload(job_id) --------------> | + |<-- download_id, ChunkedResultHeader ---------- | + | | + |-- GetResultChunk(download_id, field, off) ----> | + |<-- data bytes --------------------------------- | + | | + |-- GetResultChunk(download_id, field, off) ----> | + |<-- data bytes --------------------------------- | + | ... | + | | + |-- FinishChunkedDownload(download_id) ---------> | + |<-- ok ----------------------------------------- | +``` + +**Key features:** +- `ChunkedResultHeader` carries all scalar fields (termination status, objectives, + residuals, solve time, warm start scalars) plus `ResultArrayDescriptor` entries + for each array (solution vectors, warm start arrays) +- Each `GetResultChunk` fetches a slice of one array, identified by `ResultFieldId` + and `element_offset` +- `FinishChunkedDownload` releases the server-side download session state +- LP results include PDLP warm start data (9 arrays + 8 scalars) for subsequent + warm-started solves + +### Automatic Routing + +The client handles size-based routing transparently: + +1. **Upload**: Estimate serialized problem size + - Below 75% of `max_message_bytes` → unary `SubmitJob` + - Above threshold → `StartChunkedUpload` + `SendArrayChunk` + `FinishChunkedUpload` +2. **Download**: Check `result_size_bytes` from `CheckStatus` + - Below `max_message_bytes` → unary `GetResult` + - Above limit (or `RESOURCE_EXHAUSTED`) → chunked download RPCs + +## Message Size Limits + +| Configuration | Default | Notes | +|---------------|---------|-------| +| Server `--max-message-mb` | 256 MiB | Per-message limit (also `--max-message-bytes` for exact byte values) | +| Server clamping | [4 KiB, ~2 GiB] | Enforced at startup to stay within protobuf's serialization limit | +| Client `max_message_bytes` | 256 MiB | Clamped to [4 MiB, ~2 GiB] at construction | +| Chunk size | 16 MiB | Payload per `SendArrayChunk`/`GetResultChunk` | +| Chunked threshold | 75% of max_message_bytes | Problems above this use chunked upload (e.g. 192 MiB when max is 256 MiB) | + +Chunked transfer allows unlimited total payload size; only individual +chunks must fit within the per-message limit. Neither client nor server +allows "unlimited" message size — both clamp to the protobuf 2 GiB ceiling. + +## Error Handling + +### gRPC Status Codes + +| Code | Meaning | Client Action | +|------|---------|---------------| +| `OK` | Success | Process result | +| `NOT_FOUND` | Job ID not found | Check job ID | +| `RESOURCE_EXHAUSTED` | Message too large | Use chunked transfer | +| `CANCELLED` | Job was cancelled | Handle gracefully | +| `DEADLINE_EXCEEDED` | Timeout | Retry or increase timeout | +| `UNAVAILABLE` | Server not reachable | Retry with backoff | +| `INTERNAL` | Server error | Report to user | +| `INVALID_ARGUMENT` | Bad request | Fix request | + +### Connection Handling + +- Client detects `context->IsCancelled()` for graceful disconnect +- Server cleans up job state on client disconnect during upload +- Automatic reconnection is NOT built-in (caller should retry) + +## Related Documentation + +- `GRPC_SERVER_ARCHITECTURE.md` — Server process model, IPC, threads, job lifecycle. +- `GRPC_QUICK_START.md` — Starting the server and solving remotely from Python, CLI, or C. +- `GRPC_CODE_GENERATION.md` — Registry format, generated file inventory, and walkthrough examples.