Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions .github/workflows/extended.yml
Original file line number Diff line number Diff line change
Expand Up @@ -173,10 +173,9 @@ jobs:
ref: ${{ github.event.inputs.pr_head_sha }} # will be empty if triggered by push
submodules: true
fetch-depth: 1
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
Copy link
Member Author

@blaginin blaginin Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this sets RUST_BACKTRACE=1 and we have a lot of expected failures in sqlogictests - getting backtrace for those is pointless and expensive

with:
rust-version: stable
- name: Install protobuf compiler
run: |
apt-get update && apt-get install -y protobuf-compiler
- name: Run sqllogictest
run: |
cargo test --features backtrace,parquet_encryption --profile ci-optimized --test sqllogictests -- --include-sqlite
13 changes: 9 additions & 4 deletions datafusion/common/src/utils/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ use std::cmp::{Ordering, min};
use std::collections::HashSet;
use std::num::NonZero;
use std::ops::Range;
use std::sync::Arc;
use std::sync::{Arc, LazyLock};
use std::thread::available_parallelism;

/// Applies an optional projection to a [`SchemaRef`], returning the
Expand Down Expand Up @@ -923,10 +923,15 @@ pub fn combine_limit(
///
/// This is a wrapper around `std::thread::available_parallelism`, providing a default value
/// of `1` if the system's parallelism cannot be determined.
///
/// The result is cached after the first call.
pub fn get_available_parallelism() -> usize {
available_parallelism()
.unwrap_or(NonZero::new(1).expect("literal value `1` shouldn't be zero"))
.get()
static PARALLELISM: LazyLock<usize> = LazyLock::new(|| {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

telegram-cloud-photo-size-2-5328039229025621418-w

when running on linux, this function is constantly running

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this obviously affects not only tests, happy to apply to only tests, but feels like a good change overall? 🤔

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm great find!

Perhaps there also seems to go something wrong with rewrite_with_subqueries + SimplifyExpressions that make it call so often / create it so often.

available_parallelism()
.unwrap_or(NonZero::new(1).expect("literal value `1` shouldn't be zero"))
.get()
});
*PARALLELISM
}

/// Converts a collection of function arguments into a fixed-size array of length N
Expand Down
Loading