Skip to content

feat(bindings/python): Add from_uri constructors for Operators#6993

Open
clbarnes wants to merge 11 commits intoapache:mainfrom
clbarnes:py-from-uri
Open

feat(bindings/python): Add from_uri constructors for Operators#6993
clbarnes wants to merge 11 commits intoapache:mainfrom
clbarnes:py-from-uri

Conversation

@clbarnes
Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #6984 .

Rationale for this change

Instantiating Operators from URLs greatly increases flexibility for consumers of the openDAL API as an abstract backend.

What changes are included in this PR?

Add from_uri static methods to the Operator and AsyncOperator classes in the python bindings.

Are there any user-facing changes?

Two new methods are available to users of the python library.

AI Usage Statement

None

Other notes

I am having issues with my development environment which the devcontainer is not making any easier, so I'm flying a bit blind and relying on CI.

@clbarnes
Copy link
Copy Markdown
Contributor Author

Tests pass but I'm not clear how best to test the new functionality as it would mean parsing the config provided by the setup_config fixture and then constructing a URI out of it, which is going to be different for each service.

@clbarnes clbarnes marked this pull request as ready for review December 12, 2025 16:35
@dosubot dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. releases-note/feat The PR implements a new feature or has a title that begins with "feat" labels Dec 12, 2025
@dosubot dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:XXL This PR changes 1000+ lines, ignoring generated files. labels Dec 12, 2025
@Xuanwo Xuanwo changed the title Python: Add from_uri constructors for Operators feat(bindings/python): Add from_uri constructors for Operators Dec 12, 2025
@Xuanwo
Copy link
Copy Markdown
Member

Xuanwo commented Dec 12, 2025

Thank you for working on this! I will invite @chitralverma to take a review on this.

@chitralverma
Copy link
Copy Markdown
Contributor

@clbarnes can you run stubgen with just to get the python stubs as well?

@clbarnes
Copy link
Copy Markdown
Contributor Author

clbarnes commented Dec 15, 2025

I found that running stubgen locally drops a bunch of services and deletes a lot of generated code as well - possibly all the services which have been split out of core into their own crates?

--- a/bindings/python/src/services.rs
+++ b/bindings/python/src/services.rs
@@ -62,16 +62,6 @@ submit! {
 #[pyo3(rename_all = "PascalCase")]
 #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
 pub enum PyScheme {
-    #[cfg(feature = "services-aliyun-drive")]
-    AliyunDrive,
-    #[cfg(feature = "services-alluxio")]
-    Alluxio,
-    #[cfg(feature = "services-azblob")]
-    Azblob,
-    #[cfg(feature = "services-azdls")]
-    Azdls,
-    #[cfg(feature = "services-azfile")]
-    Azfile,
     #[cfg(feature = "services-b2")]
     B2,
     #[cfg(feature = "services-cacache")]
@@ -84,24 +74,14 @@ pub enum PyScheme {
     Dropbox,
     #[cfg(feature = "services-fs")]
     Fs,
-    #[cfg(feature = "services-ftp")]
-    Ftp,
-    #[cfg(feature = "services-gcs")]
-    Gcs,
     #[cfg(feature = "services-gdrive")]
     Gdrive,
-    #[cfg(feature = "services-ghac")]
-    Ghac,
     #[cfg(feature = "services-gridfs")]
     Gridfs,
-    #[cfg(feature = "services-hdfs-native")]
-    HdfsNative,
     #[cfg(feature = "services-http")]
     Http,
     #[cfg(feature = "services-huggingface")]
     Huggingface,
-    #[cfg(feature = "services-ipfs")]
-    Ipfs,
     #[cfg(feature = "services-ipmfs")]
     Ipmfs,
     #[cfg(feature = "services-koofr")]

etc.. That comes from the cargo run --quiet --manifest-path=../../dev/Cargo.toml -- generate -l python step.

The second step doesn't finish successfully - cargo run --quiet --bin stub_gen.

opendal/bindings/python on  py-from-uri [$!⇡] is 📦 v0.47.0 via 🐍 v3.14.0 (opendal) via 🦀 v1.91.0 took 1m3s 
❯ cargo run --quiet --bin stub_gen
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000001f16b6080 [main] (most recent call first):
  <no Python frame>

but python -c 'import encodings' works.

This could possibly resolved using the devcontainer, but that has issues too: #6992

@TennyZhuang
Copy link
Copy Markdown
Contributor

Hi @clbarnes, this PR has been quiet for ~4 months. Is the implementation still in progress, or would you welcome help getting it over the line? Happy to assist with rebase/CI fixes or take it over with attribution if that would help.

@clbarnes
Copy link
Copy Markdown
Contributor Author

clbarnes commented May 1, 2026

I will rebase and give it another try, but I was blocked last time with setting up an environment with everything in place for getting stubgen to run properly. I think the actual code is functional. Thanks for the reminder!

TennyZhuang added a commit that referenced this pull request May 2, 2026
Add a static fromUri(uri, options?) factory method to the Node.js
Operator class, mirroring Rust Operator::from_uri and Python
Operator.from_uri (PR #6993).

The URI encodes scheme and configuration in a single string, e.g.
  memory://localhost/
  s3://bucket/path?region=us-east-1

Optional extra key-value options can be passed to override or
supplement values encoded in the URI.

Intentional divergence: none. Direct 1:1 mapping of Rust
Operator::from_uri((uri, options)) via IntoOperatorUri.

Scope: bindings/nodejs only (lib.rs + generated.d.ts).
AI-assisted: od-claude-lead-050121, reviewed by OpenDAL regression team.

Co-authored-by: tianyizhuang <tianyizhuang@tianyizhuangs-Mac-mini.local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

releases-note/feat The PR implements a new feature or has a title that begins with "feat" size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

new feature: expose Operator::from_uri to other language bindings

4 participants