Split gem into Ruby and extension parts

[headius]

I would like to split the current Prism gem into two pieces: "prism" which contains the Ruby parts of the library, and "prism-ext" which contains the native extension and native library. This makes most sense for JRuby, but it would be simpler if done as a general move.

This is less of an issue and more of an idea that would make our lives easier in the JRuby world.

For JRuby users, the dependency on a native library is usually problematic. A large percentage of JRuby users (perhaps a majority) deploy JRuby into environments where there's no C build tools and frequently no allowances for random native libraries on the filesystem. Some of these deployments are all-in-one bundles that don't even unpack files to the filesystem. In such cases, the native library can't be built or loaded.

The most sensible option for the prism gem for JRuby would be that it ships only what can be run without native libraries; specifically this would mean prism would ship the WASM version of Prism so it can be embedded and bundled in all the ways JRuby users depend upon.

Users could opt-into the faster native Prism build by installing prism-ext or prism-library, which would build the dynamic library and make it available to the prism library. We would only ship prism as part of JRuby releases, since we need our distribution to always work across platforms.

The native-extension implementations would continue to just use the ext or the dynamic library as today.

The artifacts would look something like this:

• prism gem includes only the Ruby parts of the library and can be patched and versioned independently of the native code.
• prism-ext would work like the current prism gem does, building and installing either the CRuby extension or the dynamic library.
• The normal non-platform-specific version of the prism gem would depend upon prism-ext.
• The java platform version of the prism gem (for JRuby) would ship only the Ruby parts and the WASM library. Users can opt into the native version by installing prism-ext.

Alternatively, these changes would only apply to JRuby, and prism-ext would only be for JRuby users. Either way, the main prism gem for JRuby will need to exclude the native library.

I can think of other benefits to separating the extension into its own gem:

• CRuby users would be able to update just the Ruby part of Prism while still using the built-in native extension. This might make some bug fixes easier to distribute.
• If TruffleRuby in the future chooses to run Prism in a different form such as with Chicory or TruffleWasm, it would be a natural progression of this split.
• Doing the split uniformly (rather than just for JRuby) will keep the differences minimal. CRuby users would just get an additional extension dependency.

[marcoroth]

I wonder if we could update this part here and, for JRuby, require something like prism-jruby instead:

prism/lib/prism.rb

Lines 121 to 131 in 0399342

	if RUBY_ENGINE == "ruby" and !ENV["PRISM_FFI_BACKEND"]
	# The C extension is the default backend on CRuby.
	Prism::BACKEND = :CEXT
	require "prism/prism"
	else
	# The FFI backend is used on other Ruby implementations.
	Prism::BACKEND = :FFI
	require_relative "prism/ffi"
	end

[kddnewton]

I see what you're saying, but I am very hesitant to split things up at the moment.

The Ruby and C files are templated off config.yml to create exact matches. If we have a split, we could end up with different versions attempting to negotiate with each other, and honestly it feels like it's a recipe for pain.

That being said, I'm more than happy to add something to the extconf/gemspec that will handle JRuby explicitly to make this easier. And of course happy to add the WASM artifact to the bundle on the jruby-specific version of the gem.

[headius]

@kddnewton I respect that, and will proceed from here with a JRuby-only solution since we have the most unusual requirement.

If we have a split, we could end up with different versions attempting to negotiate with each other, and honestly it feels like it's a recipe for pain.

A hard dependency from a specific version of "prism" to a specific version of "prism-native" would largely prevent this, fwiw. It would not be possible to upgrade one without the other.

If splitting is not acceptable, then this will likely require pushing three gems for every release:

• prism (no platform): containing the Ruby code and the C extension and plain dynamic library
• prism (JRuby): containing the Ruby code and the WASM library
• prism-native: containing just the dynamic library (with pre-built versions for common platforms)

I'll see what I can come up with.

[Earlopain]

For prism-native, what's the performance difference actually like? If people have to add the more performant version themselves then I think it would have to be pretty significant to be worth the trouble. Especially if a large percentage/majority can't even make use of it, even if they know it exists.

[eregon]

I'm against splitting prism in multiple gems, it would bring a lot of complications.
Especially on non-JRuby (so CRuby & TruffleRuby) it does not make sense to split as the full Prism Ruby API cannot work without the native parts (be it the extension or FFI).
The extra noise in Gemfile.lock would already be a big negative.
Having different dependencies based on the "gem platform" (Gem::Platform::JAVA vs rest) seems a recipe for Gemfile.lock problems.

I think this would be better solved with coordination and some JRuby API, specifically if JRuby could provide some predicate for "native tools available and want native compilation for performance". Then Prism extconf.rb could decide to use WASM or Native based on that predicate, and there is no need for separate gems.
"native tools available" is something that could be checked by JRuby when staring e.g. if gcc/clang/make are installed.
"want native compilation for performance" could be opt-in or opt-out and controlled by env var or JRuby option etc, whatever makes more sense.
Maybe that predicate could be a method on RbConfig, which already holds info about the C compiler, etc.

@Earlopain makes a good point though that if the WASM version is close enough there is likely not enough value to worry about this.

[headius]

The performance of the WASM version varies from 2x to 10x slower and is highly dependant on JIT. Work continues to improve those metrics, but prism is already so fast I'm not concerned.

native tools available and want native compilation for performance

It must be an explicit opt-in. Having the behavior vary depending on whether C build tools exist would introduce an unpredictable element. Most users won't care that they are using the WASM version, and eventually we will ship some pre-built binaries to cover major platforms.

The default WASM gem is needed because many (maybe most) commercial JRuby users package their apps as all-in-one jar or war files, and native libraries will be problematic to load in such environments. We have already heard concerns from several users about having to pin other gems to earlier versions due to the growing number of transitive dependencies on prism.

In any case, I'll have to split the gem in two for JRuby so that's my cross to bear.

[headius]

Having different dependencies based on the "gem platform" (Gem::Platform::JAVA vs rest) seems a recipe for Gemfile.lock problems.

This is not unusual at all and bundler handles it just fine. Nearly every gem that has a native extension on CRuby has a different platform gem for JRuby if it is supported (some just ship an FFI version in the same gem).

[eregon]

It must be an explicit opt-in.

Right, but that opt-in could be a separate gem, or could be something more generic and/or simple like an env var e.g. JRUBY_GEMS_COMPILE_NATIVE_FOR_PERFORMANCE=true (default false). (or the reverse, JRUBY_GEMS_MUST_BE_PORTABLE=false, default true)
The advantages of that are:

• It can work not only for Prism but for other gems for which this decision needs to be made
• There is no separate gem, which is simpler to release and avoids confusion (e.g. CRuby users might wonder what is "prism-ext" and whether they need it).

[eregon]

and eventually we will ship some pre-built binaries to cover major platforms.

How would that work? Some extra pre-built binaries in the java-platform version of the prism gem?
Sounds like a good plan, and at that point then there would be very little value for the separate native gem, since the most common platforms would be already covered.
IMO this makes a lot of sense and similar logic is already needed e.g. to support FFI on JRuby.

There is also another way where the "ruby platform gem" (or the "java platform gem") dynamically installs a precompiled prism gem for the local platform. IIRC some gem(s) already do this.
It seems rather clear that it would be good to have prism as a precompiled gem on CRuby given it's a common dependency, so then JRuby could reuse those builds (including libprism.so, and ignoring the native extension).

Maybe it's worth going directly there, and until then to just use the WASM version?
The prism-ext gem would just be temporary so it might be better to not have it at all then.

[headius]

Right, but that opt-in could be a separate gem, or could be something more generic and/or simple like an env var

Env var is not feasible. It would require that people know about the variable and it would have to be set in production environments. Just defaulting to WASM, perhaps with an install-time note about the native gem for perf, still seems like the best option.

CRuby users might wonder what is "prism-ext" and whether they need it

If they accidentally install it, it shouldn't hurt anything. If it builds the ext, there's just another copy of the same ext built by the main prism gem. If it just builds the dynamic library, they wouldn't be impacted in any way.

I like to think some people will read the description, though, and not use it if it's not necessary.

If it is dylib-only it might also make sense to call it something like prism-native or prism-dylib or something more descriptive.

My first priority is getting the wasm gem 100% functional and released, though. I'm betting very few people using the prism gem right now (and even fewer using it on JRuby) need blindingly fast performance.

there would be very little value for the separate native gem
The prism-ext gem would just be temporary

It would still need to be available for JRuby users on platforms without a packaged binary (and there are many of them out there on exotic platforms). But we're getting ahead of ourselves anyway and can consider that scenario when it's relevant.

dynamically installs a precompiled prism gem for the local platform.

I can't say I've seen this, but it sounds interesting. My understanding of Bundler, though, is that all gems you use would need to be direct dependencies, so a dynamic gem would end up having to be manually added to Gemfile anyway.

[headius]

Oh and...

Some extra pre-built binaries in the java-platform version of the prism gem?

Yes, something like that, or more likely some pre-built binaries in the prism-native gem. The two issues for JRuby users are:

• C build tools may not be available
• Dynamic libraries may not be usable

Putting the binaries in the gem would only address the first issue.

[eregon]

dynamically installs a precompiled prism gem for the local platform.

I can't say I've seen this, but it sounds interesting. My understanding of Bundler, though, is that all gems you use would need to be direct dependencies, so a dynamic gem would end up having to be manually added to Gemfile anyway.

I found it, sass-embedded does this, from this description:

I'm the maintainer of sass-embedded gem. This gem has "native extension" that is a Dart VM binary executable, which is completely independent and not linked to libruby. On the platform "ruby", there is a Rakefile based extension that downloads the correct binary for the current platform [...]

And this is the logic.
It's not expressed a dependency in the gem, and it's a single gem.

So if Prism would have precompiled binaries, JRuby could reuse those in a similar way, i.e. by having the extconf.rb download the platform-specific gem and extract the libprism.{so,dylib}.
That would also work on TruffleRuby BTW (where the main gain would be faster installation due to no compilation).

My first priority is getting the wasm gem 100% functional and released, though. I'm betting very few people using the prism gem right now (and even fewer using it on JRuby) need blindingly fast performance.

💯 in any case WASM seems the clear first step for JRuby, and the native is nice-to-have and would likely become significantly easier if Prism ships precompiled gems on CRuby.

[eregon]

As a side note more flexibility for platform-specific/precompiled gems would be helpful here, such as this.