Skip to content

Conversation

@stefanhahmann
Copy link
Contributor

This draft PR is a suggestion to fix an issue with appose in Windows. It is very similar to #5 and probably again not the ideal solution.

I am using this env:

name: cellpose3
channels:
  - nvidia
  - pytorch
  - conda-forge
channel_priority: strict
dependencies:
  - python=3.10
  - cellpose==3.1.1.2
  - appose==0.7.0
  - pytorch
  - pytorch-cuda
  - numpy

Within this env I run:

micromamba run -p C:/Users/xxxx/.local/share/appose/cellpose3 C:/Users/xxxx/.local/share/appose/cellpose3/python.exe -c "import appose.python_worker; appose.python_worker.main()"

and then interactively inputing this task to stdin

{"task":"c388f270-d6be-48df-9e02-e42b48cab98b","requestType":"EXECUTE","inputs":{},"script":"import numpy\n"}

This results in a perfectly working process:

[SERVICE-0] {"task":"c388f270-d6be-48df-9e02-e42b48cab98b","requestType":"EXECUTE","inputs":{},"script":"import numpy\n"}
[SERVICE-0] {"task":"c388f270-d6be-48df-9e02-e42b48cab98b","responseType":"LAUNCH"}
[SERVICE-0] {"task":"c388f270-d6be-48df-9e02-e42b48cab98b","responseType":"COMPLETION","outputs":{}}

However, when I try to run this from a Java process similar to the code example (https://github.com/mastodon-sc/mastodon-deep-lineage/blob/appose/src/test/java/org/mastodon/mamut/appose/ApposeCellpose3Example.java), I only get this response and then the whole process is stuck in a dead lock:

[SERVICE-0] {"task":"c388f270-d6be-48df-9e02-e42b48cab98b","requestType":"EXECUTE","inputs":{},"script":"import numpy\n"}
[SERVICE-0] {"task":"c388f270-d6be-48df-9e02-e42b48cab98b","responseType":"LAUNCH"}

The code added in this PR fixes this situation. It seems that loading imports like (e.g.) numpy can lead to deadlocks when being loaded in (multiple) threads.

I am again not sure, if the proposed fix is a good one however. Since the problem occurs only on Windows, one could think about adding if sys.platform == 'win32': or something like this to only perform the extra code on Windows. Also, one could think about making this optional and controlable from outside the python_worker by perhaps only performing the new code, if some global variable is set.

@ctrueden curious to learn your opinion about this.

Pre-load imports in the main thread to avoid issues on Windows.
@stefanhahmann
Copy link
Contributor Author

I could just verify that Appose 0.8.0 (java) together with Appose 0.7.1 (python) fixes this Windows hanging bug in https://github.com/mastodon-sc/mastodon-deep-lineage/blob/dff692442470e23a224364e3a0f7ffb0862fdb01/src/test/java/org/mastodon/mamut/appose/ApposeCellpose3Example.java

All I needed to do is adding the numpy import to the init() method: Service python = env.python().init( "import numpy" )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant