brainvisa » brainvisa-development

• on michael / gabriel (CentOS 5), python2 points to the system python, which is a 2.4, not the one we are using
• on Mac the link does not exist
  We can easily fix those 2 by manually creating the symlink on both machines
• in binary packages we ship "python" and "pyton2.7" as the former is a link to the latter, but not "python2", so we need to change the packaging of python
• on Windows I can't tell right now (can't connect to the Windows VM now - I don't know why) but as symlinks don't exist there I doubt both python and python2 exe are here.
• aren't we going against a kind of standard if we do that, or is it a standard practice ?

It is indeed recommended practice. From https://www.python.org/dev/peps/pep-0394/:

In order to tolerate differences across platforms, all new code that needs to invoke the Python interpreter should not specify python , but rather should specify either python2 or python3 [...]. This distinction should be made in shebangs, when invoking from a shell script, when invoking via the system() call, or when invoking in any other context.

The python2 link is actually created when building Python from source (since 2.7.3), so I think that it makes sense to include it in the packaging.

As for Windows, I do not know how the Python scripts are called right now, but a launcher is available that actually interprets the #! /usr/bin/env python2 line and automatically launches the required version (https://docs.python.org/3/using/windows.html#shebang-lines).

OK let's go for it.
Symlinks have been added on build systems (Linux/Mac).
I'm currently modifying python packaging.
We have also to check what's going on on Windows (I'm not sure how this python launcher is installed/used, if it is: we are using a bash shell most of the time, so it probably works as on unix).

Well, the context.pythonSystem() call runs python as a child process, on the same machine as the caller. Remote execution, at least currently, runs a brainvisa on a remote node, which may spawn subprocesses for system commands (python or other). So in this context I think it's safe to use the same python as the parent brainvisa.
In the code at the moment "python" is used unqualified to let the PATH find it, but (at least most of the time) it will be the same as sys.executable, so maybe it's OK to use sys.executable.

For remote python access, the question would rather be in soma-workflow jobs building from brainvisa processes. Here we may wonder which python should be used (in workflow.py, ProcessToWorkflow.__init__()). Here again, for now we are relying on the PATH, and running the unqualified "python" command. Maybe we could use the basename of sys.executable (because we are running brainvisa, so we are supposed to use the same python version)?

I agree that commands using python3 (or any other python which is not the one used by brainvisa) must use it explicitely. Maybe we could provide a kwarg in pythonSystem() to specify a different python command, but this is not the hot question now.

It should be noted that several old systems do not provide a "python2" command or symlink, we faced a number of such situations, but making a symlink somewhere in the PATH is an easy solution.

I do not think that we should use sys.executable, because this would basically prevent people from writing Python3 scripts.

That's half true, yes. In my last message I was suggesting that scripts using python3 should not use pythonsystem(), or use it with an addditional parameter to specify it (python_command="python3" for instance). But you're right in the way that in this situation the correct python version must be known from "outside" of the script, and it's a "bidouille" anyway. The existance of this pythonSystem() method is, in itself, a poor attempt at fixing something that does not work well by itself...
On Linux we would totally skip it.

The problem with the current interface of context.pythonSystem is that it is one wrapper for two different languages (Python2 and Python3).

I think that the best approach would be to rely on the shebang (initial #!/usr/bin/env python line) for running the correct version. Under Linux/Mac this is straightforward, all that should be done is to make the program executable and run it as /path/to/program.py instead of python /path/to/program.py. Under Windows an official Python launcher exists that reads the shebang line and launches the right version of Python (py.exe / pyw.exe, see https://www.python.org/dev/peps/pep-0397/). We can then use py.exe /path/to/program.py, or maybe with the help of file type associations we do not even need to call py.exe explicitly (I do not know much about Windows).

Perhape yes, but it would require to install and package correctly this py.exe in binary distributions. It could be done.

Another thing: on the latest Mac systems (10.11), a "funny" "security" feature, called "system integrity protection" prevents shells to run python scripts via the shebang if the python script is not approved by Apple (or something like that, not sure I have really understood) (see http://brainvisa.info/forum/viewtopic.php?f=2&t=1761#p6563). Of course we are not approved by Apple, and BrainVisa is not distributed via the apple app store, so there is no chance we will be one day, and custom processes will not, either. Running python the way we do it in BrainVisa avoids to run a shell, and avoids this problem of system blocking python scripts execution. So it's still useful for this, also.

For remote python access,
...

Again, I don't think that we should make the assumption that the script is using the same version as brainvisa/soma-workflow. Disentangling the two would allow the writers of Python child processes to use the version that they want.

Here it's not really an issue I think, because workflows are generated by brainvisa, and they are always running brainvisa (it's brainvisa processes which are in workflow nodes), so they do need the same python version anyway.

As for systems that do not have a python2 executable, we should probably provide a sane PATH that sticks to the recommendations of https://www.python.org/dev/peps/pep-0394/#recommendation, independent of sys.executable:

• python2 -> preferred python2 version
• python3 -> preferred python3 version
• python -> python2

In binary distributions we ship a python2 symlink along with the python command. The problem is only for people who develop by taking the sources of brainvisa, on an old system (but it's a common situation on clusters, for instance). And again it's a very small problem that is easily overcome.