Shortcuts expansion can fail with Python version

[ctrueden]

Maybe an issue with shortcut containing concatenated endpoints?

Create the following .jgorc:

[shortcuts]
groovy = org.codehaus.groovy:groovy-groovysh:org.codehaus.groovy.tools.shell.Main+commons-cli:commons-cli:1.3.1

Then try launching it:

$ jgo -v groovy
DEBUG 2019-06-28 18:56:35,467: Using settings:      {'m2repo': '/Users/curtis/.m2/repository', 'cachedir': '/usr/local/miniconda3/envs/scijava/jgo', 'links': 'auto'}
DEBUG 2019-06-28 18:56:35,467: Using respositories: {}
DEBUG 2019-06-28 18:56:35,467: Using shortcuts:     {'groovy': 'org.codehaus.groovy:groovy-groovysh:org.codehaus.groovy.tools.shell.Main+commons-cli:commons-cli:1.3.1'}
DEBUG 2019-06-28 18:56:35,467: Replacing groovy with org.codehaus.groovy:groovy-groovysh:org.codehaus.groovy.tools.shell.Main+commons-cli:commons-cli:1.3.1 in groovy.
DEBUG 2019-06-28 18:56:35,467: Returning expanded coordinate org.codehaus.groovy:groovy-groovysh:org.codehaus.groovy.tools.shell.Main+commons-cli:commons-cli:1.3.1.
INFO 2019-06-28 18:56:36,834: Failed to bootstrap the artifact.
INFO 2019-06-28 18:56:36,834:
INFO 2019-06-28 18:56:36,834: Possible solutions:
INFO 2019-06-28 18:56:36,834: * Double check the endpoint for correctness (https://search.maven.org/).
INFO 2019-06-28 18:56:36,834: * Add needed repositories to ~/.jgorc [repositories] block (see README).
INFO 2019-06-28 18:56:36,834: * Try with an explicit version number (release metadata might be wrong).

ERROR 2019-06-28 18:56:36,834: Error in `/usr/local/miniconda3/envs/scijava/bin/mvn -B -f /usr/local/miniconda3/envs/scijava/jgo/org/codehaus/groovy/groovy-groovysh/org.codehaus.groovy.tools.shell.Main+commons-cli/commons-cli/pom.xml dependency:resolve': 1
DEBUG 2019-06-28 18:56:36,834: Exception: Command '('/usr/local/miniconda3/envs/scijava/bin/mvn', '-B', '-f', '/usr/local/miniconda3/envs/scijava/jgo/org/codehaus/groovy/groovy-groovysh/org.codehaus.groovy.tools.shell.Main+commons-cli/commons-cli/pom.xml', 'dependency:resolve')' returned non-zero exit status 1.
DEBUG 2019-06-28 18:56:36,834: Debug Trace:
Traceback (most recent call last):
  File "/usr/local/miniconda3/envs/scijava/lib/python3.7/site-packages/jgo/jgo.py", line 258, in jgo_main
    completed_process = run(parser, argv=argv, stdout=stdout, stderr=stderr)
  File "/usr/local/miniconda3/envs/scijava/lib/python3.7/site-packages/jgo/jgo.py", line 554, in run
    link_type           = link_type)
  File "/usr/local/miniconda3/envs/scijava/lib/python3.7/site-packages/jgo/jgo.py", line 458, in resolve_dependencies
    raise e
  File "/usr/local/miniconda3/envs/scijava/lib/python3.7/site-packages/jgo/jgo.py", line 449, in resolve_dependencies
    mvn_out = run_and_combine_outputs(mvn, *mvn_args)
  File "/usr/local/miniconda3/envs/scijava/lib/python3.7/site-packages/jgo/jgo.py", line 200, in run_and_combine_outputs
    return subprocess.check_output((command,) + args, stderr=subprocess.STDOUT)
  File "/usr/local/miniconda3/envs/scijava/lib/python3.7/subprocess.py", line 395, in check_output
    **kwargs).stdout
  File "/usr/local/miniconda3/envs/scijava/lib/python3.7/subprocess.py", line 487, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '('/usr/local/miniconda3/envs/scijava/bin/mvn', '-B', '-f', '/usr/local/miniconda3/envs/scijava/jgo/org/codehaus/groovy/groovy-groovysh/org.codehaus.groovy.tools.shell.Main+commons-cli/commons-cli/pom.xml', 'dependency:resolve')' returned non-zero exit status 1.
DEBUG 2019-06-28 18:56:36,838: 	std out:
DEBUG 2019-06-28 18:56:36,839: 		b'[INFO] Scanning for projects...
DEBUG 2019-06-28 18:56:36,839: 		[INFO]
DEBUG 2019-06-28 18:56:36,839: 		[INFO] ---< org.codehaus.groovy-BOOTSTRAPPER:groovy-groovysh-BOOTSTRAPPER >----
DEBUG 2019-06-28 18:56:36,839: 		[INFO] Building groovy-groovysh-BOOTSTRAPPER 0
DEBUG 2019-06-28 18:56:36,839: 		[INFO] --------------------------------[ jar ]---------------------------------
DEBUG 2019-06-28 18:56:36,839: 		[WARNING] The POM for org.codehaus.groovy:groovy-groovysh:jar:commons-cli:org.codehaus.groovy.tools.shell.Main+commons-cli is missing, no dependency information available
DEBUG 2019-06-28 18:56:36,839: 		[INFO] ------------------------------------------------------------------------
DEBUG 2019-06-28 18:56:36,839: 		[INFO] BUILD FAILURE
DEBUG 2019-06-28 18:56:36,839: 		[INFO] ------------------------------------------------------------------------
DEBUG 2019-06-28 18:56:36,839: 		[INFO] Total time:  0.328 s
DEBUG 2019-06-28 18:56:36,839: 		[INFO] Finished at: 2019-06-28T18:56:36+02:00
DEBUG 2019-06-28 18:56:36,839: 		[INFO] ------------------------------------------------------------------------
DEBUG 2019-06-28 18:56:36,839: 		[ERROR] Failed to execute goal on project groovy-groovysh-BOOTSTRAPPER: Could not resolve dependencies for project org.codehaus.groovy-BOOTSTRAPPER:groovy-groovysh-BOOTSTRAPPER:jar:0: Failure to find org.codehaus.groovy:groovy-groovysh:jar:commons-cli:org.codehaus.groovy.tools.shell.Main+commons-cli in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]
DEBUG 2019-06-28 18:56:36,839: 		[ERROR]
DEBUG 2019-06-28 18:56:36,839: 		[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
DEBUG 2019-06-28 18:56:36,839: 		[ERROR] Re-run Maven using the -X switch to enable full debug logging.
DEBUG 2019-06-28 18:56:36,839: 		[ERROR]
DEBUG 2019-06-28 18:56:36,839: 		[ERROR] For more information about the errors and possible solutions, please read the following articles:
DEBUG 2019-06-28 18:56:36,839: 		[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
DEBUG 2019-06-28 18:56:36,839: 		'

[hanslovsky]

As far as I can tell, the issue is that short cut expansion happens after splitting into endpoints at + in Python jgo, for example for INPUT=endpoint1+endpoint2

jgo INPUT

will be expanded to the endpoints

expand_if_shortcut(endpoint1)
expand_if_shortcut(endpoint2)

In the example provided in this issue, however, INPUT=groovy:

expand_if_shortcut(groovy)

which results in a single, mal-formed endpoint

org.codehaus.groovy:groovy-groovysh:org.codehaus.groovy.tools.shell.Main+commons-cli:commons-cli:1.3.1

Now this raises the following questions about the order of shortcut expansion and splitting into endpoints:

1. Split first, then expand (as in current Python version):
   • pro Shortcuts can be applied to all endpoints
   • con Shortcuts cannot contain additional endpoints (+)
   • con Inconsistent with jgo.sh
2. Expand first, then split (as suggested in this example):
   • pro Shortcuts can contain additional endpoints (+)
   • pro Consistent with jgo.sh
   • con Additional endpoints cannot be created through shortcut chaining, e.g. groovy-groovysh+commons-cli
3. Expand first, then split, then expand again
   • pro can do both (1) and (2)
   • con probably very confusing

I do not have a strong opinion on (1) or (2) but I think we should give it some thought and then decide either way. I personally would like to rule out (3) because it will probably be very confusing and likely cause more harm than good.

[hanslovsky]

I have a simple solution that would be a fourth solution:
4. Split first, expand, then split again

diff --git a/jgo/jgo.py b/jgo/jgo.py
index 0adae45..5ea9ee1 100644
--- a/jgo/jgo.py
+++ b/jgo/jgo.py
@@ -368,7 +368,7 @@ def split_endpoint_string(endpoint_string):
 
 def endpoints_from_strings(endpoint_strings, shortcuts={}):
     _logger.debug('Creating endpoints from strings %s with shortcuts %s', endpoint_strings, shortcuts)
-    return [Endpoint.parse_endpoint(expand_coordinate(ep, shortcuts=shortcuts)) for ep in endpoint_strings]
+    return [Endpoint.parse_endpoint(ep2) for ep in endpoint_strings for ep2 in expand_coordinate(ep, shortcuts=shortcuts).split('+')]
 
 def coordinates_from_endpoints(endpoints):
     return [ep.get_coordinates() for ep in endpoints]

In this case, expanded shortcuts would be split again:

split:  groovy -> groovy
expand: groovy -> org.codehaus.groovy:groovy-groovysh:org.codehaus.groovy.tools.shell.Main+commons-cli:commons-cli:1.3.1
split:  org.codehaus.groovy:groovy-groovysh:org.codehaus.groovy.tools.shell.Main+commons-cli:commons-cli:1.3.1 -> [
    org.codehaus.groovy:groovy-groovysh:org.codehaus.groovy.tools.shell.Main,
    commons-cli:commons-cli:1.3.1
]

I still have to get a better intuition. Let me know what you think @ctrueden