Skip to content
GitLab
Closed
Issue created by Mike Kaplinskiy@mikekap

Maven resolver doesn't work for spark

$ java -jar /tmp/resolver.jar . thirdparty /tmp/mvn/ http://repo1.maven.org/maven2 org.apache.spark:spark-core_2.11:1.3.1
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" org.eclipse.aether.resolution.DependencyResolutionException: The following artifacts could not be resolved: org.apache.hadoop:hadoop-yarn-client:jar:1.0.4, org.apache.hadoop:hadoop-yarn-api:jar:1.0.4, org.apache.hadoop:hadoop-yarn-common:jar:1.0.4: Could not find artifact org.apache.hadoop:hadoop-yarn-client:jar:1.0.4 in http://repo1.maven.org/maven2 (http://repo1.maven.org/maven2)
    at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:384)
    at com.facebook.buck.maven.Resolver.getRunTimeTransitiveDeps(Resolver.java:435)
    at com.facebook.buck.maven.Resolver.resolve(Resolver.java:121)
    at com.facebook.buck.maven.Resolver.main(Resolver.java:482)
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: The following artifacts could not be resolved: org.apache.hadoop:hadoop-yarn-client:jar:1.0.4, org.apache.hadoop:hadoop-yarn-api:jar:1.0.4, org.apache.hadoop:hadoop-yarn-common:jar:1.0.4: Could not find artifact org.apache.hadoop:hadoop-yarn-client:jar:1.0.4 in http://repo1.maven.org/maven2 (http://repo1.maven.org/maven2)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:444)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:246)
    at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:367)
    ... 3 more
Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact org.apache.hadoop:hadoop-yarn-client:jar:1.0.4 in http://repo1.maven.org/maven2 (http://repo1.maven.org/maven2)
    at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed(ArtifactTransportListener.java:39)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:355)
    at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:67)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

I don't know enough about java artifact resolution, but some googling suggests that hadoop.version isn't being propagated correctly and is defaulting to 1.0.4 in the spark pom (and hadoop-yarn-client doesn't exist on 1.0.4).