Skip to content

Improve application start API#7

Open
kmaehashi wants to merge 3 commits intodevelopfrom
add-yarn-application-name
Open

Improve application start API#7
kmaehashi wants to merge 3 commits intodevelopfrom
add-yarn-application-name

Conversation

@kmaehashi
Copy link
Member

This patch improves the Jubatus-on-YARN API to start the Jubatus cluster.

@rimms
Copy link
Member

rimms commented Oct 21, 2015

I faced compile error:

[info] Set current project to jubatus-on-yarn-client (in build file:/path/to/jubatus-on-yarn/jubatusonyarn/)
[info] Compiling 7 Scala sources to /path/to/jubatus-on-yarn/jubatusonyarn/jubatus-on-yarn-client/target/scala-2.10/classes...
[error] /path/to/jubatus-on-yarn/jubatusonyarn/jubatus-on-yarn-client/src/main/scala/us/jubat/yarn/client/Model.scala:31: overloaded method value start with alternatives:
[error]   (aLearningMachineName: String,aLearningMachineType: us.jubat.yarn.common.LearningMachineType,aZookeepers: List[us.jubat.yarn.common.Location],aConfigFile: org.apache.hadoop.fs.Path,aResource: us.jubat.yarn.client.Resource,aNodeCount: Int,aBasePath: org.apache.hadoop.fs.Path,aApplicationName: String)scala.concurrent.Future[us.jubat.yarn.client.JubatusYarnApplication] <and>
[error]   (aLearningMachineName: String,aLearningMachineType: us.jubat.yarn.common.LearningMachineType,aZookeepers: List[us.jubat.yarn.common.Location],aConfigFile: org.apache.hadoop.fs.Path,aResource: us.jubat.yarn.client.Resource,aNodeCount: Int,aApplicationName: String)scala.concurrent.Future[us.jubat.yarn.client.JubatusYarnApplication] <and>
[error]   (aLearningMachineName: String,aLearningMachineType: us.jubat.yarn.common.LearningMachineType,aZookeepers: List[us.jubat.yarn.common.Location],aConfigString: String,aResource: us.jubat.yarn.client.Resource,aNodeCount: Int,aBasePath: org.apache.hadoop.fs.Path,aApplicationName: String)scala.concurrent.Future[us.jubat.yarn.client.JubatusYarnApplication] <and>
[error]   (aLearningMachineName: String,aLearningMachineType: us.jubat.yarn.common.LearningMachineType,aZookeepers: List[us.jubat.yarn.common.Location],aConfigString: String,aResource: us.jubat.yarn.client.Resource,aNodeCount: Int,aApplicationName: String)scala.concurrent.Future[us.jubat.yarn.client.JubatusYarnApplication]
[error]  cannot be applied to (String, us.jubat.yarn.common.LearningMachineType.Classifier.type, List[us.jubat.yarn.common.Location], org.apache.hadoop.fs.Path, us.jubat.yarn.client.Resource, Int)
[error]   val tApplicationFuture = JubatusYarnApplication.start(
[error]                                                   ^
[error] /path/to/jubatus-on-yarn/jubatusonyarn/jubatus-on-yarn-client/src/main/scala/us/jubat/yarn/client/YarnClientServlet.scala:59: overloaded method value startJubatusApplication with alternatives:
[error]   (aName: String,aLearningMachineType: us.jubat.yarn.common.LearningMachineType,aZookeepers: List[us.jubat.yarn.common.Location],aConfigFile: org.apache.hadoop.fs.Path,aResource: us.jubat.yarn.client.Resource,aNodeCount: Int,aBasePath: org.apache.hadoop.fs.Path,aApplicationName: String)us.jubat.yarn.client.ApplicationMasterProxy <and>
[error]   (aName: String,aLearningMachineType: us.jubat.yarn.common.LearningMachineType,aZookeepers: List[us.jubat.yarn.common.Location],aConfigString: String,aResource: us.jubat.yarn.client.Resource,aNodeCount: Int,aBasePath: org.apache.hadoop.fs.Path,aApplicationName: String)us.jubat.yarn.client.ApplicationMasterProxy
[error]  cannot be applied to (String, us.jubat.yarn.common.LearningMachineType, List[us.jubat.yarn.common.Location], String, us.jubat.yarn.client.Resource, Int, org.apache.hadoop.fs.Path)
[error]       mController.get.startJubatusApplication(tSettings.learningMachineInstanceName, tSettings.learningMachineType, tListZooKeeper, tSettings.configJson, tSettings.resources, tSettings.nodeCount, new Path(tSettings.basePath))
[error]                       ^
[error] two errors found
[error] (jubatus-on-yarn-client/compile:compile) Compilation failed

@kmaehashi kmaehashi force-pushed the add-yarn-application-name branch from e58715a to 5e8221d Compare November 18, 2015 11:58
@kmaehashi kmaehashi changed the title add interface to specify YARN application name Improve application start API Nov 18, 2015
@kmaehashi kmaehashi force-pushed the add-yarn-application-name branch from 5e8221d to 508105a Compare November 18, 2015 12:51
@kmaehashi kmaehashi force-pushed the add-yarn-application-name branch from 508105a to 74604f3 Compare November 18, 2015 12:59
@kmaehashi
Copy link
Member Author

I revised the contents of this PR.

  • YARN application name for JoY can now be specified externally.
  • The old (four overloaded) start APIs had too many arguments. I added new start API that accepts cluster configurations via one object, and deprecated old ones.
  • Previously, list of ZooKeepers are given as List[Location], however this was inappropriate, as ZooKeeper location string is a kind of DSL parsed by ZooKeeper client library. I fixed this to directly pass String instead.
  • Cluster launching options are now well-validated.
  • Removed /start HTTP API in YarnClientServlet, which was unused.

@kmaehashi kmaehashi force-pushed the add-yarn-application-name branch 4 times, most recently from 260aae7 to 88c2cb6 Compare November 19, 2015 01:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants