Dare To Think, Strive To Execute

scopt使用以及NoSuchMethodError处理

发表于

scopt的使用

  在plugins.sbt文件中添加Assembly插件, addSbtPlugin(“com.eed3si9n” % “sbt-assembly” % “0.11.2”)
创建一个参数类，ParaConfig

1
2
3
4
5
6
7
8
9
10
package NMF
case class ParaConfig(
edgesFile: String = "hdfs://bda00:8020/user/liping/edges.txt",
output: String = "hdfs://bda00:8020/user/liping/NMFResult",
edgesMinPartition: Int = 2,
reducedDim: Int = 20,
learnRate: Double = 0.01,
reg: Double = 0.1,
maxIteration: Int = 40
)

解析参数的例子

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
val parser = new scopt.OptionParser[ParaConfig]("runNMF"){
head("NMF", "1.0")
opt[String]("edgesFile") optional() action { (x, c) =>
c.copy(edgesFile = x)
} text ("edgesFile is the input file that includes the graph information")
opt[String]("output") optional() action { (x, c) =>
c.copy(output = x)
} text ("output is the output file that stores two matrix W and H")
opt[Int]("reducedDim") optional() action { (x, c) =>
c.copy(reducedDim = x)
} validate { x => if (x > 0) success else failure("Option --reducedDim must >0")
} text ("reduceDim is the factorized dimension which is known as K")
opt[Int]("maxIteration") optional() action { (x, c) =>
c.copy(maxIteration = x)
} validate { x => if (x > 0) success else failure("Option --maxIteration must >0")
} text ("maxIteraton is the max Iteration count of this factorization program")
opt[Int]("edgesMinPartition") optional() action { (x, c) =>
c.copy(edgesMinPartition = x)
} validate { x => if (x > 0) success else failure("Option --edgesMinPartition must >0")
} text ("edgesMinPartition is the min number of RDD's split parts, default is 2")
opt[Double]("reg") optional() action { (x, c) =>
c.copy(reg = x)
} validate { x => if (x > 0.0) success else failure("Option --reg must >0.0")
} text ("reg is the regularization part, default if 0.1")
opt[Double]("learnRate") optional() action { (x, c) =>
c.copy(learnRate = x)
} validate { x => if (x > 0.00) success else failure("Option --learnRate must >0.00")
} text("learning rate, default is 0.01")
}
val para: ParaConfig = parser.parse(args, ParaConfig()).get
val edgesMinPartition = para.edgesMinPartition
val reducedDim = para.reducedDim
val learnRate = para.learnRate
val reg = para.reg
val edgesFile = para.edgesFile
val output = para.output
val maxIteration = para.maxIteration

Assembly 打包

  打包过程中会出现很多冲突，可以在.sbt中依赖的包后面加上provided，一来不用解决冲突，另一方面也会减少包的容量。

NoSuchMethodError解决

  Assembly打包成功并不是万事大吉了，待运行的时候可能会遇到NoSuchMethodError这样的错误。
  这个错误是因为scala 的版本问题造成了，将build.sbt中scala的版本写成2.10.4这个ERROR就可以避免。至此就没有大的问题了。