Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
3fe087d9
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
3fe087d9
编写于
2月 11, 2013
作者:
S
sewen
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Implemented dangling pagerank with generic data types.
上级
5f58ce70
变更
27
隐藏空白更改
内联
并排
Showing
27 changed file
with
1003 addition
and
146 deletion
+1003
-146
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/AsciiLongArrayView.java
...ve/compensatable/danglingpagerank/AsciiLongArrayView.java
+4
-4
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/CompensatableDanglingPageRank.java
...table/danglingpagerank/CompensatableDanglingPageRank.java
+2
-2
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/CustomCompensatableDanglingPageRank.java
...gpagerank/custom/CustomCompensatableDanglingPageRank.java
+305
-0
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/CustomCompensatableDotProductCoGroup.java
...pagerank/custom/CustomCompensatableDotProductCoGroup.java
+110
-0
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/CustomCompensatableDotProductMatch.java
...ngpagerank/custom/CustomCompensatableDotProductMatch.java
+73
-0
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/CustomCompensatingMap.java
...atable/danglingpagerank/custom/CustomCompensatingMap.java
+61
-0
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/CustomImprovedAdjacencyListInputFormat.java
...gerank/custom/CustomImprovedAdjacencyListInputFormat.java
+46
-0
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/CustomImprovedDanglingPageRankInputFormat.java
...ank/custom/CustomImprovedDanglingPageRankInputFormat.java
+47
-0
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/CustomPageWithRankOutFormat.java
.../danglingpagerank/custom/CustomPageWithRankOutFormat.java
+40
-0
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/types/VertexWithAdjacencyList.java
...anglingpagerank/custom/types/VertexWithAdjacencyList.java
+10
-10
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/types/VertexWithAdjacencyListComparator.java
...erank/custom/types/VertexWithAdjacencyListComparator.java
+14
-14
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/types/VertexWithAdjacencyListComparatorFactory.java
...ustom/types/VertexWithAdjacencyListComparatorFactory.java
+4
-4
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/types/VertexWithAdjacencyListSerializer.java
...erank/custom/types/VertexWithAdjacencyListSerializer.java
+13
-13
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/types/VertexWithAdjacencyListSerializerFactory.java
...ustom/types/VertexWithAdjacencyListSerializerFactory.java
+6
-6
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/types/VertexWithRank.java
...satable/danglingpagerank/custom/types/VertexWithRank.java
+10
-10
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/types/VertexWithRankAndDangling.java
...glingpagerank/custom/types/VertexWithRankAndDangling.java
+10
-10
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/types/VertexWithRankAndDanglingComparator.java
...ank/custom/types/VertexWithRankAndDanglingComparator.java
+14
-14
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/types/VertexWithRankAndDanglingComparatorFactory.java
...tom/types/VertexWithRankAndDanglingComparatorFactory.java
+4
-4
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/types/VertexWithRankAndDanglingSerializer.java
...ank/custom/types/VertexWithRankAndDanglingSerializer.java
+12
-12
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/types/VertexWithRankAndDanglingSerializerFactory.java
...tom/types/VertexWithRankAndDanglingSerializerFactory.java
+6
-6
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/types/VertexWithRankComparator.java
...nglingpagerank/custom/types/VertexWithRankComparator.java
+14
-14
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/types/VertexWithRankComparatorFactory.java
...agerank/custom/types/VertexWithRankComparatorFactory.java
+4
-4
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/types/VertexWithRankDanglingToVertexWithAdjacencyListPairComparatorFactory.java
...nglingToVertexWithAdjacencyListPairComparatorFactory.java
+87
-0
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/types/VertexWithRankDanglingToVertexWithRankPairComparatorFactory.java
...ithRankDanglingToVertexWithRankPairComparatorFactory.java
+87
-0
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/types/VertexWithRankSerializer.java
...nglingpagerank/custom/types/VertexWithRankSerializer.java
+12
-12
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/types/VertexWithRankSerializerFactory.java
...agerank/custom/types/VertexWithRankSerializerFactory.java
+6
-6
pact/pact-tests/src/test/java/eu/stratosphere/pact/test/iterative/DanglingPageRankITCase.java
...atosphere/pact/test/iterative/DanglingPageRankITCase.java
+2
-1
未找到文件。
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/AsciiLongArrayView.java
浏览文件 @
3fe087d9
...
@@ -129,10 +129,10 @@ public class AsciiLongArrayView {
...
@@ -129,10 +129,10 @@ public class AsciiLongArrayView {
}
}
}
}
public
double
elementAsDouble
()
{
//
public double elementAsDouble() {
String
token
=
new
String
(
buffer
,
tokenOffset
,
tokenNumBytes
,
Charsets
.
US_ASCII
);
//
String token = new String(buffer, tokenOffset, tokenNumBytes, Charsets.US_ASCII);
return
Double
.
valueOf
(
token
);
//
return Double.valueOf(token);
}
//
}
@Override
@Override
...
...
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/CompensatableDanglingPageRank.java
浏览文件 @
3fe087d9
...
@@ -111,7 +111,7 @@ public class CompensatableDanglingPageRank {
...
@@ -111,7 +111,7 @@ public class CompensatableDanglingPageRank {
// --------------- the inputs ---------------------
// --------------- the inputs ---------------------
// page rank input
// page rank input
JobInputVertex
pageWithRankInput
=
JobGraphUtils
.
createInput
(
DanglingPageGenerat
eRankInputFormat
.
class
,
JobInputVertex
pageWithRankInput
=
JobGraphUtils
.
createInput
(
ImprovedDanglingPag
eRankInputFormat
.
class
,
pageWithRankInputPath
,
"DanglingPageWithRankInput"
,
jobGraph
,
degreeOfParallelism
,
numSubTasksPerInstance
);
pageWithRankInputPath
,
"DanglingPageWithRankInput"
,
jobGraph
,
degreeOfParallelism
,
numSubTasksPerInstance
);
TaskConfig
pageWithRankInputConfig
=
new
TaskConfig
(
pageWithRankInput
.
getConfiguration
());
TaskConfig
pageWithRankInputConfig
=
new
TaskConfig
(
pageWithRankInput
.
getConfiguration
());
pageWithRankInputConfig
.
addOutputShipStrategy
(
ShipStrategyType
.
PARTITION_HASH
);
pageWithRankInputConfig
.
addOutputShipStrategy
(
ShipStrategyType
.
PARTITION_HASH
);
...
@@ -120,7 +120,7 @@ public class CompensatableDanglingPageRank {
...
@@ -120,7 +120,7 @@ public class CompensatableDanglingPageRank {
pageWithRankInputConfig
.
setStubParameter
(
"pageRank.numVertices"
,
String
.
valueOf
(
numVertices
));
pageWithRankInputConfig
.
setStubParameter
(
"pageRank.numVertices"
,
String
.
valueOf
(
numVertices
));
// edges as adjacency list
// edges as adjacency list
JobInputVertex
adjacencyListInput
=
JobGraphUtils
.
createInput
(
AdjacencyListInputFormat
.
class
,
JobInputVertex
adjacencyListInput
=
JobGraphUtils
.
createInput
(
Improved
AdjacencyListInputFormat
.
class
,
adjacencyListInputPath
,
"AdjancencyListInput"
,
jobGraph
,
degreeOfParallelism
,
numSubTasksPerInstance
);
adjacencyListInputPath
,
"AdjancencyListInput"
,
jobGraph
,
degreeOfParallelism
,
numSubTasksPerInstance
);
TaskConfig
adjacencyListInputConfig
=
new
TaskConfig
(
adjacencyListInput
.
getConfiguration
());
TaskConfig
adjacencyListInputConfig
=
new
TaskConfig
(
adjacencyListInput
.
getConfiguration
());
adjacencyListInputConfig
.
addOutputShipStrategy
(
ShipStrategyType
.
PARTITION_HASH
);
adjacencyListInputConfig
.
addOutputShipStrategy
(
ShipStrategyType
.
PARTITION_HASH
);
...
...
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/CustomCompensatableDanglingPageRank.java
0 → 100644
浏览文件 @
3fe087d9
/***********************************************************************************************************************
*
* Copyright (C) 2012 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom
;
import
eu.stratosphere.nephele.configuration.Configuration
;
import
eu.stratosphere.nephele.configuration.GlobalConfiguration
;
import
eu.stratosphere.nephele.io.DistributionPattern
;
import
eu.stratosphere.nephele.io.channels.ChannelType
;
import
eu.stratosphere.nephele.jobgraph.JobGraph
;
import
eu.stratosphere.nephele.jobgraph.JobInputVertex
;
import
eu.stratosphere.nephele.jobgraph.JobOutputVertex
;
import
eu.stratosphere.nephele.jobgraph.JobTaskVertex
;
import
eu.stratosphere.pact.common.io.FileOutputFormat
;
import
eu.stratosphere.pact.generic.types.TypeComparatorFactory
;
import
eu.stratosphere.pact.generic.types.TypePairComparatorFactory
;
import
eu.stratosphere.pact.generic.types.TypeSerializerFactory
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.DiffL1NormConvergenceCriterion
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types.VertexWithAdjacencyList
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types.VertexWithAdjacencyListComparatorFactory
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types.VertexWithAdjacencyListSerializerFactory
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types.VertexWithRank
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types.VertexWithRankAndDangling
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types.VertexWithRankAndDanglingComparatorFactory
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types.VertexWithRankAndDanglingSerializerFactory
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types.VertexWithRankComparatorFactory
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types.VertexWithRankDanglingToVertexWithAdjacencyListPairComparatorFactory
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types.VertexWithRankDanglingToVertexWithRankPairComparatorFactory
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types.VertexWithRankSerializerFactory
;
import
eu.stratosphere.pact.runtime.iterative.playing.JobGraphUtils
;
import
eu.stratosphere.pact.runtime.iterative.playing.PlayConstants
;
import
eu.stratosphere.pact.runtime.iterative.task.IterationHeadPactTask
;
import
eu.stratosphere.pact.runtime.iterative.task.IterationIntermediatePactTask
;
import
eu.stratosphere.pact.runtime.iterative.task.IterationTailPactTask
;
import
eu.stratosphere.pact.runtime.shipping.ShipStrategyType
;
import
eu.stratosphere.pact.runtime.task.BuildSecondCachedMatchDriver
;
import
eu.stratosphere.pact.runtime.task.CoGroupDriver
;
import
eu.stratosphere.pact.runtime.task.DriverStrategy
;
import
eu.stratosphere.pact.runtime.task.MapDriver
;
import
eu.stratosphere.pact.runtime.task.util.LocalStrategy
;
import
eu.stratosphere.pact.runtime.task.util.TaskConfig
;
public
class
CustomCompensatableDanglingPageRank
{
private
static
final
int
NUM_FILE_HANDLES_PER_SORT
=
64
;
private
static
final
float
SORT_SPILL_THRESHOLD
=
0.85f
;
private
static
TypeSerializerFactory
<
VertexWithRank
>
vertexWithRankSerializer
=
new
VertexWithRankSerializerFactory
();
private
static
TypeSerializerFactory
<
VertexWithRankAndDangling
>
vertexWithRankAndDanglingSerializer
=
new
VertexWithRankAndDanglingSerializerFactory
();
private
static
TypeSerializerFactory
<
VertexWithAdjacencyList
>
vertexWithAdjacencyListSerializer
=
new
VertexWithAdjacencyListSerializerFactory
();
private
static
TypeComparatorFactory
<
VertexWithRank
>
vertexWithRankComparator
=
new
VertexWithRankComparatorFactory
();
private
static
TypeComparatorFactory
<
VertexWithRankAndDangling
>
vertexWithRankAndDanglingComparator
=
new
VertexWithRankAndDanglingComparatorFactory
();
private
static
TypeComparatorFactory
<
VertexWithAdjacencyList
>
vertexWithAdjacencyListComparator
=
new
VertexWithAdjacencyListComparatorFactory
();
private
static
TypePairComparatorFactory
<
VertexWithRankAndDangling
,
VertexWithAdjacencyList
>
matchComparator
=
new
VertexWithRankDanglingToVertexWithAdjacencyListPairComparatorFactory
();
private
static
TypePairComparatorFactory
<
VertexWithRankAndDangling
,
VertexWithRank
>
coGroupComparator
=
new
VertexWithRankDanglingToVertexWithRankPairComparatorFactory
();
public
static
void
main
(
String
[]
args
)
throws
Exception
{
String
confPath
=
args
.
length
>=
6
?
confPath
=
args
[
5
]
:
PlayConstants
.
PLAY_DIR
+
"local-conf"
;
GlobalConfiguration
.
loadConfiguration
(
confPath
);
Configuration
conf
=
GlobalConfiguration
.
getConfiguration
();
JobGraph
jobGraph
=
getJobGraph
(
args
);
JobGraphUtils
.
submit
(
jobGraph
,
conf
);
}
public
static
JobGraph
getJobGraph
(
String
[]
args
)
throws
Exception
{
int
degreeOfParallelism
=
2
;
int
numSubTasksPerInstance
=
degreeOfParallelism
;
String
pageWithRankInputPath
=
"file://"
+
PlayConstants
.
PLAY_DIR
+
"test-inputs/danglingpagerank/pageWithRank"
;
String
adjacencyListInputPath
=
"file://"
+
PlayConstants
.
PLAY_DIR
+
"test-inputs/danglingpagerank/adjacencylists"
;
String
outputPath
=
"file:///tmp/stratosphere/iterations"
;
// String confPath = PlayConstants.PLAY_DIR + "local-conf";
int
minorConsumer
=
25
;
int
matchMemory
=
50
;
int
coGroupSortMemory
=
50
;
int
numIterations
=
25
;
long
numVertices
=
5
;
long
numDanglingVertices
=
1
;
String
failingWorkers
=
"1"
;
int
failingIteration
=
2
;
double
messageLoss
=
0.75
;
if
(
args
.
length
>=
15
)
{
degreeOfParallelism
=
Integer
.
parseInt
(
args
[
0
]);
numSubTasksPerInstance
=
Integer
.
parseInt
(
args
[
1
]);
pageWithRankInputPath
=
args
[
2
];
adjacencyListInputPath
=
args
[
3
];
outputPath
=
args
[
4
];
// confPath = args[5];
minorConsumer
=
Integer
.
parseInt
(
args
[
6
]);
matchMemory
=
Integer
.
parseInt
(
args
[
7
]);
coGroupSortMemory
=
Integer
.
parseInt
(
args
[
8
]);
numIterations
=
Integer
.
parseInt
(
args
[
9
]);
numVertices
=
Long
.
parseLong
(
args
[
10
]);
numDanglingVertices
=
Long
.
parseLong
(
args
[
11
]);
failingWorkers
=
args
[
12
];
failingIteration
=
Integer
.
parseInt
(
args
[
13
]);
messageLoss
=
Double
.
parseDouble
(
args
[
14
]);
}
JobGraph
jobGraph
=
new
JobGraph
(
"CompensatableDanglingPageRank"
);
// --------------- the inputs ---------------------
// page rank input
JobInputVertex
pageWithRankInput
=
JobGraphUtils
.
createInput
(
CustomImprovedDanglingPageRankInputFormat
.
class
,
pageWithRankInputPath
,
"DanglingPageWithRankInput"
,
jobGraph
,
degreeOfParallelism
,
numSubTasksPerInstance
);
TaskConfig
pageWithRankInputConfig
=
new
TaskConfig
(
pageWithRankInput
.
getConfiguration
());
pageWithRankInputConfig
.
addOutputShipStrategy
(
ShipStrategyType
.
PARTITION_HASH
);
pageWithRankInputConfig
.
setOutputComparator
(
vertexWithRankAndDanglingComparator
,
0
);
pageWithRankInputConfig
.
setOutputSerializer
(
vertexWithRankAndDanglingSerializer
);
pageWithRankInputConfig
.
setStubParameter
(
"pageRank.numVertices"
,
String
.
valueOf
(
numVertices
));
// edges as adjacency list
JobInputVertex
adjacencyListInput
=
JobGraphUtils
.
createInput
(
CustomImprovedAdjacencyListInputFormat
.
class
,
adjacencyListInputPath
,
"AdjancencyListInput"
,
jobGraph
,
degreeOfParallelism
,
numSubTasksPerInstance
);
TaskConfig
adjacencyListInputConfig
=
new
TaskConfig
(
adjacencyListInput
.
getConfiguration
());
adjacencyListInputConfig
.
addOutputShipStrategy
(
ShipStrategyType
.
PARTITION_HASH
);
adjacencyListInputConfig
.
setOutputSerializer
(
vertexWithAdjacencyListSerializer
);
adjacencyListInputConfig
.
setOutputComparator
(
vertexWithAdjacencyListComparator
,
0
);
// --------------- the head ---------------------
JobTaskVertex
head
=
JobGraphUtils
.
createTask
(
IterationHeadPactTask
.
class
,
"IterationHead"
,
jobGraph
,
degreeOfParallelism
,
numSubTasksPerInstance
);
TaskConfig
headConfig
=
new
TaskConfig
(
head
.
getConfiguration
());
// initial input / partial solution
headConfig
.
addInputToGroup
(
0
);
headConfig
.
setIterationHeadPartialSolutionInputIndex
(
0
);
headConfig
.
setInputSerializer
(
vertexWithRankAndDanglingSerializer
,
0
);
headConfig
.
setInputComparator
(
vertexWithRankAndDanglingComparator
,
0
);
headConfig
.
setInputLocalStrategy
(
0
,
LocalStrategy
.
SORT
);
headConfig
.
setMemoryInput
(
0
,
minorConsumer
*
JobGraphUtils
.
MEGABYTE
);
headConfig
.
setFilehandlesInput
(
0
,
NUM_FILE_HANDLES_PER_SORT
);
headConfig
.
setSpillingThresholdInput
(
0
,
SORT_SPILL_THRESHOLD
);
// back channel / iterations
headConfig
.
setBackChannelMemory
(
minorConsumer
*
JobGraphUtils
.
MEGABYTE
);
// output into iteration
headConfig
.
setOutputSerializer
(
vertexWithRankAndDanglingSerializer
);
headConfig
.
addOutputShipStrategy
(
ShipStrategyType
.
FORWARD
);
headConfig
.
addOutputShipStrategy
(
ShipStrategyType
.
FORWARD
);
// final output
TaskConfig
headFinalOutConfig
=
new
TaskConfig
(
new
Configuration
());
headFinalOutConfig
.
setOutputSerializer
(
vertexWithRankAndDanglingSerializer
);
headFinalOutConfig
.
addOutputShipStrategy
(
ShipStrategyType
.
FORWARD
);
headConfig
.
setIterationHeadFinalOutputConfig
(
headFinalOutConfig
);
// the sync
headConfig
.
setIterationHeadIndexOfSyncOutput
(
3
);
headConfig
.
setNumberOfIterations
(
numIterations
);
// the driver
headConfig
.
setDriver
(
MapDriver
.
class
);
headConfig
.
setDriverStrategy
(
DriverStrategy
.
MAP
);
headConfig
.
setStubClass
(
CustomCompensatingMap
.
class
);
headConfig
.
setStubParameter
(
"pageRank.numVertices"
,
String
.
valueOf
(
numVertices
));
headConfig
.
setStubParameter
(
"compensation.failingWorker"
,
failingWorkers
);
headConfig
.
setStubParameter
(
"compensation.failingIteration"
,
String
.
valueOf
(
failingIteration
));
headConfig
.
setStubParameter
(
"compensation.messageLoss"
,
String
.
valueOf
(
messageLoss
));
// --------------- the join ---------------------
JobTaskVertex
intermediate
=
JobGraphUtils
.
createTask
(
IterationIntermediatePactTask
.
class
,
"IterationIntermediate"
,
jobGraph
,
degreeOfParallelism
,
numSubTasksPerInstance
);
TaskConfig
intermediateConfig
=
new
TaskConfig
(
intermediate
.
getConfiguration
());
// intermediateConfig.setDriver(RepeatableHashjoinMatchDriverWithCachedBuildside.class);
intermediateConfig
.
setDriver
(
BuildSecondCachedMatchDriver
.
class
);
intermediateConfig
.
setDriverStrategy
(
DriverStrategy
.
HYBRIDHASH_BUILD_SECOND
);
intermediateConfig
.
setMemoryDriver
(
matchMemory
*
JobGraphUtils
.
MEGABYTE
);
intermediateConfig
.
addInputToGroup
(
0
);
intermediateConfig
.
addInputToGroup
(
1
);
intermediateConfig
.
setInputSerializer
(
vertexWithRankAndDanglingSerializer
,
0
);
intermediateConfig
.
setInputSerializer
(
vertexWithAdjacencyListSerializer
,
1
);
intermediateConfig
.
setDriverComparator
(
vertexWithRankAndDanglingComparator
,
0
);
intermediateConfig
.
setDriverComparator
(
vertexWithAdjacencyListComparator
,
1
);
intermediateConfig
.
setDriverPairComparator
(
matchComparator
);
intermediateConfig
.
setOutputSerializer
(
vertexWithRankSerializer
);
intermediateConfig
.
addOutputShipStrategy
(
ShipStrategyType
.
PARTITION_HASH
);
intermediateConfig
.
setOutputComparator
(
vertexWithRankComparator
,
0
);
intermediateConfig
.
setStubClass
(
CustomCompensatableDotProductMatch
.
class
);
intermediateConfig
.
setStubParameter
(
"pageRank.numVertices"
,
String
.
valueOf
(
numVertices
));
intermediateConfig
.
setStubParameter
(
"compensation.failingWorker"
,
failingWorkers
);
intermediateConfig
.
setStubParameter
(
"compensation.failingIteration"
,
String
.
valueOf
(
failingIteration
));
intermediateConfig
.
setStubParameter
(
"compensation.messageLoss"
,
String
.
valueOf
(
messageLoss
));
// ---------------- the tail (co group) --------------------
JobTaskVertex
tail
=
JobGraphUtils
.
createTask
(
IterationTailPactTask
.
class
,
"IterationTail"
,
jobGraph
,
degreeOfParallelism
,
numSubTasksPerInstance
);
TaskConfig
tailConfig
=
new
TaskConfig
(
tail
.
getConfiguration
());
// TODO we need to combine!
// inputs and driver
tailConfig
.
setDriver
(
CoGroupDriver
.
class
);
tailConfig
.
setDriverStrategy
(
DriverStrategy
.
CO_GROUP
);
tailConfig
.
addInputToGroup
(
0
);
tailConfig
.
addInputToGroup
(
1
);
tailConfig
.
setInputSerializer
(
vertexWithRankAndDanglingSerializer
,
0
);
tailConfig
.
setInputSerializer
(
vertexWithRankSerializer
,
1
);
tailConfig
.
setDriverComparator
(
vertexWithRankAndDanglingComparator
,
0
);
tailConfig
.
setDriverComparator
(
vertexWithRankComparator
,
1
);
tailConfig
.
setDriverPairComparator
(
coGroupComparator
);
tailConfig
.
setInputAsynchronouslyMaterialized
(
0
,
true
);
tailConfig
.
setInputMaterializationMemory
(
0
,
minorConsumer
*
JobGraphUtils
.
MEGABYTE
);
tailConfig
.
setInputLocalStrategy
(
1
,
LocalStrategy
.
SORT
);
tailConfig
.
setInputComparator
(
vertexWithRankComparator
,
1
);
tailConfig
.
setMemoryInput
(
1
,
coGroupSortMemory
*
JobGraphUtils
.
MEGABYTE
);
tailConfig
.
setFilehandlesInput
(
1
,
NUM_FILE_HANDLES_PER_SORT
);
tailConfig
.
setSpillingThresholdInput
(
1
,
SORT_SPILL_THRESHOLD
);
// output
tailConfig
.
addOutputShipStrategy
(
ShipStrategyType
.
FORWARD
);
tailConfig
.
setOutputSerializer
(
vertexWithRankAndDanglingSerializer
);
// the stub
tailConfig
.
setStubClass
(
CustomCompensatableDotProductCoGroup
.
class
);
tailConfig
.
setStubParameter
(
"pageRank.numVertices"
,
String
.
valueOf
(
numVertices
));
tailConfig
.
setStubParameter
(
"pageRank.numDanglingVertices"
,
String
.
valueOf
(
numDanglingVertices
));
tailConfig
.
setStubParameter
(
"compensation.failingWorker"
,
failingWorkers
);
tailConfig
.
setStubParameter
(
"compensation.failingIteration"
,
String
.
valueOf
(
failingIteration
));
tailConfig
.
setStubParameter
(
"compensation.messageLoss"
,
String
.
valueOf
(
messageLoss
));
// --------------- the output ---------------------
JobOutputVertex
output
=
JobGraphUtils
.
createFileOutput
(
jobGraph
,
"FinalOutput"
,
degreeOfParallelism
,
numSubTasksPerInstance
);
TaskConfig
outputConfig
=
new
TaskConfig
(
output
.
getConfiguration
());
outputConfig
.
addInputToGroup
(
0
);
outputConfig
.
setInputSerializer
(
vertexWithRankAndDanglingSerializer
,
0
);
outputConfig
.
setStubClass
(
CustomPageWithRankOutFormat
.
class
);
outputConfig
.
setStubParameter
(
FileOutputFormat
.
FILE_PARAMETER_KEY
,
outputPath
);
// --------------- the auxiliaries ---------------------
JobOutputVertex
fakeTailOutput
=
JobGraphUtils
.
createFakeOutput
(
jobGraph
,
"FakeTailOutput"
,
degreeOfParallelism
,
numSubTasksPerInstance
);
JobOutputVertex
sync
=
JobGraphUtils
.
createSync
(
jobGraph
,
degreeOfParallelism
);
TaskConfig
syncConfig
=
new
TaskConfig
(
sync
.
getConfiguration
());
syncConfig
.
setNumberOfIterations
(
numIterations
);
syncConfig
.
setConvergenceCriterion
(
DiffL1NormConvergenceCriterion
.
class
);
// --------------- the wiring ---------------------
JobGraphUtils
.
connect
(
pageWithRankInput
,
head
,
ChannelType
.
NETWORK
,
DistributionPattern
.
BIPARTITE
);
JobGraphUtils
.
connect
(
head
,
intermediate
,
ChannelType
.
INMEMORY
,
DistributionPattern
.
POINTWISE
);
intermediateConfig
.
setGateIterativeWithNumberOfEventsUntilInterrupt
(
0
,
1
);
JobGraphUtils
.
connect
(
adjacencyListInput
,
intermediate
,
ChannelType
.
NETWORK
,
DistributionPattern
.
BIPARTITE
);
JobGraphUtils
.
connect
(
head
,
tail
,
ChannelType
.
NETWORK
,
DistributionPattern
.
POINTWISE
);
JobGraphUtils
.
connect
(
intermediate
,
tail
,
ChannelType
.
NETWORK
,
DistributionPattern
.
BIPARTITE
);
tailConfig
.
setGateIterativeWithNumberOfEventsUntilInterrupt
(
0
,
1
);
tailConfig
.
setGateIterativeWithNumberOfEventsUntilInterrupt
(
1
,
degreeOfParallelism
);
JobGraphUtils
.
connect
(
head
,
output
,
ChannelType
.
INMEMORY
,
DistributionPattern
.
POINTWISE
);
JobGraphUtils
.
connect
(
tail
,
fakeTailOutput
,
ChannelType
.
INMEMORY
,
DistributionPattern
.
POINTWISE
);
JobGraphUtils
.
connect
(
head
,
sync
,
ChannelType
.
NETWORK
,
DistributionPattern
.
POINTWISE
);
fakeTailOutput
.
setVertexToShareInstancesWith
(
tail
);
tail
.
setVertexToShareInstancesWith
(
head
);
pageWithRankInput
.
setVertexToShareInstancesWith
(
head
);
adjacencyListInput
.
setVertexToShareInstancesWith
(
head
);
intermediate
.
setVertexToShareInstancesWith
(
head
);
output
.
setVertexToShareInstancesWith
(
head
);
sync
.
setVertexToShareInstancesWith
(
head
);
return
jobGraph
;
}
}
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/CustomCompensatableDotProductCoGroup.java
0 → 100644
浏览文件 @
3fe087d9
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom
;
import
eu.stratosphere.nephele.configuration.Configuration
;
import
eu.stratosphere.pact.common.stubs.Collector
;
import
eu.stratosphere.pact.generic.stub.AbstractStub
;
import
eu.stratosphere.pact.generic.stub.GenericCoGrouper
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.ConfigUtils
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.DiffL1NormConvergenceCriterion
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.PageRankStats
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.PageRankStatsAggregator
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types.VertexWithRank
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types.VertexWithRankAndDangling
;
import
eu.stratosphere.pact.runtime.iterative.concurrent.IterationContext
;
import
java.util.Iterator
;
import
java.util.Set
;
public
class
CustomCompensatableDotProductCoGroup
extends
AbstractStub
implements
GenericCoGrouper
<
VertexWithRankAndDangling
,
VertexWithRank
,
VertexWithRankAndDangling
>
{
private
VertexWithRankAndDangling
accumulator
=
new
VertexWithRankAndDangling
();
private
PageRankStatsAggregator
aggregator
=
(
PageRankStatsAggregator
)
new
DiffL1NormConvergenceCriterion
().
createAggregator
();
private
long
numVertices
;
private
long
numDanglingVertices
;
private
double
dampingFactor
;
private
double
danglingRankFactor
;
private
static
final
double
BETA
=
0.85
;
private
int
workerIndex
;
private
int
currentIteration
;
private
int
failingIteration
;
private
Set
<
Integer
>
failingWorkers
;
@Override
public
void
open
(
Configuration
parameters
)
throws
Exception
{
workerIndex
=
ConfigUtils
.
asInteger
(
"pact.parallel.task.id"
,
parameters
);
currentIteration
=
ConfigUtils
.
asInteger
(
"pact.iterations.currentIteration"
,
parameters
);
failingIteration
=
ConfigUtils
.
asInteger
(
"compensation.failingIteration"
,
parameters
);
failingWorkers
=
ConfigUtils
.
asIntSet
(
"compensation.failingWorker"
,
parameters
);
numVertices
=
ConfigUtils
.
asLong
(
"pageRank.numVertices"
,
parameters
);
numDanglingVertices
=
ConfigUtils
.
asLong
(
"pageRank.numDanglingVertices"
,
parameters
);
aggregator
.
reset
();
dampingFactor
=
(
1
d
-
BETA
)
/
(
double
)
numVertices
;
if
(
currentIteration
==
1
)
{
danglingRankFactor
=
BETA
*
(
double
)
numDanglingVertices
/
((
double
)
numVertices
*
(
double
)
numVertices
);
}
else
{
PageRankStats
previousAggregate
=
(
PageRankStats
)
IterationContext
.
instance
().
getGlobalAggregate
(
workerIndex
);
danglingRankFactor
=
BETA
*
previousAggregate
.
danglingRank
()
/
(
double
)
numVertices
;
}
}
@Override
public
void
coGroup
(
Iterator
<
VertexWithRankAndDangling
>
currentPageRankIterator
,
Iterator
<
VertexWithRank
>
partialRanks
,
Collector
<
VertexWithRankAndDangling
>
collector
)
{
if
(!
currentPageRankIterator
.
hasNext
())
{
long
missingVertex
=
partialRanks
.
next
().
getVertexID
();
throw
new
IllegalStateException
(
"No current page rank for vertex ["
+
missingVertex
+
"]!"
);
}
VertexWithRankAndDangling
currentPageRank
=
currentPageRankIterator
.
next
();
long
edges
=
0
;
double
summedRank
=
0
;
while
(
partialRanks
.
hasNext
())
{
summedRank
+=
partialRanks
.
next
().
getRank
();
edges
++;
}
double
rank
=
BETA
*
summedRank
+
dampingFactor
+
danglingRankFactor
;
double
currentRank
=
currentPageRank
.
getRank
();
boolean
isDangling
=
currentPageRank
.
isDangling
();
double
danglingRankToAggregate
=
isDangling
?
rank
:
0
;
long
danglingVerticesToAggregate
=
isDangling
?
1
:
0
;
double
diff
=
Math
.
abs
(
currentRank
-
rank
);
aggregator
.
aggregate
(
diff
,
rank
,
danglingRankToAggregate
,
danglingVerticesToAggregate
,
1
,
edges
,
summedRank
,
0
);
accumulator
.
setVertexID
(
currentPageRank
.
getVertexID
());
accumulator
.
setRank
(
rank
);
accumulator
.
setDangling
(
isDangling
);
collector
.
collect
(
accumulator
);
}
@Override
public
void
close
()
throws
Exception
{
if
(
currentIteration
==
failingIteration
&&
failingWorkers
.
contains
(
workerIndex
))
{
aggregator
.
reset
();
}
IterationContext
.
instance
().
setAggregate
(
workerIndex
,
aggregator
.
getAggregate
());
}
}
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/CustomCompensatableDotProductMatch.java
0 → 100644
浏览文件 @
3fe087d9
/***********************************************************************************************************************
*
* Copyright (C) 2012 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom
;
import
eu.stratosphere.nephele.configuration.Configuration
;
import
eu.stratosphere.pact.common.stubs.Collector
;
import
eu.stratosphere.pact.generic.stub.AbstractStub
;
import
eu.stratosphere.pact.generic.stub.GenericMatcher
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.ConfigUtils
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types.VertexWithAdjacencyList
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types.VertexWithRank
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types.VertexWithRankAndDangling
;
import
java.util.Random
;
import
java.util.Set
;
public
class
CustomCompensatableDotProductMatch
extends
AbstractStub
implements
GenericMatcher
<
VertexWithRankAndDangling
,
VertexWithAdjacencyList
,
VertexWithRank
>
{
private
VertexWithRank
record
=
new
VertexWithRank
();
private
Random
random
=
new
Random
();
private
double
messageLoss
;
private
boolean
isFailure
;
@Override
public
void
open
(
Configuration
parameters
)
throws
Exception
{
int
workerIndex
=
ConfigUtils
.
asInteger
(
"pact.parallel.task.id"
,
parameters
);
int
currentIteration
=
ConfigUtils
.
asInteger
(
"pact.iterations.currentIteration"
,
parameters
);
int
failingIteration
=
ConfigUtils
.
asInteger
(
"compensation.failingIteration"
,
parameters
);
Set
<
Integer
>
failingWorkers
=
ConfigUtils
.
asIntSet
(
"compensation.failingWorker"
,
parameters
);
isFailure
=
currentIteration
==
failingIteration
&&
failingWorkers
.
contains
(
workerIndex
);
messageLoss
=
ConfigUtils
.
asDouble
(
"compensation.messageLoss"
,
parameters
);
}
@Override
public
void
match
(
VertexWithRankAndDangling
pageWithRank
,
VertexWithAdjacencyList
adjacencyList
,
Collector
<
VertexWithRank
>
collector
)
throws
Exception
{
double
rank
=
pageWithRank
.
getRank
();
long
[]
adjacentNeighbors
=
adjacencyList
.
getTargets
();
int
numNeighbors
=
adjacencyList
.
getNumTargets
();
double
rankToDistribute
=
rank
/
(
double
)
numNeighbors
;
record
.
setRank
(
rankToDistribute
);
for
(
int
n
=
0
;
n
<
numNeighbors
;
n
++)
{
record
.
setVertexID
(
adjacentNeighbors
[
n
]);
if
(
isFailure
)
{
if
(
random
.
nextDouble
()
>=
messageLoss
)
{
collector
.
collect
(
record
);
}
}
else
{
collector
.
collect
(
record
);
}
}
}
}
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/CustomCompensatingMap.java
0 → 100644
浏览文件 @
3fe087d9
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom
;
import
eu.stratosphere.nephele.configuration.Configuration
;
import
eu.stratosphere.pact.common.stubs.Collector
;
import
eu.stratosphere.pact.generic.stub.AbstractStub
;
import
eu.stratosphere.pact.generic.stub.GenericMapper
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.ConfigUtils
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.PageRankStats
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types.VertexWithRankAndDangling
;
import
eu.stratosphere.pact.runtime.iterative.concurrent.IterationContext
;
import
java.util.Set
;
public
class
CustomCompensatingMap
extends
AbstractStub
implements
GenericMapper
<
VertexWithRankAndDangling
,
VertexWithRankAndDangling
>
{
private
boolean
isFailureIteration
;
private
boolean
isFailingWorker
;
private
double
uniformRank
;
private
double
rescaleFactor
;
@Override
public
void
open
(
Configuration
parameters
)
throws
Exception
{
int
currentIteration
=
ConfigUtils
.
asInteger
(
"pact.iterations.currentIteration"
,
parameters
);
int
failingIteration
=
ConfigUtils
.
asInteger
(
"compensation.failingIteration"
,
parameters
);
isFailureIteration
=
currentIteration
==
failingIteration
+
1
;
int
workerIndex
=
ConfigUtils
.
asInteger
(
"pact.parallel.task.id"
,
parameters
);
Set
<
Integer
>
failingWorkers
=
ConfigUtils
.
asIntSet
(
"compensation.failingWorker"
,
parameters
);
isFailingWorker
=
failingWorkers
.
contains
(
workerIndex
);
long
numVertices
=
ConfigUtils
.
asLong
(
"pageRank.numVertices"
,
parameters
);
if
(
currentIteration
>
1
)
{
PageRankStats
stats
=
(
PageRankStats
)
IterationContext
.
instance
().
getGlobalAggregate
(
workerIndex
);
uniformRank
=
1
d
/
(
double
)
numVertices
;
double
lostMassFactor
=
(
numVertices
-
stats
.
numVertices
())
/
(
double
)
numVertices
;
rescaleFactor
=
(
1
-
lostMassFactor
)
/
stats
.
rank
();
}
}
@Override
public
void
map
(
VertexWithRankAndDangling
pageWithRank
,
Collector
<
VertexWithRankAndDangling
>
out
)
throws
Exception
{
if
(
isFailureIteration
)
{
double
rank
=
pageWithRank
.
getRank
();
if
(
isFailingWorker
)
{
pageWithRank
.
setRank
(
uniformRank
);
}
else
{
pageWithRank
.
setRank
(
rank
*
rescaleFactor
);
}
}
out
.
collect
(
pageWithRank
);
}
}
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/CustomImprovedAdjacencyListInputFormat.java
0 → 100644
浏览文件 @
3fe087d9
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom
;
import
eu.stratosphere.pact.generic.io.DelimitedInputFormat
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.AsciiLongArrayView
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types.VertexWithAdjacencyList
;
public
class
CustomImprovedAdjacencyListInputFormat
extends
DelimitedInputFormat
<
VertexWithAdjacencyList
>
{
private
final
AsciiLongArrayView
arrayView
=
new
AsciiLongArrayView
();
@Override
public
boolean
readRecord
(
VertexWithAdjacencyList
target
,
byte
[]
bytes
,
int
offset
,
int
numBytes
)
{
if
(
numBytes
==
0
)
{
return
false
;
}
arrayView
.
set
(
bytes
,
offset
,
numBytes
);
long
[]
list
=
target
.
getTargets
();
try
{
int
pos
=
0
;
while
(
arrayView
.
next
())
{
if
(
pos
==
0
)
{
target
.
setVertexID
(
arrayView
.
element
());
}
else
{
if
(
list
.
length
<=
pos
-
1
)
{
list
=
new
long
[
list
.
length
<
16
?
16
:
list
.
length
*
2
];
target
.
setTargets
(
list
);
}
list
[
pos
-
1
]
=
arrayView
.
element
();
}
pos
++;
}
target
.
setNumTargets
(
pos
-
1
);
}
catch
(
RuntimeException
e
)
{
throw
new
RuntimeException
(
"Error parsing: "
+
arrayView
.
toString
(),
e
);
}
return
true
;
}
}
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/CustomImprovedDanglingPageRankInputFormat.java
0 → 100644
浏览文件 @
3fe087d9
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom
;
import
eu.stratosphere.nephele.configuration.Configuration
;
import
eu.stratosphere.pact.generic.io.DelimitedInputFormat
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.ConfigUtils
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.AsciiLongArrayView
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types.VertexWithRankAndDangling
;
public
class
CustomImprovedDanglingPageRankInputFormat
extends
DelimitedInputFormat
<
VertexWithRankAndDangling
>
{
private
AsciiLongArrayView
arrayView
=
new
AsciiLongArrayView
();
private
static
final
long
DANGLING_MARKER
=
1
l
;
private
double
initialRank
;
@Override
public
void
configure
(
Configuration
parameters
)
{
long
numVertices
=
ConfigUtils
.
asLong
(
"pageRank.numVertices"
,
parameters
);
initialRank
=
1.0
/
numVertices
;
super
.
configure
(
parameters
);
}
@Override
public
boolean
readRecord
(
VertexWithRankAndDangling
target
,
byte
[]
bytes
,
int
offset
,
int
numBytes
)
{
arrayView
.
set
(
bytes
,
offset
,
numBytes
);
try
{
arrayView
.
next
();
target
.
setVertexID
(
arrayView
.
element
());
if
(
arrayView
.
next
())
{
target
.
setDangling
(
arrayView
.
element
()
==
DANGLING_MARKER
);
}
else
{
target
.
setDangling
(
false
);
}
}
catch
(
NumberFormatException
e
)
{
throw
new
RuntimeException
(
"Error parsing "
+
arrayView
.
toString
(),
e
);
}
target
.
setRank
(
initialRank
);
return
true
;
}
}
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/CustomPageWithRankOutFormat.java
0 → 100644
浏览文件 @
3fe087d9
/***********************************************************************************************************************
*
* Copyright (C) 2012 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom
;
import
com.google.common.base.Charsets
;
import
eu.stratosphere.pact.generic.io.FileOutputFormat
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types.VertexWithRankAndDangling
;
import
java.io.IOException
;
public
class
CustomPageWithRankOutFormat
extends
FileOutputFormat
<
VertexWithRankAndDangling
>
{
private
final
StringBuilder
buffer
=
new
StringBuilder
();
@Override
public
void
writeRecord
(
VertexWithRankAndDangling
record
)
throws
IOException
{
buffer
.
setLength
(
0
);
buffer
.
append
(
record
.
getVertexID
());
buffer
.
append
(
'\t'
);
buffer
.
append
(
record
.
getRank
());
buffer
.
append
(
'\n'
);
byte
[]
bytes
=
buffer
.
toString
().
getBytes
(
Charsets
.
UTF_8
);
stream
.
write
(
bytes
);
}
}
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
types/Node
WithAdjacencyList.java
→
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
custom/types/Vertex
WithAdjacencyList.java
浏览文件 @
3fe087d9
...
@@ -12,38 +12,38 @@
...
@@ -12,38 +12,38 @@
* specific language governing permissions and limitations under the License.
* specific language governing permissions and limitations under the License.
*
*
**********************************************************************************************************************/
**********************************************************************************************************************/
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.types
;
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.
custom.
types
;
/**
/**
*
*
*/
*/
public
final
class
Node
WithAdjacencyList
{
public
final
class
Vertex
WithAdjacencyList
{
private
static
final
long
[]
EMPTY
=
new
long
[
0
];
private
static
final
long
[]
EMPTY
=
new
long
[
0
];
private
long
nodeId
;
private
long
vertexID
;
private
long
[]
targets
;
private
long
[]
targets
;
private
int
numTargets
;
private
int
numTargets
;
public
Node
WithAdjacencyList
()
{
public
Vertex
WithAdjacencyList
()
{
this
.
targets
=
EMPTY
;
this
.
targets
=
EMPTY
;
}
}
public
NodeWithAdjacencyList
(
long
nodeId
,
long
[]
targets
)
{
public
VertexWithAdjacencyList
(
long
vertexID
,
long
[]
targets
)
{
this
.
nodeId
=
nodeId
;
this
.
vertexID
=
vertexID
;
this
.
targets
=
targets
;
this
.
targets
=
targets
;
}
}
public
long
get
NodeId
()
{
public
long
get
VertexID
()
{
return
nodeId
;
return
vertexID
;
}
}
public
void
set
NodeId
(
long
nodeId
)
{
public
void
set
VertexID
(
long
vertexID
)
{
this
.
nodeId
=
nodeId
;
this
.
vertexID
=
vertexID
;
}
}
public
long
[]
getTargets
()
{
public
long
[]
getTargets
()
{
...
...
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
types/NodeWithRankAndDangling
Comparator.java
→
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
custom/types/VertexWithAdjacencyList
Comparator.java
浏览文件 @
3fe087d9
...
@@ -12,7 +12,7 @@
...
@@ -12,7 +12,7 @@
* specific language governing permissions and limitations under the License.
* specific language governing permissions and limitations under the License.
*
*
**********************************************************************************************************************/
**********************************************************************************************************************/
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.types
;
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.
custom.
types
;
import
java.io.IOException
;
import
java.io.IOException
;
...
@@ -23,29 +23,29 @@ import eu.stratosphere.pact.generic.types.TypeComparator;
...
@@ -23,29 +23,29 @@ import eu.stratosphere.pact.generic.types.TypeComparator;
/**
/**
*
*
*/
*/
public
final
class
NodeWithRankAndDanglingComparator
implements
TypeComparator
<
NodeWithRankAndDangling
>
{
public
final
class
VertexWithAdjacencyListComparator
implements
TypeComparator
<
VertexWithAdjacencyList
>
{
private
long
reference
;
private
long
reference
;
@Override
@Override
public
int
hash
(
NodeWithRankAndDangling
record
)
{
public
int
hash
(
VertexWithAdjacencyList
record
)
{
final
long
value
=
record
.
get
NodeId
();
final
long
value
=
record
.
get
VertexID
();
return
43
+
(
int
)
(
value
^
value
>>>
32
);
return
43
+
(
int
)
(
value
^
value
>>>
32
);
}
}
@Override
@Override
public
void
setReference
(
NodeWithRankAndDangling
toCompare
)
{
public
void
setReference
(
VertexWithAdjacencyList
toCompare
)
{
this
.
reference
=
toCompare
.
get
NodeId
();
this
.
reference
=
toCompare
.
get
VertexID
();
}
}
@Override
@Override
public
boolean
equalToReference
(
NodeWithRankAndDangling
candidate
)
{
public
boolean
equalToReference
(
VertexWithAdjacencyList
candidate
)
{
return
candidate
.
get
NodeId
()
==
this
.
reference
;
return
candidate
.
get
VertexID
()
==
this
.
reference
;
}
}
@Override
@Override
public
int
compareToReference
(
TypeComparator
<
NodeWithRankAndDangling
>
referencedComparator
)
{
public
int
compareToReference
(
TypeComparator
<
VertexWithAdjacencyList
>
referencedComparator
)
{
NodeWithRankAndDanglingComparator
comp
=
(
NodeWithRankAndDangling
Comparator
)
referencedComparator
;
VertexWithAdjacencyListComparator
comp
=
(
VertexWithAdjacencyList
Comparator
)
referencedComparator
;
final
long
diff
=
comp
.
reference
-
this
.
reference
;
final
long
diff
=
comp
.
reference
-
this
.
reference
;
return
diff
<
0
?
-
1
:
diff
>
0
?
1
:
0
;
return
diff
<
0
?
-
1
:
diff
>
0
?
1
:
0
;
}
}
...
@@ -72,8 +72,8 @@ public final class NodeWithRankAndDanglingComparator implements TypeComparator<N
...
@@ -72,8 +72,8 @@ public final class NodeWithRankAndDanglingComparator implements TypeComparator<N
}
}
@Override
@Override
public
void
putNormalizedKey
(
NodeWithRankAndDangling
record
,
byte
[]
target
,
int
offset
,
int
len
)
{
public
void
putNormalizedKey
(
VertexWithAdjacencyList
record
,
byte
[]
target
,
int
offset
,
int
len
)
{
final
long
value
=
record
.
get
NodeId
();
final
long
value
=
record
.
get
VertexID
();
if
(
len
==
8
)
{
if
(
len
==
8
)
{
// default case, full normalized key
// default case, full normalized key
...
@@ -122,7 +122,7 @@ public final class NodeWithRankAndDanglingComparator implements TypeComparator<N
...
@@ -122,7 +122,7 @@ public final class NodeWithRankAndDanglingComparator implements TypeComparator<N
}
}
@Override
@Override
public
NodeWithRankAndDangling
Comparator
duplicate
()
{
public
VertexWithAdjacencyList
Comparator
duplicate
()
{
return
new
NodeWithRankAndDangling
Comparator
();
return
new
VertexWithAdjacencyList
Comparator
();
}
}
}
}
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
types/NodeWithRankAndDangling
ComparatorFactory.java
→
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
custom/types/VertexWithAdjacencyList
ComparatorFactory.java
浏览文件 @
3fe087d9
...
@@ -12,7 +12,7 @@
...
@@ -12,7 +12,7 @@
* specific language governing permissions and limitations under the License.
* specific language governing permissions and limitations under the License.
*
*
**********************************************************************************************************************/
**********************************************************************************************************************/
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.types
;
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.
custom.
types
;
import
eu.stratosphere.nephele.configuration.Configuration
;
import
eu.stratosphere.nephele.configuration.Configuration
;
import
eu.stratosphere.pact.generic.types.TypeComparatorFactory
;
import
eu.stratosphere.pact.generic.types.TypeComparatorFactory
;
...
@@ -20,7 +20,7 @@ import eu.stratosphere.pact.generic.types.TypeComparatorFactory;
...
@@ -20,7 +20,7 @@ import eu.stratosphere.pact.generic.types.TypeComparatorFactory;
/**
/**
*
*
*/
*/
public
final
class
NodeWithRankAndDanglingComparatorFactory
implements
TypeComparatorFactory
<
NodeWithRankAndDangling
>
{
public
final
class
VertexWithAdjacencyListComparatorFactory
implements
TypeComparatorFactory
<
VertexWithAdjacencyList
>
{
@Override
@Override
public
void
writeParametersToConfig
(
Configuration
config
)
{}
public
void
writeParametersToConfig
(
Configuration
config
)
{}
...
@@ -29,7 +29,7 @@ public final class NodeWithRankAndDanglingComparatorFactory implements TypeCompa
...
@@ -29,7 +29,7 @@ public final class NodeWithRankAndDanglingComparatorFactory implements TypeCompa
public
void
readParametersFromConfig
(
Configuration
config
,
ClassLoader
cl
)
throws
ClassNotFoundException
{}
public
void
readParametersFromConfig
(
Configuration
config
,
ClassLoader
cl
)
throws
ClassNotFoundException
{}
@Override
@Override
public
NodeWithRankAndDangling
Comparator
createComparator
()
{
public
VertexWithAdjacencyList
Comparator
createComparator
()
{
return
new
NodeWithRankAndDangling
Comparator
();
return
new
VertexWithAdjacencyList
Comparator
();
}
}
}
}
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
types/Node
WithAdjacencyListSerializer.java
→
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
custom/types/Vertex
WithAdjacencyListSerializer.java
浏览文件 @
3fe087d9
...
@@ -12,7 +12,7 @@
...
@@ -12,7 +12,7 @@
* specific language governing permissions and limitations under the License.
* specific language governing permissions and limitations under the License.
*
*
**********************************************************************************************************************/
**********************************************************************************************************************/
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.types
;
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.
custom.
types
;
import
java.io.IOException
;
import
java.io.IOException
;
...
@@ -24,32 +24,32 @@ import eu.stratosphere.pact.generic.types.TypeSerializer;
...
@@ -24,32 +24,32 @@ import eu.stratosphere.pact.generic.types.TypeSerializer;
/**
/**
*
*
*/
*/
public
final
class
NodeWithAdjacencyListSerializer
implements
TypeSerializer
<
Node
WithAdjacencyList
>
{
public
final
class
VertexWithAdjacencyListSerializer
implements
TypeSerializer
<
Vertex
WithAdjacencyList
>
{
@Override
@Override
public
Node
WithAdjacencyList
createInstance
()
{
public
Vertex
WithAdjacencyList
createInstance
()
{
return
new
Node
WithAdjacencyList
();
return
new
Vertex
WithAdjacencyList
();
}
}
@Override
@Override
public
NodeWithAdjacencyList
createCopy
(
Node
WithAdjacencyList
from
)
{
public
VertexWithAdjacencyList
createCopy
(
Vertex
WithAdjacencyList
from
)
{
long
[]
targets
=
new
long
[
from
.
getTargets
().
length
];
long
[]
targets
=
new
long
[
from
.
getTargets
().
length
];
System
.
arraycopy
(
from
.
getTargets
(),
0
,
targets
,
0
,
targets
.
length
);
System
.
arraycopy
(
from
.
getTargets
(),
0
,
targets
,
0
,
targets
.
length
);
NodeWithAdjacencyList
copy
=
new
Node
WithAdjacencyList
();
VertexWithAdjacencyList
copy
=
new
Vertex
WithAdjacencyList
();
copy
.
set
NodeId
(
from
.
getNodeId
());
copy
.
set
VertexID
(
from
.
getVertexID
());
copy
.
setNumTargets
(
from
.
getNumTargets
());
copy
.
setNumTargets
(
from
.
getNumTargets
());
copy
.
setTargets
(
targets
);
copy
.
setTargets
(
targets
);
return
copy
;
return
copy
;
}
}
@Override
@Override
public
void
copyTo
(
NodeWithAdjacencyList
from
,
Node
WithAdjacencyList
to
)
{
public
void
copyTo
(
VertexWithAdjacencyList
from
,
Vertex
WithAdjacencyList
to
)
{
if
(
to
.
getTargets
().
length
<
from
.
getTargets
().
length
)
{
if
(
to
.
getTargets
().
length
<
from
.
getTargets
().
length
)
{
to
.
setTargets
(
new
long
[
from
.
getTargets
().
length
]);
to
.
setTargets
(
new
long
[
from
.
getTargets
().
length
]);
}
}
to
.
set
NodeId
(
from
.
getNodeId
());
to
.
set
VertexID
(
from
.
getVertexID
());
to
.
setNumTargets
(
from
.
getNumTargets
());
to
.
setNumTargets
(
from
.
getNumTargets
());
System
.
arraycopy
(
from
.
getTargets
(),
0
,
to
.
getTargets
(),
0
,
from
.
getNumTargets
());
System
.
arraycopy
(
from
.
getTargets
(),
0
,
to
.
getTargets
(),
0
,
from
.
getNumTargets
());
}
}
...
@@ -60,8 +60,8 @@ public final class NodeWithAdjacencyListSerializer implements TypeSerializer<Nod
...
@@ -60,8 +60,8 @@ public final class NodeWithAdjacencyListSerializer implements TypeSerializer<Nod
}
}
@Override
@Override
public
void
serialize
(
Node
WithAdjacencyList
record
,
DataOutputView
target
)
throws
IOException
{
public
void
serialize
(
Vertex
WithAdjacencyList
record
,
DataOutputView
target
)
throws
IOException
{
target
.
writeLong
(
record
.
get
NodeId
());
target
.
writeLong
(
record
.
get
VertexID
());
final
long
[]
targets
=
record
.
getTargets
();
final
long
[]
targets
=
record
.
getTargets
();
final
int
numTargets
=
record
.
getNumTargets
();
final
int
numTargets
=
record
.
getNumTargets
();
...
@@ -73,8 +73,8 @@ public final class NodeWithAdjacencyListSerializer implements TypeSerializer<Nod
...
@@ -73,8 +73,8 @@ public final class NodeWithAdjacencyListSerializer implements TypeSerializer<Nod
}
}
@Override
@Override
public
void
deserialize
(
Node
WithAdjacencyList
target
,
DataInputView
source
)
throws
IOException
{
public
void
deserialize
(
Vertex
WithAdjacencyList
target
,
DataInputView
source
)
throws
IOException
{
target
.
set
NodeId
(
source
.
readLong
());
target
.
set
VertexID
(
source
.
readLong
());
final
int
numTargets
=
source
.
readInt
();
final
int
numTargets
=
source
.
readInt
();
long
[]
targets
=
target
.
getTargets
();
long
[]
targets
=
target
.
getTargets
();
...
...
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
types/NodeWithRankAndDangling
SerializerFactory.java
→
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
custom/types/VertexWithAdjacencyList
SerializerFactory.java
浏览文件 @
3fe087d9
...
@@ -12,7 +12,7 @@
...
@@ -12,7 +12,7 @@
* specific language governing permissions and limitations under the License.
* specific language governing permissions and limitations under the License.
*
*
**********************************************************************************************************************/
**********************************************************************************************************************/
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.types
;
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.
custom.
types
;
import
eu.stratosphere.nephele.configuration.Configuration
;
import
eu.stratosphere.nephele.configuration.Configuration
;
import
eu.stratosphere.pact.generic.types.TypeSerializerFactory
;
import
eu.stratosphere.pact.generic.types.TypeSerializerFactory
;
...
@@ -20,9 +20,9 @@ import eu.stratosphere.pact.generic.types.TypeSerializerFactory;
...
@@ -20,9 +20,9 @@ import eu.stratosphere.pact.generic.types.TypeSerializerFactory;
/**
/**
*
*
*/
*/
public
final
class
NodeWithRankAndDanglingSerializerFactory
implements
TypeSerializerFactory
<
NodeWithRankAndDangling
>
{
public
final
class
VertexWithAdjacencyListSerializerFactory
implements
TypeSerializerFactory
<
VertexWithAdjacencyList
>
{
private
static
final
NodeWithRankAndDanglingSerializer
INSTANCE
=
new
NodeWithRankAndDangling
Serializer
();
private
static
final
VertexWithAdjacencyListSerializer
INSTANCE
=
new
VertexWithAdjacencyList
Serializer
();
@Override
@Override
public
void
writeParametersToConfig
(
Configuration
config
)
{}
public
void
writeParametersToConfig
(
Configuration
config
)
{}
...
@@ -31,12 +31,12 @@ public final class NodeWithRankAndDanglingSerializerFactory implements TypeSeria
...
@@ -31,12 +31,12 @@ public final class NodeWithRankAndDanglingSerializerFactory implements TypeSeria
public
void
readParametersFromConfig
(
Configuration
config
,
ClassLoader
cl
)
throws
ClassNotFoundException
{}
public
void
readParametersFromConfig
(
Configuration
config
,
ClassLoader
cl
)
throws
ClassNotFoundException
{}
@Override
@Override
public
NodeWithRankAndDangling
Serializer
getSerializer
()
{
public
VertexWithAdjacencyList
Serializer
getSerializer
()
{
return
INSTANCE
;
return
INSTANCE
;
}
}
@Override
@Override
public
Class
<
NodeWithRankAndDangling
>
getDataType
()
{
public
Class
<
VertexWithAdjacencyList
>
getDataType
()
{
return
NodeWithRankAndDangling
.
class
;
return
VertexWithAdjacencyList
.
class
;
}
}
}
}
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
types/Node
WithRank.java
→
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
custom/types/Vertex
WithRank.java
浏览文件 @
3fe087d9
...
@@ -12,34 +12,34 @@
...
@@ -12,34 +12,34 @@
* specific language governing permissions and limitations under the License.
* specific language governing permissions and limitations under the License.
*
*
**********************************************************************************************************************/
**********************************************************************************************************************/
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.types
;
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.
custom.
types
;
/**
/**
*
*
*/
*/
public
final
class
Node
WithRank
{
public
final
class
Vertex
WithRank
{
private
long
nodeId
;
private
long
vertexID
;
private
double
rank
;
private
double
rank
;
public
Node
WithRank
()
{
public
Vertex
WithRank
()
{
}
}
public
NodeWithRank
(
long
nodeId
,
double
rank
)
{
public
VertexWithRank
(
long
vertexID
,
double
rank
)
{
this
.
nodeId
=
nodeId
;
this
.
vertexID
=
vertexID
;
this
.
rank
=
rank
;
this
.
rank
=
rank
;
}
}
public
long
get
NodeId
()
{
public
long
get
VertexID
()
{
return
nodeId
;
return
vertexID
;
}
}
public
void
set
NodeId
(
long
nodeId
)
{
public
void
set
VertexID
(
long
vertexID
)
{
this
.
nodeId
=
nodeId
;
this
.
vertexID
=
vertexID
;
}
}
public
double
getRank
()
{
public
double
getRank
()
{
...
...
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
types/Node
WithRankAndDangling.java
→
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
custom/types/Vertex
WithRankAndDangling.java
浏览文件 @
3fe087d9
...
@@ -12,37 +12,37 @@
...
@@ -12,37 +12,37 @@
* specific language governing permissions and limitations under the License.
* specific language governing permissions and limitations under the License.
*
*
**********************************************************************************************************************/
**********************************************************************************************************************/
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.types
;
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.
custom.
types
;
/**
/**
*
*
*/
*/
public
final
class
Node
WithRankAndDangling
{
public
final
class
Vertex
WithRankAndDangling
{
private
long
nodeId
;
private
long
vertexID
;
private
double
rank
;
private
double
rank
;
private
boolean
dangling
;
private
boolean
dangling
;
public
Node
WithRankAndDangling
()
{
public
Vertex
WithRankAndDangling
()
{
}
}
public
NodeWithRankAndDangling
(
long
nodeId
,
double
rank
,
boolean
dangling
)
{
public
VertexWithRankAndDangling
(
long
vertexID
,
double
rank
,
boolean
dangling
)
{
this
.
nodeId
=
nodeId
;
this
.
vertexID
=
vertexID
;
this
.
rank
=
rank
;
this
.
rank
=
rank
;
this
.
dangling
=
dangling
;
this
.
dangling
=
dangling
;
}
}
public
long
get
NodeId
()
{
public
long
get
VertexID
()
{
return
nodeId
;
return
vertexID
;
}
}
public
void
set
NodeId
(
long
nodeId
)
{
public
void
set
VertexID
(
long
vertexID
)
{
this
.
nodeId
=
nodeId
;
this
.
vertexID
=
vertexID
;
}
}
public
double
getRank
()
{
public
double
getRank
()
{
...
...
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
types/NodeWithAdjacencyList
Comparator.java
→
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
custom/types/VertexWithRankAndDangling
Comparator.java
浏览文件 @
3fe087d9
...
@@ -12,7 +12,7 @@
...
@@ -12,7 +12,7 @@
* specific language governing permissions and limitations under the License.
* specific language governing permissions and limitations under the License.
*
*
**********************************************************************************************************************/
**********************************************************************************************************************/
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.types
;
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.
custom.
types
;
import
java.io.IOException
;
import
java.io.IOException
;
...
@@ -23,29 +23,29 @@ import eu.stratosphere.pact.generic.types.TypeComparator;
...
@@ -23,29 +23,29 @@ import eu.stratosphere.pact.generic.types.TypeComparator;
/**
/**
*
*
*/
*/
public
final
class
NodeWithAdjacencyListComparator
implements
TypeComparator
<
NodeWithAdjacencyList
>
{
public
final
class
VertexWithRankAndDanglingComparator
implements
TypeComparator
<
VertexWithRankAndDangling
>
{
private
long
reference
;
private
long
reference
;
@Override
@Override
public
int
hash
(
NodeWithAdjacencyList
record
)
{
public
int
hash
(
VertexWithRankAndDangling
record
)
{
final
long
value
=
record
.
get
NodeId
();
final
long
value
=
record
.
get
VertexID
();
return
43
+
(
int
)
(
value
^
value
>>>
32
);
return
43
+
(
int
)
(
value
^
value
>>>
32
);
}
}
@Override
@Override
public
void
setReference
(
NodeWithAdjacencyList
toCompare
)
{
public
void
setReference
(
VertexWithRankAndDangling
toCompare
)
{
this
.
reference
=
toCompare
.
get
NodeId
();
this
.
reference
=
toCompare
.
get
VertexID
();
}
}
@Override
@Override
public
boolean
equalToReference
(
NodeWithAdjacencyList
candidate
)
{
public
boolean
equalToReference
(
VertexWithRankAndDangling
candidate
)
{
return
candidate
.
get
NodeId
()
==
this
.
reference
;
return
candidate
.
get
VertexID
()
==
this
.
reference
;
}
}
@Override
@Override
public
int
compareToReference
(
TypeComparator
<
NodeWithAdjacencyList
>
referencedComparator
)
{
public
int
compareToReference
(
TypeComparator
<
VertexWithRankAndDangling
>
referencedComparator
)
{
NodeWithAdjacencyListComparator
comp
=
(
NodeWithAdjacencyList
Comparator
)
referencedComparator
;
VertexWithRankAndDanglingComparator
comp
=
(
VertexWithRankAndDangling
Comparator
)
referencedComparator
;
final
long
diff
=
comp
.
reference
-
this
.
reference
;
final
long
diff
=
comp
.
reference
-
this
.
reference
;
return
diff
<
0
?
-
1
:
diff
>
0
?
1
:
0
;
return
diff
<
0
?
-
1
:
diff
>
0
?
1
:
0
;
}
}
...
@@ -72,8 +72,8 @@ public final class NodeWithAdjacencyListComparator implements TypeComparator<Nod
...
@@ -72,8 +72,8 @@ public final class NodeWithAdjacencyListComparator implements TypeComparator<Nod
}
}
@Override
@Override
public
void
putNormalizedKey
(
NodeWithAdjacencyList
record
,
byte
[]
target
,
int
offset
,
int
len
)
{
public
void
putNormalizedKey
(
VertexWithRankAndDangling
record
,
byte
[]
target
,
int
offset
,
int
len
)
{
final
long
value
=
record
.
get
NodeId
();
final
long
value
=
record
.
get
VertexID
();
if
(
len
==
8
)
{
if
(
len
==
8
)
{
// default case, full normalized key
// default case, full normalized key
...
@@ -122,7 +122,7 @@ public final class NodeWithAdjacencyListComparator implements TypeComparator<Nod
...
@@ -122,7 +122,7 @@ public final class NodeWithAdjacencyListComparator implements TypeComparator<Nod
}
}
@Override
@Override
public
NodeWithAdjacencyList
Comparator
duplicate
()
{
public
VertexWithRankAndDangling
Comparator
duplicate
()
{
return
new
NodeWithAdjacencyList
Comparator
();
return
new
VertexWithRankAndDangling
Comparator
();
}
}
}
}
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
types/NodeWithAdjacencyList
ComparatorFactory.java
→
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
custom/types/VertexWithRankAndDangling
ComparatorFactory.java
浏览文件 @
3fe087d9
...
@@ -12,7 +12,7 @@
...
@@ -12,7 +12,7 @@
* specific language governing permissions and limitations under the License.
* specific language governing permissions and limitations under the License.
*
*
**********************************************************************************************************************/
**********************************************************************************************************************/
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.types
;
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.
custom.
types
;
import
eu.stratosphere.nephele.configuration.Configuration
;
import
eu.stratosphere.nephele.configuration.Configuration
;
import
eu.stratosphere.pact.generic.types.TypeComparatorFactory
;
import
eu.stratosphere.pact.generic.types.TypeComparatorFactory
;
...
@@ -20,7 +20,7 @@ import eu.stratosphere.pact.generic.types.TypeComparatorFactory;
...
@@ -20,7 +20,7 @@ import eu.stratosphere.pact.generic.types.TypeComparatorFactory;
/**
/**
*
*
*/
*/
public
final
class
NodeWithAdjacencyListComparatorFactory
implements
TypeComparatorFactory
<
NodeWithAdjacencyList
>
{
public
final
class
VertexWithRankAndDanglingComparatorFactory
implements
TypeComparatorFactory
<
VertexWithRankAndDangling
>
{
@Override
@Override
public
void
writeParametersToConfig
(
Configuration
config
)
{}
public
void
writeParametersToConfig
(
Configuration
config
)
{}
...
@@ -29,7 +29,7 @@ public final class NodeWithAdjacencyListComparatorFactory implements TypeCompara
...
@@ -29,7 +29,7 @@ public final class NodeWithAdjacencyListComparatorFactory implements TypeCompara
public
void
readParametersFromConfig
(
Configuration
config
,
ClassLoader
cl
)
throws
ClassNotFoundException
{}
public
void
readParametersFromConfig
(
Configuration
config
,
ClassLoader
cl
)
throws
ClassNotFoundException
{}
@Override
@Override
public
NodeWithAdjacencyList
Comparator
createComparator
()
{
public
VertexWithRankAndDangling
Comparator
createComparator
()
{
return
new
NodeWithAdjacencyList
Comparator
();
return
new
VertexWithRankAndDangling
Comparator
();
}
}
}
}
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
types/Node
WithRankAndDanglingSerializer.java
→
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
custom/types/Vertex
WithRankAndDanglingSerializer.java
浏览文件 @
3fe087d9
...
@@ -12,7 +12,7 @@
...
@@ -12,7 +12,7 @@
* specific language governing permissions and limitations under the License.
* specific language governing permissions and limitations under the License.
*
*
**********************************************************************************************************************/
**********************************************************************************************************************/
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.types
;
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.
custom.
types
;
import
java.io.IOException
;
import
java.io.IOException
;
...
@@ -24,16 +24,16 @@ import eu.stratosphere.pact.generic.types.TypeSerializer;
...
@@ -24,16 +24,16 @@ import eu.stratosphere.pact.generic.types.TypeSerializer;
/**
/**
*
*
*/
*/
public
final
class
NodeWithRankAndDanglingSerializer
implements
TypeSerializer
<
Node
WithRankAndDangling
>
{
public
final
class
VertexWithRankAndDanglingSerializer
implements
TypeSerializer
<
Vertex
WithRankAndDangling
>
{
@Override
@Override
public
Node
WithRankAndDangling
createInstance
()
{
public
Vertex
WithRankAndDangling
createInstance
()
{
return
new
Node
WithRankAndDangling
();
return
new
Vertex
WithRankAndDangling
();
}
}
@Override
@Override
public
NodeWithRankAndDangling
createCopy
(
Node
WithRankAndDangling
from
)
{
public
VertexWithRankAndDangling
createCopy
(
Vertex
WithRankAndDangling
from
)
{
NodeWithRankAndDangling
n
=
new
Node
WithRankAndDangling
();
VertexWithRankAndDangling
n
=
new
Vertex
WithRankAndDangling
();
copyTo
(
from
,
n
);
copyTo
(
from
,
n
);
return
n
;
return
n
;
}
}
...
@@ -42,8 +42,8 @@ public final class NodeWithRankAndDanglingSerializer implements TypeSerializer<N
...
@@ -42,8 +42,8 @@ public final class NodeWithRankAndDanglingSerializer implements TypeSerializer<N
* @see eu.stratosphere.pact.generic.types.TypeSerializer#copyTo(java.lang.Object, java.lang.Object)
* @see eu.stratosphere.pact.generic.types.TypeSerializer#copyTo(java.lang.Object, java.lang.Object)
*/
*/
@Override
@Override
public
void
copyTo
(
NodeWithRankAndDangling
from
,
Node
WithRankAndDangling
to
)
{
public
void
copyTo
(
VertexWithRankAndDangling
from
,
Vertex
WithRankAndDangling
to
)
{
to
.
set
NodeId
(
from
.
getNodeId
());
to
.
set
VertexID
(
from
.
getVertexID
());
to
.
setRank
(
from
.
getRank
());
to
.
setRank
(
from
.
getRank
());
to
.
setDangling
(
from
.
isDangling
());
to
.
setDangling
(
from
.
isDangling
());
}
}
...
@@ -54,15 +54,15 @@ public final class NodeWithRankAndDanglingSerializer implements TypeSerializer<N
...
@@ -54,15 +54,15 @@ public final class NodeWithRankAndDanglingSerializer implements TypeSerializer<N
}
}
@Override
@Override
public
void
serialize
(
Node
WithRankAndDangling
record
,
DataOutputView
target
)
throws
IOException
{
public
void
serialize
(
Vertex
WithRankAndDangling
record
,
DataOutputView
target
)
throws
IOException
{
target
.
writeLong
(
record
.
get
NodeId
());
target
.
writeLong
(
record
.
get
VertexID
());
target
.
writeDouble
(
record
.
getRank
());
target
.
writeDouble
(
record
.
getRank
());
target
.
writeBoolean
(
record
.
isDangling
());
target
.
writeBoolean
(
record
.
isDangling
());
}
}
@Override
@Override
public
void
deserialize
(
Node
WithRankAndDangling
target
,
DataInputView
source
)
throws
IOException
{
public
void
deserialize
(
Vertex
WithRankAndDangling
target
,
DataInputView
source
)
throws
IOException
{
target
.
set
NodeId
(
source
.
readLong
());
target
.
set
VertexID
(
source
.
readLong
());
target
.
setRank
(
source
.
readDouble
());
target
.
setRank
(
source
.
readDouble
());
target
.
setDangling
(
source
.
readBoolean
());
target
.
setDangling
(
source
.
readBoolean
());
}
}
...
...
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
types/NodeWithAdjacencyList
SerializerFactory.java
→
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
custom/types/VertexWithRankAndDangling
SerializerFactory.java
浏览文件 @
3fe087d9
...
@@ -12,7 +12,7 @@
...
@@ -12,7 +12,7 @@
* specific language governing permissions and limitations under the License.
* specific language governing permissions and limitations under the License.
*
*
**********************************************************************************************************************/
**********************************************************************************************************************/
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.types
;
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.
custom.
types
;
import
eu.stratosphere.nephele.configuration.Configuration
;
import
eu.stratosphere.nephele.configuration.Configuration
;
import
eu.stratosphere.pact.generic.types.TypeSerializerFactory
;
import
eu.stratosphere.pact.generic.types.TypeSerializerFactory
;
...
@@ -20,9 +20,9 @@ import eu.stratosphere.pact.generic.types.TypeSerializerFactory;
...
@@ -20,9 +20,9 @@ import eu.stratosphere.pact.generic.types.TypeSerializerFactory;
/**
/**
*
*
*/
*/
public
final
class
NodeWithAdjacencyListSerializerFactory
implements
TypeSerializerFactory
<
NodeWithAdjacencyList
>
{
public
final
class
VertexWithRankAndDanglingSerializerFactory
implements
TypeSerializerFactory
<
VertexWithRankAndDangling
>
{
private
static
final
NodeWithAdjacencyListSerializer
INSTANCE
=
new
NodeWithAdjacencyList
Serializer
();
private
static
final
VertexWithRankAndDanglingSerializer
INSTANCE
=
new
VertexWithRankAndDangling
Serializer
();
@Override
@Override
public
void
writeParametersToConfig
(
Configuration
config
)
{}
public
void
writeParametersToConfig
(
Configuration
config
)
{}
...
@@ -31,12 +31,12 @@ public final class NodeWithAdjacencyListSerializerFactory implements TypeSeriali
...
@@ -31,12 +31,12 @@ public final class NodeWithAdjacencyListSerializerFactory implements TypeSeriali
public
void
readParametersFromConfig
(
Configuration
config
,
ClassLoader
cl
)
throws
ClassNotFoundException
{}
public
void
readParametersFromConfig
(
Configuration
config
,
ClassLoader
cl
)
throws
ClassNotFoundException
{}
@Override
@Override
public
NodeWithAdjacencyList
Serializer
getSerializer
()
{
public
VertexWithRankAndDangling
Serializer
getSerializer
()
{
return
INSTANCE
;
return
INSTANCE
;
}
}
@Override
@Override
public
Class
<
NodeWithAdjacencyList
>
getDataType
()
{
public
Class
<
VertexWithRankAndDangling
>
getDataType
()
{
return
NodeWithAdjacencyList
.
class
;
return
VertexWithRankAndDangling
.
class
;
}
}
}
}
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
types/Node
WithRankComparator.java
→
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
custom/types/Vertex
WithRankComparator.java
浏览文件 @
3fe087d9
...
@@ -12,7 +12,7 @@
...
@@ -12,7 +12,7 @@
* specific language governing permissions and limitations under the License.
* specific language governing permissions and limitations under the License.
*
*
**********************************************************************************************************************/
**********************************************************************************************************************/
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.types
;
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.
custom.
types
;
import
java.io.IOException
;
import
java.io.IOException
;
...
@@ -23,29 +23,29 @@ import eu.stratosphere.pact.generic.types.TypeComparator;
...
@@ -23,29 +23,29 @@ import eu.stratosphere.pact.generic.types.TypeComparator;
/**
/**
*
*
*/
*/
public
final
class
NodeWithRankComparator
implements
TypeComparator
<
Node
WithRank
>
{
public
final
class
VertexWithRankComparator
implements
TypeComparator
<
Vertex
WithRank
>
{
private
long
reference
;
private
long
reference
;
@Override
@Override
public
int
hash
(
Node
WithRank
record
)
{
public
int
hash
(
Vertex
WithRank
record
)
{
final
long
value
=
record
.
get
NodeId
();
final
long
value
=
record
.
get
VertexID
();
return
43
+
(
int
)
(
value
^
value
>>>
32
);
return
43
+
(
int
)
(
value
^
value
>>>
32
);
}
}
@Override
@Override
public
void
setReference
(
Node
WithRank
toCompare
)
{
public
void
setReference
(
Vertex
WithRank
toCompare
)
{
this
.
reference
=
toCompare
.
get
NodeId
();
this
.
reference
=
toCompare
.
get
VertexID
();
}
}
@Override
@Override
public
boolean
equalToReference
(
Node
WithRank
candidate
)
{
public
boolean
equalToReference
(
Vertex
WithRank
candidate
)
{
return
candidate
.
get
NodeId
()
==
this
.
reference
;
return
candidate
.
get
VertexID
()
==
this
.
reference
;
}
}
@Override
@Override
public
int
compareToReference
(
TypeComparator
<
Node
WithRank
>
referencedComparator
)
{
public
int
compareToReference
(
TypeComparator
<
Vertex
WithRank
>
referencedComparator
)
{
NodeWithRankComparator
comp
=
(
Node
WithRankComparator
)
referencedComparator
;
VertexWithRankComparator
comp
=
(
Vertex
WithRankComparator
)
referencedComparator
;
final
long
diff
=
comp
.
reference
-
this
.
reference
;
final
long
diff
=
comp
.
reference
-
this
.
reference
;
return
diff
<
0
?
-
1
:
diff
>
0
?
1
:
0
;
return
diff
<
0
?
-
1
:
diff
>
0
?
1
:
0
;
}
}
...
@@ -72,8 +72,8 @@ public final class NodeWithRankComparator implements TypeComparator<NodeWithRank
...
@@ -72,8 +72,8 @@ public final class NodeWithRankComparator implements TypeComparator<NodeWithRank
}
}
@Override
@Override
public
void
putNormalizedKey
(
Node
WithRank
record
,
byte
[]
target
,
int
offset
,
int
len
)
{
public
void
putNormalizedKey
(
Vertex
WithRank
record
,
byte
[]
target
,
int
offset
,
int
len
)
{
final
long
value
=
record
.
get
NodeId
();
final
long
value
=
record
.
get
VertexID
();
if
(
len
==
8
)
{
if
(
len
==
8
)
{
// default case, full normalized key
// default case, full normalized key
...
@@ -122,7 +122,7 @@ public final class NodeWithRankComparator implements TypeComparator<NodeWithRank
...
@@ -122,7 +122,7 @@ public final class NodeWithRankComparator implements TypeComparator<NodeWithRank
}
}
@Override
@Override
public
Node
WithRankComparator
duplicate
()
{
public
Vertex
WithRankComparator
duplicate
()
{
return
new
Node
WithRankComparator
();
return
new
Vertex
WithRankComparator
();
}
}
}
}
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
types/Node
WithRankComparatorFactory.java
→
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
custom/types/Vertex
WithRankComparatorFactory.java
浏览文件 @
3fe087d9
...
@@ -12,7 +12,7 @@
...
@@ -12,7 +12,7 @@
* specific language governing permissions and limitations under the License.
* specific language governing permissions and limitations under the License.
*
*
**********************************************************************************************************************/
**********************************************************************************************************************/
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.types
;
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.
custom.
types
;
import
eu.stratosphere.nephele.configuration.Configuration
;
import
eu.stratosphere.nephele.configuration.Configuration
;
import
eu.stratosphere.pact.generic.types.TypeComparatorFactory
;
import
eu.stratosphere.pact.generic.types.TypeComparatorFactory
;
...
@@ -20,7 +20,7 @@ import eu.stratosphere.pact.generic.types.TypeComparatorFactory;
...
@@ -20,7 +20,7 @@ import eu.stratosphere.pact.generic.types.TypeComparatorFactory;
/**
/**
*
*
*/
*/
public
final
class
NodeWithRankComparatorFactory
implements
TypeComparatorFactory
<
Node
WithRank
>
{
public
final
class
VertexWithRankComparatorFactory
implements
TypeComparatorFactory
<
Vertex
WithRank
>
{
@Override
@Override
public
void
writeParametersToConfig
(
Configuration
config
)
{}
public
void
writeParametersToConfig
(
Configuration
config
)
{}
...
@@ -29,7 +29,7 @@ public final class NodeWithRankComparatorFactory implements TypeComparatorFactor
...
@@ -29,7 +29,7 @@ public final class NodeWithRankComparatorFactory implements TypeComparatorFactor
public
void
readParametersFromConfig
(
Configuration
config
,
ClassLoader
cl
)
throws
ClassNotFoundException
{}
public
void
readParametersFromConfig
(
Configuration
config
,
ClassLoader
cl
)
throws
ClassNotFoundException
{}
@Override
@Override
public
Node
WithRankComparator
createComparator
()
{
public
Vertex
WithRankComparator
createComparator
()
{
return
new
Node
WithRankComparator
();
return
new
Vertex
WithRankComparator
();
}
}
}
}
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/types/VertexWithRankDanglingToVertexWithAdjacencyListPairComparatorFactory.java
0 → 100644
浏览文件 @
3fe087d9
/***********************************************************************************************************************
*
* Copyright (C) 2012 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types
;
import
eu.stratosphere.pact.generic.types.TypeComparator
;
import
eu.stratosphere.pact.generic.types.TypePairComparator
;
import
eu.stratosphere.pact.generic.types.TypePairComparatorFactory
;
/**
*
*/
public
class
VertexWithRankDanglingToVertexWithAdjacencyListPairComparatorFactory
implements
TypePairComparatorFactory
<
VertexWithRankAndDangling
,
VertexWithAdjacencyList
>
{
@Override
public
VertexWithRankDanglingToVertexWithAdjacencyListPairComparator
createComparator12
(
TypeComparator
<
VertexWithRankAndDangling
>
comparator1
,
TypeComparator
<
VertexWithAdjacencyList
>
comparator2
)
{
return
new
VertexWithRankDanglingToVertexWithAdjacencyListPairComparator
();
}
@Override
public
VertexWithAdjacencyListToVertexWithRankDanglingPairComparator
createComparator21
(
TypeComparator
<
VertexWithRankAndDangling
>
comparator1
,
TypeComparator
<
VertexWithAdjacencyList
>
comparator2
)
{
return
new
VertexWithAdjacencyListToVertexWithRankDanglingPairComparator
();
}
public
static
final
class
VertexWithRankDanglingToVertexWithAdjacencyListPairComparator
implements
TypePairComparator
<
VertexWithRankAndDangling
,
VertexWithAdjacencyList
>
{
private
long
reference
;
@Override
public
void
setReference
(
VertexWithRankAndDangling
reference
)
{
this
.
reference
=
reference
.
getVertexID
();
}
@Override
public
boolean
equalToReference
(
VertexWithAdjacencyList
candidate
)
{
return
this
.
reference
==
candidate
.
getVertexID
();
}
@Override
public
int
compareToReference
(
VertexWithAdjacencyList
candidate
)
{
long
diff
=
candidate
.
getVertexID
()
-
this
.
reference
;
return
diff
<
0
?
-
1
:
diff
>
0
?
1
:
0
;
}
}
public
static
final
class
VertexWithAdjacencyListToVertexWithRankDanglingPairComparator
implements
TypePairComparator
<
VertexWithAdjacencyList
,
VertexWithRankAndDangling
>
{
private
long
reference
;
@Override
public
void
setReference
(
VertexWithAdjacencyList
reference
)
{
this
.
reference
=
reference
.
getVertexID
();
}
@Override
public
boolean
equalToReference
(
VertexWithRankAndDangling
candidate
)
{
return
this
.
reference
==
candidate
.
getVertexID
();
}
@Override
public
int
compareToReference
(
VertexWithRankAndDangling
candidate
)
{
long
diff
=
candidate
.
getVertexID
()
-
this
.
reference
;
return
diff
<
0
?
-
1
:
diff
>
0
?
1
:
0
;
}
}
}
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/custom/types/VertexWithRankDanglingToVertexWithRankPairComparatorFactory.java
0 → 100644
浏览文件 @
3fe087d9
/***********************************************************************************************************************
*
* Copyright (C) 2012 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.types
;
import
eu.stratosphere.pact.generic.types.TypeComparator
;
import
eu.stratosphere.pact.generic.types.TypePairComparator
;
import
eu.stratosphere.pact.generic.types.TypePairComparatorFactory
;
/**
*
*/
public
class
VertexWithRankDanglingToVertexWithRankPairComparatorFactory
implements
TypePairComparatorFactory
<
VertexWithRankAndDangling
,
VertexWithRank
>
{
@Override
public
VertexWithRankDanglingToVertexWithRankComparator
createComparator12
(
TypeComparator
<
VertexWithRankAndDangling
>
comparator1
,
TypeComparator
<
VertexWithRank
>
comparator2
)
{
return
new
VertexWithRankDanglingToVertexWithRankComparator
();
}
@Override
public
VertexWithRankToVertexWithRankDanglingPairComparator
createComparator21
(
TypeComparator
<
VertexWithRankAndDangling
>
comparator1
,
TypeComparator
<
VertexWithRank
>
comparator2
)
{
return
new
VertexWithRankToVertexWithRankDanglingPairComparator
();
}
public
static
final
class
VertexWithRankDanglingToVertexWithRankComparator
implements
TypePairComparator
<
VertexWithRankAndDangling
,
VertexWithRank
>
{
private
long
reference
;
@Override
public
void
setReference
(
VertexWithRankAndDangling
reference
)
{
this
.
reference
=
reference
.
getVertexID
();
}
@Override
public
boolean
equalToReference
(
VertexWithRank
candidate
)
{
return
this
.
reference
==
candidate
.
getVertexID
();
}
@Override
public
int
compareToReference
(
VertexWithRank
candidate
)
{
long
diff
=
candidate
.
getVertexID
()
-
this
.
reference
;
return
diff
<
0
?
-
1
:
diff
>
0
?
1
:
0
;
}
}
public
static
final
class
VertexWithRankToVertexWithRankDanglingPairComparator
implements
TypePairComparator
<
VertexWithRank
,
VertexWithRankAndDangling
>
{
private
long
reference
;
@Override
public
void
setReference
(
VertexWithRank
reference
)
{
this
.
reference
=
reference
.
getVertexID
();
}
@Override
public
boolean
equalToReference
(
VertexWithRankAndDangling
candidate
)
{
return
this
.
reference
==
candidate
.
getVertexID
();
}
@Override
public
int
compareToReference
(
VertexWithRankAndDangling
candidate
)
{
long
diff
=
candidate
.
getVertexID
()
-
this
.
reference
;
return
diff
<
0
?
-
1
:
diff
>
0
?
1
:
0
;
}
}
}
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
types/Node
WithRankSerializer.java
→
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
custom/types/Vertex
WithRankSerializer.java
浏览文件 @
3fe087d9
...
@@ -12,7 +12,7 @@
...
@@ -12,7 +12,7 @@
* specific language governing permissions and limitations under the License.
* specific language governing permissions and limitations under the License.
*
*
**********************************************************************************************************************/
**********************************************************************************************************************/
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.types
;
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.
custom.
types
;
import
java.io.IOException
;
import
java.io.IOException
;
...
@@ -24,16 +24,16 @@ import eu.stratosphere.pact.generic.types.TypeSerializer;
...
@@ -24,16 +24,16 @@ import eu.stratosphere.pact.generic.types.TypeSerializer;
/**
/**
*
*
*/
*/
public
final
class
NodeWithRankSerializer
implements
TypeSerializer
<
Node
WithRank
>
{
public
final
class
VertexWithRankSerializer
implements
TypeSerializer
<
Vertex
WithRank
>
{
@Override
@Override
public
Node
WithRank
createInstance
()
{
public
Vertex
WithRank
createInstance
()
{
return
new
Node
WithRank
();
return
new
Vertex
WithRank
();
}
}
@Override
@Override
public
NodeWithRank
createCopy
(
Node
WithRank
from
)
{
public
VertexWithRank
createCopy
(
Vertex
WithRank
from
)
{
NodeWithRank
n
=
new
Node
WithRank
();
VertexWithRank
n
=
new
Vertex
WithRank
();
copyTo
(
from
,
n
);
copyTo
(
from
,
n
);
return
n
;
return
n
;
}
}
...
@@ -42,8 +42,8 @@ public final class NodeWithRankSerializer implements TypeSerializer<NodeWithRank
...
@@ -42,8 +42,8 @@ public final class NodeWithRankSerializer implements TypeSerializer<NodeWithRank
* @see eu.stratosphere.pact.generic.types.TypeSerializer#copyTo(java.lang.Object, java.lang.Object)
* @see eu.stratosphere.pact.generic.types.TypeSerializer#copyTo(java.lang.Object, java.lang.Object)
*/
*/
@Override
@Override
public
void
copyTo
(
NodeWithRank
from
,
Node
WithRank
to
)
{
public
void
copyTo
(
VertexWithRank
from
,
Vertex
WithRank
to
)
{
to
.
set
NodeId
(
from
.
getNodeId
());
to
.
set
VertexID
(
from
.
getVertexID
());
to
.
setRank
(
from
.
getRank
());
to
.
setRank
(
from
.
getRank
());
}
}
...
@@ -53,14 +53,14 @@ public final class NodeWithRankSerializer implements TypeSerializer<NodeWithRank
...
@@ -53,14 +53,14 @@ public final class NodeWithRankSerializer implements TypeSerializer<NodeWithRank
}
}
@Override
@Override
public
void
serialize
(
Node
WithRank
record
,
DataOutputView
target
)
throws
IOException
{
public
void
serialize
(
Vertex
WithRank
record
,
DataOutputView
target
)
throws
IOException
{
target
.
writeLong
(
record
.
get
NodeId
());
target
.
writeLong
(
record
.
get
VertexID
());
target
.
writeDouble
(
record
.
getRank
());
target
.
writeDouble
(
record
.
getRank
());
}
}
@Override
@Override
public
void
deserialize
(
Node
WithRank
target
,
DataInputView
source
)
throws
IOException
{
public
void
deserialize
(
Vertex
WithRank
target
,
DataInputView
source
)
throws
IOException
{
target
.
set
NodeId
(
source
.
readLong
());
target
.
set
VertexID
(
source
.
readLong
());
target
.
setRank
(
source
.
readDouble
());
target
.
setRank
(
source
.
readDouble
());
}
}
...
...
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
types/Node
WithRankSerializerFactory.java
→
pact/pact-runtime/src/main/java/eu/stratosphere/pact/runtime/iterative/compensatable/danglingpagerank/
custom/types/Vertex
WithRankSerializerFactory.java
浏览文件 @
3fe087d9
...
@@ -12,7 +12,7 @@
...
@@ -12,7 +12,7 @@
* specific language governing permissions and limitations under the License.
* specific language governing permissions and limitations under the License.
*
*
**********************************************************************************************************************/
**********************************************************************************************************************/
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.types
;
package
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.
custom.
types
;
import
eu.stratosphere.nephele.configuration.Configuration
;
import
eu.stratosphere.nephele.configuration.Configuration
;
import
eu.stratosphere.pact.generic.types.TypeSerializerFactory
;
import
eu.stratosphere.pact.generic.types.TypeSerializerFactory
;
...
@@ -20,9 +20,9 @@ import eu.stratosphere.pact.generic.types.TypeSerializerFactory;
...
@@ -20,9 +20,9 @@ import eu.stratosphere.pact.generic.types.TypeSerializerFactory;
/**
/**
*
*
*/
*/
public
final
class
NodeWithRankSerializerFactory
implements
TypeSerializerFactory
<
Node
WithRank
>
{
public
final
class
VertexWithRankSerializerFactory
implements
TypeSerializerFactory
<
Vertex
WithRank
>
{
private
static
final
NodeWithRankSerializer
INSTANCE
=
new
Node
WithRankSerializer
();
private
static
final
VertexWithRankSerializer
INSTANCE
=
new
Vertex
WithRankSerializer
();
@Override
@Override
public
void
writeParametersToConfig
(
Configuration
config
)
{}
public
void
writeParametersToConfig
(
Configuration
config
)
{}
...
@@ -31,12 +31,12 @@ public final class NodeWithRankSerializerFactory implements TypeSerializerFactor
...
@@ -31,12 +31,12 @@ public final class NodeWithRankSerializerFactory implements TypeSerializerFactor
public
void
readParametersFromConfig
(
Configuration
config
,
ClassLoader
cl
)
throws
ClassNotFoundException
{}
public
void
readParametersFromConfig
(
Configuration
config
,
ClassLoader
cl
)
throws
ClassNotFoundException
{}
@Override
@Override
public
Node
WithRankSerializer
getSerializer
()
{
public
Vertex
WithRankSerializer
getSerializer
()
{
return
INSTANCE
;
return
INSTANCE
;
}
}
@Override
@Override
public
Class
<
Node
WithRank
>
getDataType
()
{
public
Class
<
Vertex
WithRank
>
getDataType
()
{
return
Node
WithRank
.
class
;
return
Vertex
WithRank
.
class
;
}
}
}
}
pact/pact-tests/src/test/java/eu/stratosphere/pact/test/iterative/DanglingPageRankITCase.java
浏览文件 @
3fe087d9
...
@@ -24,6 +24,7 @@ import org.junit.runners.Parameterized.Parameters;
...
@@ -24,6 +24,7 @@ import org.junit.runners.Parameterized.Parameters;
import
eu.stratosphere.nephele.configuration.Configuration
;
import
eu.stratosphere.nephele.configuration.Configuration
;
import
eu.stratosphere.nephele.jobgraph.JobGraph
;
import
eu.stratosphere.nephele.jobgraph.JobGraph
;
import
eu.stratosphere.pact.runtime.iterative.compensatable.danglingpagerank.custom.CustomCompensatableDanglingPageRank
;
import
eu.stratosphere.pact.test.util.TestBase
;
import
eu.stratosphere.pact.test.util.TestBase
;
@RunWith
(
Parameterized
.
class
)
@RunWith
(
Parameterized
.
class
)
...
@@ -102,7 +103,7 @@ public class DanglingPageRankITCase extends TestBase {
...
@@ -102,7 +103,7 @@ public class DanglingPageRankITCase extends TestBase {
"0"
"0"
};
};
return
CompensatableDanglingPageRank
.
getJobGraph
(
parameters
);
return
C
ustomC
ompensatableDanglingPageRank
.
getJobGraph
(
parameters
);
}
}
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录