博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
hbase排序 java,Java HBaseConfiguration.merge方法代码示例
阅读量:7029 次
发布时间:2019-06-28

本文共 2918 字,大约阅读时间需要 9 分钟。

import org.apache.hadoop.hbase.HBaseConfiguration; //导入方法依赖的package包/类

/**

* Use this before submitting a TableReduce job. It will

* appropriately set up the JobConf.

*

* @param table The output table.

* @param reducer The reducer class to use.

* @param job The current job to adjust. Make sure the passed job is

* carrying all necessary HBase configuration.

* @param partitioner Partitioner to use. Pass null to use

* default partitioner.

* @param quorumAddress Distant cluster to write to; default is null for

* output to the cluster that is designated in hbase-site.xml.

* Set this String to the zookeeper ensemble of an alternate remote cluster

* when you would have the reduce write a cluster that is other than the

* default; e.g. copying tables between clusters, the source would be

* designated by hbase-site.xml and this param would have the

* ensemble address of the remote cluster. The format to pass is particular.

* Pass <hbase.zookeeper.quorum>:<

* hbase.zookeeper.client.port>:<zookeeper.znode.parent>

* such as server,server2,server3:2181:/hbase.

* @param serverClass redefined hbase.regionserver.class

* @param serverImpl redefined hbase.regionserver.impl

* @param addDependencyJars upload HBase jars and jars for any of the configured

* job classes via the distributed cache (tmpjars).

* @throws IOException When determining the region count fails.

*/

public static void initTableReducerJob(String table,

Class extends TableReducer> reducer, Job job,

Class partitioner, String quorumAddress, String serverClass,

String serverImpl, boolean addDependencyJars) throws IOException {

Configuration conf = job.getConfiguration();

HBaseConfiguration.merge(conf, HBaseConfiguration.create(conf));

job.setOutputFormatClass(TableOutputFormat.class);

if (reducer != null) job.setReducerClass(reducer);

conf.set(TableOutputFormat.OUTPUT_TABLE, table);

conf.setStrings("io.serializations", conf.get("io.serializations"),

MutationSerialization.class.getName(), ResultSerialization.class.getName());

// If passed a quorum/ensemble address, pass it on to TableOutputFormat.

if (quorumAddress != null) {

// Calling this will validate the format

ZKConfig.validateClusterKey(quorumAddress);

conf.set(TableOutputFormat.QUORUM_ADDRESS,quorumAddress);

}

if (serverClass != null && serverImpl != null) {

conf.set(TableOutputFormat.REGION_SERVER_CLASS, serverClass);

conf.set(TableOutputFormat.REGION_SERVER_IMPL, serverImpl);

}

job.setOutputKeyClass(ImmutableBytesWritable.class);

job.setOutputValueClass(Writable.class);

if (partitioner == HRegionPartitioner.class) {

job.setPartitionerClass(HRegionPartitioner.class);

int regions = MetaTableAccessor.getRegionCount(conf, TableName.valueOf(table));

if (job.getNumReduceTasks() > regions) {

job.setNumReduceTasks(regions);

}

} else if (partitioner != null) {

job.setPartitionerClass(partitioner);

}

if (addDependencyJars) {

addDependencyJars(job);

}

initCredentials(job);

}

转载地址:http://beexl.baihongyu.com/

你可能感兴趣的文章
day20 Python 装饰器
查看>>
限制性与非限制性定语从句区别
查看>>
fiddler工具的使用
查看>>
jquery源码分析(二)——架构设计
查看>>
javascript深入理解js闭包(转)
查看>>
207. Course Schedule
查看>>
如何优化您的 Android 应用 (Go 版)
查看>>
Trie树实现
查看>>
Opencv无法调用cvCaptureFromCAM无法打开电脑自带摄像头
查看>>
Exception异常处理机制
查看>>
复杂的web---web中B/S网络架构
查看>>
编写文档的时候各种问题
查看>>
Eclipse里maven的project报Unbound classpath variable: 'M2_REPO/**/***/***.jar
查看>>
新旅程CSS 基础篇分享一
查看>>
查看内核函数调用的调试方法【原创】
查看>>
个人项目中遇到的问题
查看>>
byte与base64string的相互转化以及加密算法
查看>>
20145103 《Java程序设计》第3周学习总结
查看>>
ubuntu声音系统
查看>>
哈哈更新资源列表2
查看>>