HBase學習之路（六）過濾器

本文轉載自查看原文 2018-04-02 18:42 10736 Hadoop/ HBase

過濾器（Filter）

　　基礎API中的查詢操作在面對大量數據的時候是非常蒼白的，這里Hbase提供了高級的查詢方法：Filter。Filter可以根據簇、列、版本等更多的條件來對數據進行過濾，基於Hbase本身提供的三維有序（主鍵有序、列有序、版本有序），這些Filter可以高效的完成查詢過濾的任務。帶有Filter條件的RPC查詢請求會把Filter分發到各個RegionServer，是一個服務器端（Server-side）的過濾器，這樣也可以降低網絡傳輸的壓力。

　　要完成一個過濾的操作，至少需要兩個參數。一個是抽象的操作符，Hbase提供了枚舉類型的變量來表示這些抽象的操作符：LESS/LESS_OR_EQUAL/EQUAL/NOT_EUQAL等；另外一個就是具體的比較器（Comparator），代表具體的比較邏輯，如果可以提高字節級的比較、字符串級的比較等。有了這兩個參數，我們就可以清晰的定義篩選的條件，過濾數據。

抽象操作符（比較運算符）

LESS <

LESS_OR_EQUAL <=

EQUAL =

NOT_EQUAL <>

GREATER_OR_EQUAL >=

GREATER >

NO_OP 排除所有

比較器（指定比較機制）

BinaryComparator 按字節索引順序比較指定字節數組，采用 Bytes.compareTo(byte[])

BinaryPrefixComparator 跟前面相同，只是比較左端的數據是否相同

NullComparator 判斷給定的是否為空

BitComparator 按位比較

RegexStringComparator 提供一個正則的比較器，僅支持 EQUAL 和非 EQUAL

SubstringComparator 判斷提供的子串是否出現在 value 中

HBase過濾器的分類

比較過濾器

1、行鍵過濾器 RowFilter

Filter rowFilter = new RowFilter(CompareOp.GREATER, new BinaryComparator("95007".getBytes()));
scan.setFilter(rowFilter);

 1 public class HbaseFilterTest {
 2 
 3     private static final String ZK_CONNECT_KEY = "hbase.zookeeper.quorum";
 4     private static final String ZK_CONNECT_VALUE = "hadoop1:2181,hadoop2:2181,hadoop3:2181";
 5 
 6     private static Connection conn = null;
 7     private static Admin admin = null;
 8     
 9     public static void main(String[] args) throws Exception {
10         
11         Configuration conf = HBaseConfiguration.create();
12         conf.set(ZK_CONNECT_KEY, ZK_CONNECT_VALUE);
13         conn = ConnectionFactory.createConnection(conf);
14         admin = conn.getAdmin();
15         Table table = conn.getTable(TableName.valueOf("student"));
16         
17         Scan scan = new Scan();
18         
19         Filter rowFilter = new RowFilter(CompareOp.GREATER, new BinaryComparator("95007".getBytes()));
20         scan.setFilter(rowFilter);
21         ResultScanner resultScanner = table.getScanner(scan);
22         for(Result result : resultScanner) {
23             List<Cell> cells = result.listCells();
24             for(Cell cell : cells) {
25                 System.out.println(cell);
26             }
27         }
28         
29         
30     }

View Code

運行結果部分截圖

2、列簇過濾器 FamilyFilter

Filter familyFilter = new FamilyFilter(CompareOp.EQUAL, new BinaryComparator("info".getBytes()));
scan.setFilter(familyFilter);

 1 public class HbaseFilterTest {
 2 
 3     private static final String ZK_CONNECT_KEY = "hbase.zookeeper.quorum";
 4     private static final String ZK_CONNECT_VALUE = "hadoop1:2181,hadoop2:2181,hadoop3:2181";
 5 
 6     private static Connection conn = null;
 7     private static Admin admin = null;
 8     
 9     public static void main(String[] args) throws Exception {
10         
11         Configuration conf = HBaseConfiguration.create();
12         conf.set(ZK_CONNECT_KEY, ZK_CONNECT_VALUE);
13         conn = ConnectionFactory.createConnection(conf);
14         admin = conn.getAdmin();
15         Table table = conn.getTable(TableName.valueOf("student"));
16         
17         Scan scan = new Scan();
18 
19         Filter familyFilter = new FamilyFilter(CompareOp.EQUAL, new BinaryComparator("info".getBytes()));
20         scan.setFilter(familyFilter);
21         ResultScanner resultScanner = table.getScanner(scan);
22         for(Result result : resultScanner) {
23             List<Cell> cells = result.listCells();
24             for(Cell cell : cells) {
25                 System.out.println(cell);
26             }
27         }
28         
29         
30     }
31     
32     
33 }

View Code

3、列過濾器 QualifierFilter

Filter qualifierFilter = new QualifierFilter(CompareOp.EQUAL, new BinaryComparator("name".getBytes()));
scan.setFilter(qualifierFilter);

 1 public class HbaseFilterTest {
 2 
 3     private static final String ZK_CONNECT_KEY = "hbase.zookeeper.quorum";
 4     private static final String ZK_CONNECT_VALUE = "hadoop1:2181,hadoop2:2181,hadoop3:2181";
 5 
 6     private static Connection conn = null;
 7     private static Admin admin = null;
 8     
 9     public static void main(String[] args) throws Exception {
10         
11         Configuration conf = HBaseConfiguration.create();
12         conf.set(ZK_CONNECT_KEY, ZK_CONNECT_VALUE);
13         conn = ConnectionFactory.createConnection(conf);
14         admin = conn.getAdmin();
15         Table table = conn.getTable(TableName.valueOf("student"));
16         
17         Scan scan = new Scan();
18         
19         Filter qualifierFilter = new QualifierFilter(CompareOp.EQUAL, new BinaryComparator("name".getBytes()));
20         scan.setFilter(qualifierFilter);
21         ResultScanner resultScanner = table.getScanner(scan);
22         for(Result result : resultScanner) {
23             List<Cell> cells = result.listCells();
24             for(Cell cell : cells) {
25                 System.out.println(cell);
26             }
27         }
28         
29         
30     }
31     
32     
33 }

View Code

4、值過濾器 ValueFilter

Filter valueFilter = new ValueFilter(CompareOp.EQUAL, new SubstringComparator("男"));
scan.setFilter(valueFilter);

 1 public class HbaseFilterTest {
 2 
 3     private static final String ZK_CONNECT_KEY = "hbase.zookeeper.quorum";
 4     private static final String ZK_CONNECT_VALUE = "hadoop1:2181,hadoop2:2181,hadoop3:2181";
 5 
 6     private static Connection conn = null;
 7     private static Admin admin = null;
 8     
 9     public static void main(String[] args) throws Exception {
10         
11         Configuration conf = HBaseConfiguration.create();
12         conf.set(ZK_CONNECT_KEY, ZK_CONNECT_VALUE);
13         conn = ConnectionFactory.createConnection(conf);
14         admin = conn.getAdmin();
15         Table table = conn.getTable(TableName.valueOf("student"));
16         
17         Scan scan = new Scan();
18         
19         Filter valueFilter = new ValueFilter(CompareOp.EQUAL, new SubstringComparator("男"));
20         scan.setFilter(valueFilter);
21         ResultScanner resultScanner = table.getScanner(scan);
22         for(Result result : resultScanner) {
23             List<Cell> cells = result.listCells();
24             for(Cell cell : cells) {
25                 System.out.println(cell);
26             }
27         }
28         
29         
30     }
31     
32     
33 }

View Code

5、時間戳過濾器 TimestampsFilter

List<Long> list = new ArrayList<>();
list.add(1522469029503l);
TimestampsFilter timestampsFilter = new TimestampsFilter(list);
scan.setFilter(timestampsFilter);

 1 public class HbaseFilterTest {
 2 
 3     private static final String ZK_CONNECT_KEY = "hbase.zookeeper.quorum";
 4     private static final String ZK_CONNECT_VALUE = "hadoop1:2181,hadoop2:2181,hadoop3:2181";
 5 
 6     private static Connection conn = null;
 7     private static Admin admin = null;
 8     
 9     public static void main(String[] args) throws Exception {
10         
11         Configuration conf = HBaseConfiguration.create();
12         conf.set(ZK_CONNECT_KEY, ZK_CONNECT_VALUE);
13         conn = ConnectionFactory.createConnection(conf);
14         admin = conn.getAdmin();
15         Table table = conn.getTable(TableName.valueOf("student"));
16         
17         Scan scan = new Scan();
18         
19         List<Long> list = new ArrayList<>();
20         list.add(1522469029503l);
21         TimestampsFilter timestampsFilter = new TimestampsFilter(list);
22         scan.setFilter(timestampsFilter);
23         ResultScanner resultScanner = table.getScanner(scan);
24         for(Result result : resultScanner) {
25             List<Cell> cells = result.listCells();
26             for(Cell cell : cells) {
27                 System.out.println(Bytes.toString(cell.getRow()) + "\t" + Bytes.toString(cell.getFamily()) + "\t" + Bytes.toString(cell.getQualifier())
28                 + "\t" + Bytes.toString(cell.getValue()) + "\t" + cell.getTimestamp());
29             }
30         }
31         
32         
33     }
34     
35     
36 }

View Code

專用過濾器

1、單列值過濾器 SingleColumnValueFilter ----會返回滿足條件的整行

SingleColumnValueFilter singleColumnValueFilter = new SingleColumnValueFilter(
                "info".getBytes(), //列簇 "name".getBytes(), //列
                CompareOp.EQUAL, 
                new SubstringComparator("劉晨"));
//如果不設置為 true，則那些不包含指定 column 的行也會返回
singleColumnValueFilter.setFilterIfMissing(true);
scan.setFilter(singleColumnValueFilter);

 1 public class HbaseFilterTest2 {
 2 
 3     private static final String ZK_CONNECT_KEY = "hbase.zookeeper.quorum";
 4     private static final String ZK_CONNECT_VALUE = "hadoop1:2181,hadoop2:2181,hadoop3:2181";
 5 
 6     private static Connection conn = null;
 7     private static Admin admin = null;
 8     
 9     public static void main(String[] args) throws Exception {
10         
11         Configuration conf = HBaseConfiguration.create();
12         conf.set(ZK_CONNECT_KEY, ZK_CONNECT_VALUE);
13         conn = ConnectionFactory.createConnection(conf);
14         admin = conn.getAdmin();
15         Table table = conn.getTable(TableName.valueOf("student"));
16         
17         Scan scan = new Scan();
18         
19         SingleColumnValueFilter singleColumnValueFilter = new SingleColumnValueFilter(
20                 "info".getBytes(), 
21                 "name".getBytes(), 
22                 CompareOp.EQUAL, 
23                 new SubstringComparator("劉晨"));
24         singleColumnValueFilter.setFilterIfMissing(true);
25         
26         scan.setFilter(singleColumnValueFilter);
27         ResultScanner resultScanner = table.getScanner(scan);
28         for(Result result : resultScanner) {
29             List<Cell> cells = result.listCells();
30             for(Cell cell : cells) {
31                 System.out.println(Bytes.toString(cell.getRow()) + "\t" + Bytes.toString(cell.getFamily()) + "\t" + Bytes.toString(cell.getQualifier())
32                 + "\t" + Bytes.toString(cell.getValue()) + "\t" + cell.getTimestamp());
33             }
34         }
35         
36         
37     }
38     
39     
40 }

View Code

2、單列值排除器 SingleColumnValueExcludeFilter

SingleColumnValueExcludeFilter singleColumnValueExcludeFilter = new SingleColumnValueExcludeFilter(
                "info".getBytes(), 
                "name".getBytes(), 
                CompareOp.EQUAL, 
                new SubstringComparator("劉晨"));
singleColumnValueExcludeFilter.setFilterIfMissing(true);
        
scan.setFilter(singleColumnValueExcludeFilter);

 1 public class HbaseFilterTest2 {
 2 
 3     private static final String ZK_CONNECT_KEY = "hbase.zookeeper.quorum";
 4     private static final String ZK_CONNECT_VALUE = "hadoop1:2181,hadoop2:2181,hadoop3:2181";
 5 
 6     private static Connection conn = null;
 7     private static Admin admin = null;
 8     
 9     public static void main(String[] args) throws Exception {
10         
11         Configuration conf = HBaseConfiguration.create();
12         conf.set(ZK_CONNECT_KEY, ZK_CONNECT_VALUE);
13         conn = ConnectionFactory.createConnection(conf);
14         admin = conn.getAdmin();
15         Table table = conn.getTable(TableName.valueOf("student"));
16         
17         Scan scan = new Scan();
18         
19         SingleColumnValueExcludeFilter singleColumnValueExcludeFilter = new SingleColumnValueExcludeFilter(
20                 "info".getBytes(), 
21                 "name".getBytes(), 
22                 CompareOp.EQUAL, 
23                 new SubstringComparator("劉晨"));
24         singleColumnValueExcludeFilter.setFilterIfMissing(true);
25         
26         scan.setFilter(singleColumnValueExcludeFilter);
27         ResultScanner resultScanner = table.getScanner(scan);
28         for(Result result : resultScanner) {
29             List<Cell> cells = result.listCells();
30             for(Cell cell : cells) {
31                 System.out.println(Bytes.toString(cell.getRow()) + "\t" + Bytes.toString(cell.getFamily()) + "\t" + Bytes.toString(cell.getQualifier())
32                 + "\t" + Bytes.toString(cell.getValue()) + "\t" + cell.getTimestamp());
33             }
34         }
35         
36         
37     }
38     
39     
40 }

View Code

3、前綴過濾器 PrefixFilter----針對行鍵

PrefixFilter prefixFilter = new PrefixFilter("9501".getBytes());
        
scan.setFilter(prefixFilter);

 1 public class HbaseFilterTest2 {
 2 
 3     private static final String ZK_CONNECT_KEY = "hbase.zookeeper.quorum";
 4     private static final String ZK_CONNECT_VALUE = "hadoop1:2181,hadoop2:2181,hadoop3:2181";
 5 
 6     private static Connection conn = null;
 7     private static Admin admin = null;
 8     
 9     public static void main(String[] args) throws Exception {
10         
11         Configuration conf = HBaseConfiguration.create();
12         conf.set(ZK_CONNECT_KEY, ZK_CONNECT_VALUE);
13         conn = ConnectionFactory.createConnection(conf);
14         admin = conn.getAdmin();
15         Table table = conn.getTable(TableName.valueOf("student"));
16         
17         Scan scan = new Scan();
18         
19         PrefixFilter prefixFilter = new PrefixFilter("9501".getBytes());
20         
21         scan.setFilter(prefixFilter);
22         ResultScanner resultScanner = table.getScanner(scan);
23         for(Result result : resultScanner) {
24             List<Cell> cells = result.listCells();
25             for(Cell cell : cells) {
26                 System.out.println(Bytes.toString(cell.getRow()) + "\t" + Bytes.toString(cell.getFamily()) + "\t" + Bytes.toString(cell.getQualifier())
27                 + "\t" + Bytes.toString(cell.getValue()) + "\t" + cell.getTimestamp());
28             }
29         }
30         
31         
32     }
33     
34     
35 }

View Code

4、列前綴過濾器 ColumnPrefixFilter

ColumnPrefixFilter columnPrefixFilter = new ColumnPrefixFilter("name".getBytes());
        
scan.setFilter(columnPrefixFilter);

 1 public class HbaseFilterTest2 {
 2 
 3     private static final String ZK_CONNECT_KEY = "hbase.zookeeper.quorum";
 4     private static final String ZK_CONNECT_VALUE = "hadoop1:2181,hadoop2:2181,hadoop3:2181";
 5 
 6     private static Connection conn = null;
 7     private static Admin admin = null;
 8     
 9     public static void main(String[] args) throws Exception {
10         
11         Configuration conf = HBaseConfiguration.create();
12         conf.set(ZK_CONNECT_KEY, ZK_CONNECT_VALUE);
13         conn = ConnectionFactory.createConnection(conf);
14         admin = conn.getAdmin();
15         Table table = conn.getTable(TableName.valueOf("student"));
16         
17         Scan scan = new Scan();
18         
19         ColumnPrefixFilter columnPrefixFilter = new ColumnPrefixFilter("name".getBytes());
20         
21         scan.setFilter(columnPrefixFilter);
22         ResultScanner resultScanner = table.getScanner(scan);
23         for(Result result : resultScanner) {
24             List<Cell> cells = result.listCells();
25             for(Cell cell : cells) {
26                 System.out.println(Bytes.toString(cell.getRow()) + "\t" + Bytes.toString(cell.getFamily()) + "\t" + Bytes.toString(cell.getQualifier())
27                 + "\t" + Bytes.toString(cell.getValue()) + "\t" + cell.getTimestamp());
28             }
29         }
30         
31         
32     }
33     
34     
35 }

View Code

5、分頁過濾器 PageFilter

免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。

猜您在找 HBase學習之路（十一）HBase的協過濾器 HBase（七）Hbase過濾器 hbase 過濾器 rowfilter Hbase(四) 過濾器查詢 HBase之過濾器 HBase Filter 過濾器之 Comparator 原理及源碼學習 Hbase Filter過濾器查詢詳解 HBase內置過濾器的總結 HBase Filter 過濾器之FamilyFilter詳解 Hbase—— rowkey 過濾器（rowfilter）